Okay. All right. I guess we can go ahead and start. All right. So thanks everyone for staying here this late for the lighting talks. So I'll talk about a new project that we're starting called pivot and this is a project to collect data from vehicles. So in vehicle data, mostly. And you might be asking yourselves, you know, what are, why do we need in vehicle data? What are the uses? What are the applications? You know, what we can do with it? And let me try to motivate this with a little video. And this is one example as to why you might want to collect in vehicle data. I'm going to get it to start. That's too bad. I was hoping to show this up. Okay. I guess we're going to have to skip this. Sorry about that. So what the video was, and I thought it was going to be interesting to you guys, it was a video showing a test crash. So it was a video that was showing a car being pulled into a truck and then another truck behind it coming in and essentially smashing the car in between two trucks. And the reason that video was done, that experiment was done, was to collect data from black boxes inside cars to collect information about crashes, right? So it was there to collect, to see what information is in black boxes so you can recreate a crash and study a crash and draw conclusions. So that's just one example as to why we may want to collect in vehicle data. There's many more other applications that will have us collect that type of data. So the data, they're needed by researchers who are trying to advance the state of the art, but currently these type of data in vehicle data are very ad hoc. So people collect them to run their experiments. They save them somewhere that don't necessarily get published to the rest of the world. The researchers do their experiments and then the data is left there to language and other people don't get to use it. So this project among others is there to collect the data that other people have produced and make it available to researchers in addition to the data that we collect. So let's see what else. Potential applications. There are applications in the many different types in many different areas that this data can be used for vehicle applications. You can use this data to do system monitoring in the car and optimization. You can collect data from in vehicle infotainment. You can do predictive maintenance and a lot of companies out there already do that. They collect data from your car and then they might send you an email saying your battery is about to die or your engine is having a problem. According to our data, you might want to make an appointment with a repair shop. Other applications include route and trip planning and also information from ADAS from connected vehicles, electric vehicles and so forth. At a higher level, these data can be used by transportation and fleet management to work on passenger safety, traffic management, do ride sharing applications and multimodal mobility where you combine cars and scooters and so forth. And even by insurance. Insurance is actually one of the major entities that require this data and they want it so they can create a new insurance model.  The data may be used by smart cities to do infrastructure monitoring, to do weather sensing, to do asset management and so forth. But also it can be used for safety and cyber security. One example that a lot of companies out there are doing is intrusion detection systems in cars by monitoring the voltages inside a canvas. And this allows them to detect whether a new device has been inserted in a can that perhaps was not authorized to be there. So what is this project? So this project is an effort between the University of Memphis where I am, Colorado State University where Jeremy Daley is one of my co-PIs. And also the USC of the University of Southern California Information Sciences Institute where David Balanson is with Wes Hatterger. The other collaborator in this project is the company, Geotub. And what Geotub does is a telematics company. They're based in Canada, but they're pretty much international. And they have agreed to work with us on this project and they provide some of the data that they collect and make it available to fleet managers. So it's aggregated data so it's anonymized, but it's data from a lot of cars that through this project can be made available to researchers. It's a three-year project started in October and since September 2025, it's about $1.8 million. And the purpose of the project is to support the NSF community or the National Science Foundation community in the U.S. But this is just, this is not exactly true. The NSF will be more than happy if we support the international community with this project. How do you build a project like that? Well, we believe that you need five pillars to build this project. You need your platform, your web server that supports all the user accounts when users can be looking for data and so forth or searching for data and so forth. We need another pillar, which is the collection and curation of the data that we are going to make available to researchers. We need services for researchers for sharing, securing and evaluating data sets. We need tools that researchers can use to manipulate the data and we need community outreach. So this is community outreach is a very important part of this project. And what we're trying to do here is advertise the fact that this infrastructure is there and engage with the community to see what they want, what kind of tools, what kind of services, what kind of data they would like to have. So this is a project that is very feedback oriented. So we work with the community and we try to respond to needs from the community. So going into the five pillars in a little more detail, the pivot platform, it's nothing that you do not expect. So we will have our web server. We will have the, will include security obviously because some of the data we collect is sensitive. It will be hosted on our machines in Memphis and it will get mirrored in Colorado. So this is your standard web service platform with the software needed to do the social interaction user accounts and so forth. The services we plan to deploy are access to data sets and tools for all researchers for data sets that are external. In other words, data sets that we do not collect, we will provide links. We will provide access to the geotab data sets, the data that they provide. We will also provide data from another project called Spindle. I'll talk about it in just a second. And we will also provide data sets that will be crowdsourced through devices that we will distribute. I'll talk about that in just a second too. We will also provide privacy support. So for data that we collect through crowdsourcing, there will be nobs, privacy nobs, that the contributors will be able to turn to decide what level of privacy they're willing to include in the data. We will also provide something called institutional review boards support. Now the IRBs are something that all U.S. universities have, and I suspect other universities in the rest of the world. And what these review boards do is whenever you do research that may involve human subjects, you talk to them, describe what the research is, and then you get approval from them that, yes, it's okay to do the research or no, you have to do this to protect your human subjects. It's a very tricky area. It's sometimes hard and not very easy for researchers to navigate. So we will engage with our IRB and share experiences with other universities who may want to do the same thing. And then we'll provide community coordination and interaction. Now, what data sets are we going to provide? So those are divided into three broad categories. The first one is pretty obvious, is data sets that are out there but researchers at large may not know about. So we will go out, find them, review them and bring them in if we need to, and make the community aware that these data sets have been produced, they have been used to write papers, they have been used to do research, and here's a description of what they are and then you decide whether you want to use it. So that's the easy part. The next category is data from GOTAP, the company, and this is data that GOTAP will provide to us. As I mentioned earlier, this is aggregated data from their devices. We will also provide data from GOTAP, but this is from a program that we run. So in other words, we have a set of a small number of volunteers who have installed devices, telematics devices in their cars, and that data, we don't have restrictions in terms of what we make available. Because it's coming from volunteers, that data is fully available to us and we are free to distribute it according to what the small group of volunteers are designed. And finally, the other type of data is through hand-lockers, devices that we are implementing and we will distribute to the community, and this is data from cars and trucks. So a little more detail on community data sets. Again, this is data that's out there. A few examples are listed here. If you haven't heard of these data sets and if you're looking for data sets and you haven't seen these, then that's why we are here to find them for you. The GOTAP telematics data, it's coming from a service that GOTAP provides called Altitude. This is data from all the cars that they serve, about two and a half million cars around the world. And it's aggregated data, and we are going to get two main data sets from them. One is called Intersection, so this is data about intersections. How long does it take to go through an intersection, what's the average speed, and so forth. And the other data set is called Roads, and this is characteristics about roads in general, about mobility and about data that allows you to do individual road analysis. So the advantage of the data from GOTAP is that while it's anonymized and aggregated, it's coming from a lot of cars, millions of cars. On the other end of the spectrum, again with GOTAP's help, we run a small program that we call Spindle, and essentially what this program is, is a few of us installing GOTAP devices in our cars and that day using GOTAP's cloud infrastructure to collect that data, but then we go to the cloud and collect that data for us. Okay, alright, I'll be done in just a second. Alright, here's an example of the data we collect from Spindle. Here's a map of me going to the airport and then engine information. This is a picture of the camp logger, so we are planning to implement and install in the cars. And this is an example of data that we will be collecting through camp loggers, and this is a knowledge pyramid, and we go from very low level data and then building on top of it up to high level data. This is the type of outreach that we will do, it's what you expect, webinars and educational events and so forth. Let me not go through the benefits of Pivot, hopefully you'll see these, but very quickly these are the benefits that we expect for the AGL community and then benefits for the NSF researchers on our side. We think there's benefits on both sides. Alright, and these are the usual suspects, and there's a website that you can go to and start getting information and very soon you'll be seeing data on that website, but feel free to use these links to send us email and engage in whatever capacity you want. So I was under the impression that I had 20 minutes and that you're stopping me at 10. Anyway, so sorry for the rush at the end. Any questions, anything that I can answer? So again, to recap, this is a project to collect in-vehicle data and make it available to the community along with all the services and the community outreach to go with them with the purpose of fostering research, developing applications and essentially giving you the data so you can come up with whatever research or ideas for applications that you may have. And we will happily take any data that you have created if you create a data but you don't want to go through the trouble of advertising, distributing it, then we'll happily take the data from you and do the advertisement for you if you want to give that data to the community. Okay, alright. If there's any noble questions, thank you.


Thank you. Welcome. My name is Arvind Tripathi and I'm an associate professor here in business school and I would like to welcome you all and I see that we have a lot of people from faculty of science and engineering and from business school and thank you for coming to this wonderful event. Before we start, just a few health and safety announcement. So exits are right in the back. In case of emergency, get out of there and get to the vineyard straight and I'm sure the building is very well designed so you can find bathrooms would not be a problem. So today the talk is about product design and we have wonderful speaker Andrew Law from Netflix and so I know many of you probably use mobile devices for streaming videos and all kinds of content and Andrew is the one who is behind all that. So in case your device or streaming is not working, you know who to call. So I would just invite Andrew now. Thank you. Thank you. Awesome. So just a little bit of context setting. I'm going to do a presentation that's roughly 25-30 minutes and then we want to open it up for questions. So if you have those burning questions about Netflix, how we work, how we develop our product, please get them ready and I'll be happy to answer as many of them as possible towards the end. Before we get started I want to show we are a streaming company so I'm going to play a video from YouTube. It's when we made. Hopefully the audio works and it's not too loud. So that's a video that we made for a feature that we created called smart downloads which launched not too long ago and that's the type of innovation work that my team is responsible for specifically on mobile as well as the website. So those are my two kind of areas of responsibility. So I gave this talk earlier today. I've been talking for like five hours but I'm really excited to be able to share some of the stuff with you guys and hopefully have a conversation about it. I got into design about 14 years ago. I started doing very very small projects just kind of for fun. So I did things like posters for friends bands and album covers and t-shirts and and fun things like that before I ever knew that this was going to be my career. I was sitting in seats very similar to this back at San Jose State which is where I went to school and I went to business school and I have a degree in business marketing and administration but I use very little of my degree. Don't worry you guys will. But a lot of kind of what got me into design was really wanting to understand why people make the decisions that they make and how I could influence those decisions through design. So I'm going to be talking a little bit about failure today. Specifically failure at Netflix and how that leads us to building better global products. And there's three different areas that I'm going to talk about. I'm going to talk about A B testing what that is and how it works and why we employ it. Research the different types of research that we cover. And I want to talk about empathy. Empathy is extremely important because I am from the Silicon Valley. A lot of my design team is from North America but we're building a global product for 190 countries for over 130 million members in 22 different languages. But we are not representative of the people that we're actually building our products for anymore. So how do we build empathy along the way so that we build the best products possible. Sound good? All right. This was supposed to be moving. So this last 20 seconds. So we'll just wait. We do lots of shows. How many people here use Netflix? Okay that's what I like to see. How many people here pay for Netflix? Okay so if we could just document the people that need to pay I can accept all forms of credit. That's great. So I want to talk a little bit about product design and that's kind of my field. That's where my expertise are. So the product designer's job at Netflix is to find if there's a there there. And what I mean by that is find if there's an opportunity that's going to grow the business. And that's really my team's job. So we don't just push pixels. We don't just create pretty things that are usable. We identify different areas of opportunity inside the existing business and potentially outside the existing business. And then we build, prototype, test, and do qualitative and quantitative research to better understand if that's something that we should actually roll out to our members. Smart downloads being an example of that. So let's say that we're all product designers at Netflix. Congratulations on your new job.  This is great. So how do you take an idea and actually validate it? So how do you take something and understand whether it's something you should pursue or something that you should kind of let fall by the way side? I kind of use this rough structure as a framework. So the first thing you want to do is try to identify an opportunity. I'm going to use smart downloads because that's a video that I showed earlier. So we already had a download feature in the app. It worked. But there was a highly manual process. You'd go in, you'd find something, you'd click download, you'd wait for it to download, and then you'd hopefully consume it later. So we thought that there was an opportunity for this when we started looking at the data of how many people are using downloads and how they're actually using it. So we had an idea and it was this idea of smart downloads. We didn't really know what it was going to be at the time. But we asked ourselves, does that align to our business objectives? And we want more people to be able to use our product in situations where they have little or no connectivity. So we said, yeah, like that actually aligns with our business goals. Does it mean a user need? Absolutely. Through a lot of the research that we did, we discovered that people oftentimes are not using the North American use case, which is what we think of where, hey, maybe I'll use downloads if I go on a flight or I'm in a subway or something like that. We studied people that are in Mumbai and in Singapore that have connectivity sometimes, but it's really spotty at best. And sometimes people are spending three hours in a car commuting to and from work. So they absolutely there was a user need. How large is that need? So it can be a need, it can be a valid thing. But if only a subset of your members are using that or would use it, is it actually meaningful for you? Probably not, it's probably not something you actually want to pursue. And then the opportunity costs. We do not have finite resources. We have a small number of people from the design side. We have a smaller engineering team. So we have to re-prioritize. So is it worth the opportunity costs? Because if you can do this, you can't do this oftentimes. And if it aligns, then you should absolutely do it and pursue it. So once you have the idea, how do you actually take that and build it? How do you take it from zero to one? The skill is not incoming up with the ideas. The skill is executing the ideas. Everybody in here has ideas. Some of you will pursue them. Some of you will throw caution to the wind and absolutely go for them. And others are just going to have good ideas that never get out the door. And everything is in execution. So we use this framework to basically, and it's very idealistic. It doesn't always happen exactly in this way. So take it with a little bit with a grain of salt and all kind of caveat when sometimes we spend backwards and come back to ideas. But we try to start with what we know. So with the data that we had, we realized for smart downloads that there was this behavior where people would watch a few episodes, usually about three, or sorry, download about three episodes. They would then consume them often when they were not connected to Wi-Fi. And then they would redo that same thing when they were connected to Wi-Fi again. So there's an opportunity there. Any time you have a repetitive process, there's an opportunity for innovation to take some of the weight off of the user and for us to do that work for them. And that's what smart downloads is. So there was foundational research that was done. So you have to start out with what you need, but you also have to ask yourself what other information do I lack? So where are the gaps in my hypothesis? Where are the gaps in my thinking? And how can I fill those? And again, ask yourself if there's a there-there. The next step is to ideate. Ideation for us looks like kind of very, very broad. We start in a bunch of different directions because we don't know exactly where we want to go. And we want to explore and not leave any stone unturned. So we will kind of take a very kind of wide, I guess, brushstroke approach at the beginning of something and then narrow that down through different qualitative research methods. The next step is to prototype and learn. It's extremely important that if you're going to show something from a personalized service like Netflix that it's as accurate as possible. So we have prototypes where we can actually sign in with your unique IDs and show you content that is familiar to you, including the last things that you've watched. But it's all in kind of a prototype, so it's all in a build that no one else has access to. But through that process, we'll try to get the most realistic understanding of how impactful that feature or idea is to you, the member. Oftentimes, what we'll see in qualitative research, we actually have to turn into actionable insights. So some of those things, when we synthesize the learnings, we'll come back and say, actually, you know what? When we explore something like smart downloads, we should really be mindful of conserving data. So we should maybe be conservative and do this as something that's only while you're connected to Wi-Fi, which is exactly what we did, and that's a learning that came from that process, from that project as well. At Netflix, we do A-B testing and a lot of people understand A-B testing as kind of if you take a user and you drop them on website A or website B, which one performed better. That is a very basic approach of A-B testing. We do multivariate testing, which means that we'll have 12, upwards of 12 to 20 different cells to identify each individual variable. So not only will I be able to understand why website B worked best, but exactly what nuance of that design outperformed the other and then why. So I'll talk a little bit about more about A-B testing in a couple slides. And then again, once you actually have something that you've learned is meaningful for our members, it's something that we will productize. I was doing a talk earlier today to project managers and product managers group, and this is where there's a little bit of friction between design and PMs. Design wants to do what's right for the user and usually PMs want to do what's right for the business, and so there oftentimes are concessions that are made around when you're going to actually productize something and ship it to your members. And then if that all works, you do it again. So to have a culture of testing, you have to support it from every single level. Reed Hastings, our CEO, started the company back in 1997 shipping DVDs in the U.S. mail. Everyone's pretty young in here. So a DVD, I'm just kidding. So that's how the company, that's how the company started. And he was really interested in understanding where that could go. So could that scale, could that go globally? They really kind of encouraged everyone in the company to explore ways that the company could continue to grow and scale. And so the level of that culture is really embedded in who we are as as Netflixers. Anybody can come up with an idea and inside of our freedom of responsibility culture, you have the opportunity to go explore if that's something that you think is actually meaningful. So I've done projects, again I have a business degree, but I've done projects that had no design, no pixels at all, and tried to understand if that was material for us as a business. But inside of that, at Netflix there are no sacred cows and every now and then you need to make a hamburger. We didn't get to where we're at today because we thought DVDs by mail were the right way to go. We got to where we're at today because we thought it was the wrong way to go and we challenged our own assumptions and so we just we disrupted initially brick-and-mortar blockbuster stores where you'd go rent a DVD and then we created streaming and made that a thing and expanded that out to North America and then we licensed more and more content to make sure that we had things that everyone in here would like to watch and then when that got really expensive we started investing in our own content and creating Netflix originals. And so we continued to disrupt ourselves and the next kind of phase of what that looks like is hopefully disrupting storytelling. So the reason that anybody watches a 30-minute show and you really only get 22 minutes of that is because the advertisers had an eight-minute ad block. We will never have ads. I'll say that right here now. You'll never see an ad on Netflix and because we don't have that we can really support the stories that people want to tell in any fashion and format that they want to tell it. So if we want to tell something that's culturally sensitive we can tell that because again we're not appealing to advertisers. We're appealing to you, our members. And if you like it then you'll tell us. You'll watch it and if not then we'll probably end up making less of that type of content in the future. So AB testing. Again this is we test everything that is in the product today so everyone in here is probably in 20 or so AB tests right now and you just don't know it. So your Netflix experience is vastly different from your Netflix experience and hopefully you guys don't compare notes. But why do we do it? So it really helps us rule out those bad ideas and focus on larger more risky product bets. The earlier you can get a read on something whether you should actually pursue it or not the better because we're talking about time we're talking about money and we're talking about resources and we want to conserve whenever possible. We apply the scientific approach. This is why we do multivariate testing because we want to isolate each of the individual variables to understand what worked and why or what didn't work and why. Because ultimately a failure is not a failure as long as you learn something. And we test large ideas we don't test like the color of a button so we would never have a test that is really kind of like nuanced into like typography or the granularity of you know iconography even. We'll get a lot of the sense of that just because of the fact that all my designers have been designing for so long that we understand what people are going to be able to gravitate to what patterns people are going to understand and be able to adapt to and learn. And one of the really important things with AB testing is you want to understand what success looks like before you start. So it's really easy when you run AB test and you get a green result or if you get a a win as we call it to just like not ask any other questions and just ship that out. But it's really difficult when something doesn't work if you haven't asked yourself why you might think it won't work before you go into it. So make sure you understand what success is before you run the tests. And failure is absolutely only the beginning. If I had a scoreboard behind me tallying my wins and my losses I would have far more in the loss column than in the win column. But all of those things all of those failures helped me to learn and helped me to get better and produce those wins. So I still look at them as failures but I was able to glean some kind of insight out of that which is why we're able to drive the business forward. So as I mentioned before we're in 190 countries 22 languages with 130 million members. We are not representative of the people that we're designing for anymore. So we really can't start with this as our basis anymore. This worked for a while when most of the netflix business was inside the united states. It's no longer. We're a fully international company. So we really have to be mindful of not just this and not just these which is where a lot of companies will tend to focus and spend their their dollars. And this is where we focus originally as well. But you have to expand into latin and understand the cultural nuances and family structures. You have to understand outside of that as well the rest of the world and understanding multi-generational households in India and how one piece of content to one family member will not be accepted by another. And so when we do when we go and do research oftentimes we will spend time in people's homes. We'll spend entire days with people shadowing them around and just seeing how they use their phone how they use their the website how they use the internet for entertainment to try to understand and identify opportunities that again will grow the business. This is by far the thing that keeps me up at night the most. It's a huge challenge. We have last year we spent six billion dollars on content. This year we're spending more. We're just about the largest studio in the world. And with all of that stuff coming to the service hopefully there's something that's great for everybody in this room. But how do you deliver the right thing at the right time to the right person in the right context? That's a huge challenge and that causes me many sleepless nights. So like I was saying some of the research that we do we're really trying to build empathy through that research. That's why we try to get on the ground as much as possible and meet folks and talk to them what works well for them what doesn't why. We try to find out hacky ways that people are using the service that we never would have thought of before. We have a share button in the app but a lot of people will take a screenshot and like SMS that to a friend. So we wanted to understand why. Why are they doing that? Why aren't they using the afford and sits right there and how can we support that behavior if we feel like it's something that we should model. So empathy is a muscle just like any other it can be built stronger and shaped over time. I have learned a lot a lot from my team in the last couple years about empathy and I've seen a lot of and heard a lot of great stories that have helped me to kind of bond and that's really what it is. We do a lot of travel we do a lot of research with participants and we do a lot of person on the street interviews to again clean some sort of insight into somebody else's perspective and life and that helps us to be better product designers. We're flying through this by the way. We'll have plenty of time for questions. So really understanding someone's frame of reference can lead to meaningful innovation. This would have been kind of scoffed at even a few years ago if you just put this up on a screen. But we can prove it through the the success that we've had in the product by understanding and cleaning insights from someone's life and then mirroring that back in the product to to create an experience that is meaningful. When you empathize you tap into emotions and you bond over those shared feelings and experiences. As a little kid in North America we always do like summer camp so we'll go out into the woods with other kids and we have tents and campfires and all those things. And what was great about those is the bonding experience that you'd have with those other kids. You'd have a shared experience and you'd often come back to your siblings you come back to your family and they wouldn't be as excited as you were. You didn't really know why. It's because you were building empathy with those kids. You were building an understanding. You had that shared emotional connection and emotional bond. I would like to replicate that as much as possible with as many people as I can so that I create a diverse perspective of understanding and then I can actually use that in material ways for the business. Everyone in here has biases. There's some amazing tests that you can do to to identify your unconscious bias. But you really need to understand these. If you're a designer this will help you immensely. If you're not a designer if you're just trying to be a good human this will also help you. We all have our biases. I was really surprised to learn what some of mine are. I won't share them with you. But the first thing is really awareness. You can do things to kind of curb them over time. And this is really important. You don't do empathy, you have empathy. It is something that you have to make a conscious effort about. You have to employ it on a daily basis. And again being aware, being conscious of it is going to help you. And you're absolutely going to going to mess this up. That's okay. I think that the more people you are able to meet and the more people you're able to connect with is going to eventually lead to you being a more well-rounded human. And again if you're if your goal is to create a business or build products for people this is a skill that you really should be developing now. Okay, research types. This is I have two slides left. So noise is extremely common in research and you have to know that going in because you want to take things with a grain of salt. People are terrible at predicting their own behavior. If I asked if I pointed to one of you and asked what would you do you know tonight after after this is done you're going to tell me something and at most at best it's going to be aspirational. Because I'm asking you in a room like this you might say something like you're going to go home and watch a documentary. When really you're going to go watch cupcake wars. So we're really bad at predicting our own behavior and more even worse at socializing what that is. So you have to understand that when you go into a research scenario that you have you have to be looking for kind of more the themes than the actual words that people are saying. So I mentioned this a little bit. The the research that we tend to employ especially at the beginning of a project is more ethnographic in nature. This is great to get a foundational basis of understanding. We did this pretty recently in Japan before that we did it in Singapore because we were seeing patterns of behavior that were kind of anomalies like it just didn't make sense. The data wasn't telling us the complete story and so we traveled there and met with folks and talked to them and spent time in their homes with their families to better understand how they were using the service. The first one can be quite expensive. The second one is you can do this on a shoestring budget. I still do this today even as a designer in a corporation. I think that this is a really really meaningful and powerful way of connecting with people and we just call them personal on the street interviews.  They still believe me like I'm going back to school or something. I say that I'm a student and I'm doing a research project and I would just like to understand how they're using their phone for entertainment and then you just have a conversation with that person and and you don't have any objectives other than to just learn and listen to them. We often give them like a $5 startling gift card or something as well. Video diaries so if you can't travel you can do different types of recruiting in different countries where you ask them to do a certain kind of step by step process. So we might send them phones or we might have them download an early build of our app and then they'll use it for a few weeks so that it's not just this lab lab scenario and that's the last one but it's not so it's a lab like an artificial home scenario. We want people to be able to use it in their daily lives and see if it's a behavior that becomes repetitive for them. And the last is lab sessions. We typically use these early on in the process and at the end of the process and a real quick note if anybody who's taking pictures like go for it totally it's totally fine I will post this whole deck to my website so that you can get everything you get the speaker notes and everything else so just know that. And we employ lab sessions at two different times early on in the process when we're when we're trying to get a good gut check of a direction of taking something and then later on in the process when we want to do a usability study or a usability qual it's what we call them. So really understand we want to see that if we create a new feature it's usable but our our user base is not is not you know 18 to 24 males. Our user base is three year olds to 93 year olds and so we have to create things that are going to work well not only for you know people in Palo Alto that are tech savvy or people here that are tech savvy but it's got to work for again that multi-generational household in Mumbai that I mentioned before. So I know all of you are students you've probably had a long day so we'll end it there. I do offer if anybody has questions if you are looking how to launch your career if you want to get into design I do offer mentorship it's free it's unlimited it might take a little bit of time for me to get back to you give them the time difference but if you ever have any questions or anything like that my emails up there my my Twitter's up there feel free to to reach out. People poured a lot of time and attention into me and my career getting getting to the place that I am today and I want to give back as much as I can. So any questions that you guys have I'm happy to field them here if it's something you don't want to ask in front of a crowd feel free to email me or DM me on Twitter and I will happy happily get back to you. Thank you very much. We have a few minutes for questions and we'll take questions from anyone. When you do do multivariate testing what kind of minimum sample size are you looking at? So that's that's a great question so if anybody didn't hear when we do multivariate testing what's the sample size that we're looking at. We typically allocate a few million members to each cell which again is not it's not feasible for everybody to do that but we're at relatively small scale compared to something like an Amazon where they can turn on an A-B test and turn it off five minutes later and and have you know millions of impressions of a page or or something like that. So we want to and it's also because of our our structure that we offer like a free 30 day trial so we need to make sure that someone comes to the service and then retains as a member past that first pay period to determine whether that's a successful product feature or idea or not. We want to make sure that they stay paying members. Yeah thanks for the question. Question there? Oh sorry. Hello I'm here. So how do you identify your biases? So there's a lot of tests that you can take if you just go online and search unconscious bias. I think Harvard has a few different tests. They're really easy it it's basically time and reaction based so they might show you a picture and then ask you a question and it's all about like how quickly you can answer these questions to try to uncover some of these unconscious biases that you have. But just do a google search you'll find there's tons of tests. So how do you find good product designers? How do you hire or call them easily? That's a great question. So my job is basically building the best team that I can giving them the right context the right resources and then getting the heck out of the way and letting them do their job. So I spend a lot of time looking for diversity of thought. I know from research has shown that the more diverse team the more likely you are to succeed. So I try to get people with diverse perspectives. It's very hard in Trump's America to hire people from outside of the U.S. but it is something that I have I currently have a centralized design team where everybody kind of sits in one location. I'm often kind of exploring the idea of hiring people in our India office hiring people from Amsterdam and our Amsterdam office and having a decentralized team to get that diversity. But a great question. Hi I'm just curious what do you test in your A.B. testing or the multivariate testing? So you mentioned that you don't of course test the color of the button or whatever but what do you test? Is it usually the type of content that they prepare to watch? No that's a great question. So for the feature that I showed called smart downloads a test cell might look like the number of episodes that we download automatically for you. So it's basically and we actually tested this. We tested whether we should automatically download like three episodes for that viewer or another test cell was we'll just we'll download as many episodes as you've consumed and then another test cell might be we'll download as many episodes as you started to download so if you downloaded one we'll delete one and swap that back in. If you downloaded three and you watch three then we'll delete three. So it's just these like different larger ideas that we're trying to understand which of those aspects of the feature are successful. Does that answer your question? It's it's not a it's not a really easy concept I wish that there was like a you know straight way of talking about it. It is and in that test we had 12 different cells that were doing those things so I'm not a professional at that stuff. I have amazing science and algorithm folks that that help us to kind of go through and identify every kind of nuance that we want to test. Isolate those variables across the cells and then we might actually product size something that has a little bit of cell eight, a little bit of cell five, and a little bit of cell two. So one is always control which is your current experience. Yeah you mentioned the challenge of becoming global a global company. One of the things of course with content is that some content in some countries will be offensive or not wanted and in other countries to be fine. So how do you manage that challenge like do you prevent some countries from watching certain things or what do you do there? So the only way that we prevent people from watching things are would be because of a licensing deal. So the the catalog is not the same in every region around the world usually because of that. A Netflix original should be available just about anywhere in the world except for some of our first few originals that were hard for us to license. But no we don't we don't gate or filter or sensor or anything like that. We want to provide as much information in the UI to allow that individual to make a good decision for themselves. So we don't want to bait and switch and tell you that it's hey this isn't graphic when it really is. We want to provide as much that information up upfront and we actually work with government agencies to kind of standardize rating systems so that we can actually tell different governments hey this is this is how you should be thinking about this type of content and then they'll kind of use that in country to figure out if that's actually right and then we'll build trust with them over time so that we're we're trying to pick up those cultural sensitivities as well. It's not a perfect system it's getting better. Hi, in your experience with product design what technical skills and soft skills do you think are important for like this sort of role? For my role? No skills at all. For your role and like the role in your team? So hard skills and soft skills. So the hard skills look like the software programs that you use. So Photoshop, Illustrator, Sketch, some sort of prototyping tool like Framer which uses coffee script or Envision which if you go on Envision's website you'll see me all over it. Any those those are the hard skills that you need. The soft skills that I find that are really really valuable are sales skills, marketing skills because it's not the best design that gets built it's the one that was socialized the best and sold the best and I don't mean sold to like other people in a in a you know used car sales way. I mean you're positioning right you're trying to position why this is impactful why this is meaningful and you want to create a case around that. So I find that people that are have had some kind of sales background even if they were a kid or something like that and you know sold cookies by the side of the road it's it's beneficial and it plays out and it kind of pays dividends as well. Great question. So how do you choose which countries have other offices because it's an international organization so yeah so we we basically created regional offices to handle kind of content in a specific region so we have offices in London and Amsterdam that handle a lot of Europe we have London's we have offices in Brazil that handles a lot of Latin Mexico City so it's it's more regional it's not about like hey this country is really important let's make sure we have an office there it's just more general than that but there are a lot of it is content plays as well we want to make sure that there are people that are representative of those markets that are buying content for those regions as well back there. So when you're designing your product do you design in sequence in terms of if you're designing for the desktop and the mobile application or do you do it simultaneously what is your process around the platform? Yeah that's a great question so we are not really rigid around having a unified product experience on all the platforms at the same time and actually because we're a very iterative culture and because we're always doing different tests oftentimes my iOS app and my Android app want to actually be the same and that's okay like we just that's something that we're just because we want to iterate because we want to kind of do these things really quickly that's just something that we have to kind of own so there are features where if you're on an iOS device today or if you're on an Android device today you're going to have kind of different things that we've rolled out and what we'll do is we'll test on one platform that has the resources for it we'll take our learnings and we'll do a hold back test on the other platform to understand if it has any negative implications on that platform and if not then then we'll have kind of unification but again we're always kind of doing the stair step yeah great question yeah um how much variation do you find between different user bases like and can you keep everyone happy uh what do you mean hold on don't give the mic back yet what do you mean by user bases like for example different age group different regions different gender or different platforms yeah so um I we we don't have gender information um that's not something that we typically I don't think we collect that in our sign-up flow or anything like that um we would only have like an email address to go off of and I don't think we're parsing those to try to figure out if that's a male or female um so I can talk a little bit more about like the platform stuff um so there's often this question of like well hey you know iOS has a platform and they have standards and they have design patterns and Android has a platform and they have standards and they have design patterns shouldn't you create two different experiences one that satisfies these and ones that satisfies these because there's different use cases and and different user needs um we've tested that many different times and many different cases and found that not to be the case at least for Netflix so our navigation pattern in both of our apps are the same we do not have an action bar for android that's at the top and a tab navigation bar for iOS that's at the bottom um we have a tab navigation bar for both um and so so a lot of times people think that there are different reasons we'll we test those to better understand it and often find that there are not that that large differences it's it's more so on the content side like people are very different with the types of stuff they like to watch which makes sense yeah um what do you guys have against movie trailers or am I in a test cell wait what do we have against them yeah they vanished from my netflix where were you seeing where are you seeing them when like navigating down and opening like the info on the website yeah you're in a test cell google google and you'll find you'll find those trailers sorry i was just wondering if i'm sorry uh do you use archetypes or personas if so how and do you iterate on them we do not use them um they kind of i know designers that that do enjoy them uh and it helps to kind of get a frame of reference to begin with um but i i think that those are often just generalizations um and it's better to to really actually like go out and meet someone and then build a kind of case study around an actual person um then then those generalizations um there's there's lots of designers like you can ask 10 different designers and they'll tell you 10 different reasons why they love those or hate those um but my team doesn't use those or employ those anyway i don't know who has the mic so if you have a mic you can ask questions you mentioned that everyone at netflix has the opportunity to innovate um and obviously it's a massive company so you've got engineering teams and analytics teams and design teams um so how do you deal with the tension between a great idea that comes from a design perspective um and the engineering side or the vice versa where you have an innovation from an engineering side but it doesn't make design sense how do how do you guys decide and deal with that tension yeah um it always comes down to opportunity size um what to one of my like first or second slides um if you think of it like a funnel um and you drew at the top of the funnel you drew 130 130 million right that's that's our member base if you do something that only is going to affect a small subset of those whether it's engineering or design driven we're probably not going to prioritize that work over something that's going to affect a larger population of folks um so it's really it's a give and take there's concessions all the time that are made um but it's it's oftentimes a conversation it's it's not like argumentative i think both as long as you come at uh as long as you come up with an idea that is in the best interest of the business and the user it's probably going to get built at some point but it'll it's just a prioritization game yeah um you mentioned testing against success criteria and defining the success criteria really clearly upfront are there any examples where you've uh defined what you believe success was and passed a test or gone gone live with the product and then realized you were actually testing against the wrong success criteria and if so what are some of the common flaws second question what are the common flaws and success metrics that you've seen and how they kind of get applied to testing so when you have success metrics you try to not make them like a snapshot in time where hey our business goals are so narrow that these are our success metrics and if we're able to accomplish those we've you know got ourselves another six months of of business before we have to reset so our our two metrics that we track are streaming which is streaming minutes and retention and they're highly correlated between the two so the more someone watches the more likely they are to be retained as a member and continue to pay for the service so those are our success metrics and they have been over quite some time but we're constantly kind of challenging the assumption that those are the right things over and over and and developing new metrics actually right now that are that are geared towards understanding a little bit more nuance that again it has to be based in the the data that we can actually gather so you could say that our our best metric is you know consumer happiness but how we measure that is is really difficult to get a concrete view of that relative to how many minutes somebody actually watched so i think the question is just always challenge the assumptions that we're making in our in our metrics but yeah does that answer your question thanks and with so many tests running at the same time how do you make sure you're getting the right insights out of them they're not impacting each other and then also along those lines how do you scale so that you can run so many tests at the same time yeah so we basically have flags in our system that determine that hey you can't be in this test and this test they're conflicting but a lot of our tests are algorithms right so we're running different algorithms to on the personalization side to make sure we're giving you better content that's more appropriate for kind of your viewing patterns so a lot of the like when i mentioned that everyone in here is in probably 20 tests half of those if not more are algorithm tests and it's easier to to get feedback on those relatively quickly because we'll buy something to you know 10 million people for a little bit of time to better understand it before kind of rolling out whether we go with that algorithm tweak or not so we we make sure that we aren't muddying the water um so that we can get those clean reads hi um i was just wondering why the content we have in new zealand is different to the ones in states or like other parts of the world yeah so that comes down to licensing um there it's it's really kind of an archaic system of uh how you buy content and distributed around the world so there's like first run rights second run rights um so when you're trying to find uh if we want to license a a marvel movie or something like that in the U.S. it's really really expensive to do that in that market it might be easier for us to do that in uh you know in New Zealand or it might be more difficult so that's why the content catalog is different depending on where you go in the world you know we're creating so many hopefully great shows and movies that people love and they'll be available everywhere Netflix is offered with no limitations and that's really the world that we're trying to get to hi there again oh hi just two questions that i have for the first one is um which other company apart from Netflix of course do you think has really great design UI design and UX design that was my first question the second one is how do you keep up with good design is that something that changes throughout years or is that through testing that you keep up with good design yeah so the first question around what other companies kind of I admire from the design perspective I think that Apple would probably be one of them just because of how how kind of clean and focused they are but another would be Airbnb around how they kind of tell stories that you can kind of create these memories through this experience when you're renting a room from a stranger right and I think the the last one would be would be Disney because of the power of storytelling that they've just developed and the brand that they create like they create worlds around their content and I think that that's something that's that's kind of really special and how they do that and staying on top of of good design I mean design in some ways is just subjective so what I want to create is I want to create something that's highly usable first and foremost and that's also hopefully provides people with joy at some level but ultimately it's a utility right you could think of us like a digital vending machine right like over down what kind of candy bar do you want what kind of show do you want the challenge is how do you kind of innovate on that canvas and how do you develop that to be something that's even more efficient and it allows people to make the right decision for them so I read tons of blogs and books and different design things to kind of inspire and another one is just listening to other people talk about designs and like kind of their passion and and things that they've tried in the past as well. Hey firstly great talk it was really awesome thanks for speaking thank you. I have a couple of questions on consistency and design systems so firstly how do you maintain consistency with such a global product in terms of user journeys if someone wants to make a change and across the UI do you have separate design systems for different products or is there one unified one? So you're ready to gasp we don't have design systems we don't have pattern libraries we don't have any of that stuff you absolutely need that if you're gonna have a junior team you absolutely need that if you're gonna have a really large team all over all of our products we have 51 product designers if you compare that to a facebook or google who have 400 500 600 designers and a lot of we don't hire I don't hire anybody directly out of school I only hire people that have like five to seven roughly years of experience so they've cut their teeth somewhere else right they've made those mistakes somewhere else so I don't have to deal with that and I don't have I have like more I'm hiring people that can make good sound judgment and that tend to be a little bit more mature with their judgment their logic and their reasoning and so I don't have to create those like hard and fast rules for them I want them to kind of go explore the edges and the fringes and find things that they think are interesting and bring them back so we don't have those systems it you know ask me in a couple years if if the team's doubled in size if we started to employ those but we don't have them today and then the consistency if you have a spectrum and you have consistency over here you have innovation over here okay like they're at odds with each other and so I'm much more and being inconsistent and innovative than I am about being consistent and not innovative okay I'm really curious to know how you factor in human ethics if at all into the design of your product so if you optimize design then someone's going to want to keep watching it all over again sort of thing is that considered or do you think that's bad if someone keeps watching it? I don't know I mean like blowing so so what somebody who asked me a question earlier today which was around you know don't you feel bad if somebody watches Netflix and gains three kilos to which I said plenty of people watch Netflix on the treadmill so so my job is not to make decisions for people my job is to help them make decisions for themselves and this might be a little bit of a somber note I'm done but my boss grew up homeless and in a very very rough neighborhood and so what video entertainment meant for him was safety he couldn't go outside because it was so dangerous outside so for him it was all about you know watching shows with his friends watching movies with his friends that's not an experience that I had at all I had the exact opposite of that and so I had similar questions when I started Netflix of like don't we have ethical more responsibilities to get people to watch you know better content right like why are we why are we jamming documentaries down people's throats but that's that's not what our job is our job is to tell great stories and distribute them around the world and let people you our members determine whether we create more of those stories by consuming them so I feel like I have an ethical responsibility to to at the very least be mindful of any negative kind of outcomes but there's a reason that I work in entertainment and not medical device manufacturing I don't want anybody's death on my head all right thank you all very much really appreciate it well thank you very much thanks well thanks for coming here and thanks for a wonderful talk and we knew Netflix a company where that is streaming all kinds of video but Andrew provided us a great insight of how understanding such a diverse content and consumers is such a challenge actually and so we have a small gift for Andrew oh thank you very much everybody's trying to get me drunk oh no it's an umbrella oh it's an umbrella okay good so keep me dry I didn't say get drunk awesome thank you guys again and please reach out on twitter or my email if you have any follow-up questions thank you


All right, so so now what in this last lecture I want to switch gears and We're going to completely forget about physics and we're going to define some natural geometric objects and I'm only going to be able to sketch it in the in the hour That I have but the point is that that in the appropriate sense the volume of these Geometric objects will end up corresponding to the scattering amplitude But I want to do enough so that you can see how the formula we saw a second ago is the actually the volume of an object that is sort of God-given and Naturally defined even if you'd never heard about it. So let me just give you a sketch of what the idea of the amplitude Hedron is and everything is going to be a Double generalization of the idea of a triangle. So we're really going to start thinking about all the points on the inside of a triangle and There's one generalization of a triangle well to tetrahedra and Simplicies in general so it's triangle to simplex and This has a Generalization into the grass manian. I'll tell you what that is But the generalization of the inside of a triangle into the grass manian is something known as the positive grass mania so that's one set of generalizations and This is something again As I was saying at the end of the last lecture you would have thought I mean the grass manian is just a space of k planes and n dimensions Well, it's projective space is just a set of lines in n dimensions The grass mania is just a set of k planes and n dimensions. Mr. Grossman who was a high school teacher In the 1850s. He couldn't get an academic job So he was a he was a high school math teacher But he invented quite quite strangely from our purposes both the grassman variables that we talked about for supersymmetry As well as the gross money in which is a set of k planes and n dimensions the same. Mr. Grossman did both things But anyway That's just a set of k planes and n dimensions again If you're a physicist you think who gives a crap about any of these things k planes boring, you know What could possibly be interesting about these things? But they're boring, but it sounds like a boring sort of thing that mathematicians do so Surely people thought hundreds of years ago about the analog of a triangle inside the grass manian, but This was not done. It wasn't done until the 2000s is to figure out What is the analog of the inside of the triangle in the aggress money around ten years ago? Sort of amazingly roughly simultaneously with one we are running into these objects from the study of scattering amplitudes It's what our mathematician friends. Well, I mean, I'm just joking. I love mathematicians there It's just a funny. It's just the usual It's the way that we express affection by making fun of each other. So But anyway, so so this this was Not studied till ten years ago. It's already sort of interesting. It took so long to study But and just like triangles are crucial for building up areas, right? That's the basic building block that like builds up areas We found that the pieces of the positive russ manian We're in one-to-one correspondence with the the building blocks out of which you could build up scattering amplitudes, okay? But it was a building blocks and not the whole amplitude itself And so we were struggling for a long time to find what is the object that actually? Builds the amplitudes rather than just gives you the building blocks and well That's the second naturalization we go from a triangle to a polygon the inside of a convex polygon and If we generalize the convex polygon into the grass manian in exactly the same way that we generalize a triangle What we get is the first example of an ample of an amputahedron that we call the tree amputahedron and This is the object whose volume calculates tree amplitudes and There's one more generalization Which is that in this geometric setup you can now ask a natural question of how you can hide particles So you have a you have this big geometry, but you have certain particles You don't want to look at anymore in a quite literal sense. It's a theory of hidden variables, okay? But it's not living in any local space time. It's living in this Funny grass manian space, but in this grass manian you learn how to hide particles And hiding particles corresponds to calculating quantum corrections That takes you to the loop amputahedron and the final up And the final thing to any loop order. Okay, so it's this double generalization and So all I'll have time for in this lecture is To just tell you a little bit about what this generalization is the first thing and a little bit about How we define the tree amputahedron? But I want to tell you enough so that we can see in one concrete example How it is that out of this all of this geometry you get an answer that looks like physics, okay? And where the locality and the uniterity come from out of the positive geometry all right So let's start again. We already talked in the previous At the end of the previous lecture that the points on the inside of a triangle 1 2 3 all the points on the inside All the points y on the inside of this triangle projectively are the form c1 z1 plus c2 z2 plus c3 z3 Okay, so it's c1 c2 c3 now I Can it's this up to any rescaling of c1 c2 and c3 so I'll write that a c1 c2 c3 mod gl1 Okay, but I have to have that all these C's are positive And I'll just say it once and for all here when I say all positive from now on I could mean either all positive or all negative I mean they're all the same sign, okay? That's only the ratios that are gl1 invariant, but just so I don't Say it annoyingly this all the time. I'm just going to from now on say they're all positive Okay, so that's the inside of a triangle c1 c2 c3 That triplet of numbers mod gl1 where they're all positive. What's the inside of a simplex a simplex Is just c1 c2 up the cn mod gl1 with all the cA is positive Okay, tetrahedron and higher simplices but now we want to find the generalization of this notion and You see here We're thinking about them this this as individual numbers mod gl1 But we can equivalently think of it this as a ray, right? It's a ray that passes through an origin a line that passes through an origin So these C's live in pn-1 right in this picture the C's live in pn-1 and they are so there's There are one planes in n dimensions Okay, so what we'd like to do is find some generalization of the notion of the simplex for k planes in n dimensions Okay, so how do I talk about k planes in n dimensions? So the space of k planes in n dimensions is called the Grasmani and GKN and Well, how do I talk about it? I just have to give you k n-dimensional vectors Here's a vector v1 up to a vector vk So these are n-dimensional vectors and they have to give you k of them If I give you k n-dimensional vectors then the span of those vectors is going to give me the k-point Right, so the data is given by giving you a k by n matrix But not quite because if I only care about the plane I should be able to do so this matrix It will be written as c alpha a okay a runs from 1 to n and alpha runs from 1 to k Okay, but c alpha a should be identified with any k by k linear transformation Okay on the rows because all I care about is the plane that's made out of them, right? So it's the space of so so this is the space of k by n matrices the alpha a k by n matrices Mod glk So we see the dimensionality of the space For example dimensionality of g kn is k times n minus the k squared from glk So that's k times n minus k Okay, so that's that's the that's just the the gross money and that's the space of k planes in n dimensions But what we want to do is find the analog of the inside of a simplex in the gross money in so the the analog of simplex is I want to take this k by n matrix and now let me imagine writing this matrix as a bunch of column vectors c1 through cn K by n and I want to somehow say that these are positive There's something positive about them right the positivity is needed to talk about a notion of inside right at the distinguished inside from from outside So what should be positive? Well, I can't say that the entries of this matrix are positive because that's not in very tend to k by k linear transformations Okay, what is invariant under k by k linear transformations are the determinants of this matrix So all I can say is that the determinants the k by k Subdeterminants are the miners. So if I take any k columns ca1 through ca k and I Contract them with an epsilon symbol. This is called a minor. This is the minor a1 a k is a minor of the matrix Then I can say well all the minors are positive So why don't I said all these minors are positive? Well, we just have to be slightly careful because the minor is anti-symmetric, right? The determinants are anti-symmetric So I can't literally say that all of them are Positive because one two and two one can't be both positive So what I have to do to make sense of this is introduce an ordering Okay, I have to order the columns one through n Okay, I order them one through n. That's not something I have to do for the simplex I didn't have to order them, but here I have to order them one through n and say that they're positive so long as a1 They're ordered Okay, so 1 3 5 is positive for example All the ordered ones are positive Okay, no, that's of course if we think back to amplitudes It's already interesting because we have the ordering and the scattering amplitudes and here We see that we need an ordering to make sense of the notion of inside in a simple way in the grass money All right, let's let's let's keep going Now when you have a simplex, it's of course extremely simple To imagine what the boundaries of the simplex look like you say that all these C's are positive And so you just put one of them to zero at a time and you go to the boundaries of the simplex And the boundaries are other simple C's the boundaries of a tetrahedron are triangles the boundary of a triangle is an interval Right and to go to the boundaries these variables that used to be positive you allow some of them to be zero Okay, so they're non negative So we're doing the same thing here. We want the non negative grass money in which I'll call the positive grass money in but But well, you might think well we do the same thing We can send any old minor that we want to zero and so the boundary structure is kind of trivial in the case of the simplex It's trivial. There are other simplices But it turns out for the grass money in the boundary structure is vastly more rich and structure and There's a whole that this is what that sort of ten years of mathematical literature Was about figuring out what the boundaries of the positive grass money and really look like and The reason is that these minors are now non-linear and they satisfy Quadratic relations between each other Okay, so so you can't just set them to zero willy-nilly one at a time instead of saying it abstractly Let me show you what the positive grass money and looks like In a particular case, which will be useful in us for a moment In principle, I should the first simple example is for k equals two But I'm actually going to jump to k equals three because it's even slightly easier to a visualize what's going on So what does I want to show you what does the positive? So there's a positive grass money in we can call g plus kn and I wanted to say what it looks like for k equals three So just for fun imagine that we have this three by n matrix But imagine that the top row of the matrix is positive all the numbers are positive There's some sub region which looks like that and let me rescale all of them to one Okay, if I rescale them to one, I don't change the fact that the minors are are positive, right? So so just just so we can visualize it I'm going to imagine a three by n matrix that looks like one in the top row and then down here There's some little two-dimensional vector x one up to x n. So this is a three by n Matrix and being positive means that one x a one x b One x c that this determinant is positive for a less than b less than c So let's get a feeling for what this means So Imagine the x's are just randomly sprinkled on the plane. These are now just literally two-dimensional vectors Okay, well obviously If I sprinkle them randomly, it's not going to be true that all these determinants will be positive What does it take for all the determinants to be positive? What do you think it takes? What what do you think these points have to look like for the determinants to be positive? They have to be organized into the vertices of a convex polygon x one x two x three and so on up to x n and Why is that because what is this determinant? This determinant is just the signed area of the triangle With vertices a b and c. So this is just telling you that every triangle a b c has got to be oriented the same way and For that to be true. Well, that's it's obviously true that they have to be inside the convex polygon So you see one two three one three four one three five everything is oriented in the in the correct way Okay, but now from this picture we can see why the boundary structure is so much richer Let's say I take so this is a generic point in positive g3 n, but let's say I want to take one of these minors I want to take the minor. Let's say 135 I want to take the minor 135 and put it to zero Okay, now what does that mean for the minor 135 to be zero means that this triangle's area is collapsing to zero Which means just that the points a b and c lie in a straight line Right, so let's say I do it. I take this point x3 and I want to drag it to to live on the line 1 5 Okay, so here I go. I'm going to take x3 and moving and moving oops before I get to hit the line 1 5 It will actually lie on the line 2 4 and Now if I keep going it won't be a convex polygon anymore Okay, so the modular space of points The modular space of convex polygons Is the positive g3 n in this case and It's very rich for this reason because as we start collapsing the sides first you can make consecutive points Lie on straight lines then you can do more interesting things like once Two three and four on a line. I can put four five and six on a line That's one thing I can do or I can take two three and four and I can take three and move it to the point four or move it to the point one and so on okay, so so there's a large set of Moves that I'm allowed to do combinatorially and you have to keep track of The thing that you're allowed to do next depends on where you came from and anyway There's a very very rich structure for what this very simple object of the modular space of convex polygons actually looks like So I'm not going to describe anything about it. It turns out to be All the facets of the positive rest money and are in one-to-one correspondence with certain kinds of permutations So there's ultimately there's a combinatorial object that sits underneath all of this But all I wanted you to see from here. Just is that is where the sort of richness and Complexity in the boundary structure comes from alright So that's the positive rest money a very simple object to define just this k by n matrices with all miners positive And that's the analog of the inside of a triangle in the rest money Okay, but now let's keep going to the second generalization of A triangle to a polygon or the inside of a convex polygon Okay, so So let's say I want to talk about the points on the inside of this polygon Well, all the points on the inside have exactly the same feature. They're all just a positive combination Imagine that there are masses on the outside is the same weighted average center of mass idea All the points are of this form. Okay, the yi are z1 up to zn again This is all projective. So I is running from 1 to 3 here Okay, it's a two-dimensional picture, but it's the vectors are three-dimensional. It's a projective picture and all the C's are all the C's are Are defined up to gl1 and the C's are all positive Okay, but I need to make sure that the Z's are Not random if the Z's are totally random, then I'm not going to get this picture I want to convex polygon not a not a random I want to convex polygon. So that means that I also have to demand we just learned what convexity means I have to demand that the miners made out of the Z's Z a1, Z a2, Z a3 are all positive if they're ordered Already you see that the notion of the inside of a polygon is very slightly Sophisticated right the inside of a polygon is jamming together two different positive structures on the one hand, we say that the external data is positive Okay, and then we take a positive combination of the positive external data Okay So the data is fixed and I scan over all the weight C's Which are positive and that? Carves out a region in the projective space P2 Which is the inside of the a polygon now? I can have a notion of I can triangulate this polygon That's a if we want to sort of cover it. I can triangulate the polygon Now what does a triangulation mean you see? The space of C's is n dimensional or n minus one dimensional when we mod up by the GL one It's much larger than the two-dimensional space of the polygon, right? So it's that's a highly redundant map. It's a highly highly redundant map from the space of C's into the Y space so a triangulation corresponds to finding a two-dimensional cells two-dimensional subregions in positive G1n So you see these guys live in positive G1n The data is in positive G3n, but what I want a triangulation corresponds to For example the inside of this triangle with with corners are 1 3 and 4 corresponds to a C Which is which is some C1 and C2 is 0 and C3 is C3 and C4 is C4 and the rest of them are 0 Okay So if I set everything to 0 but 3 of the C's and they're positive then I cover the inside of this little triangle So a triangulation corresponds to picking certain two-dimensional cells of positive G1n Right any two-dimensional cell positive G1n has an image with a triangle and a triangulation is choosing a particularly nice combination of these cells that covers the space so a triangulation for example Is the sum of 1i i plus 1 these triangles like 1 2 3 1 3 4 1 4 5 And so these are a bunch of triangles that cover the inside of the space Okay Now in this case this is all It's all very visual and we can see it we can see it all in action just by drawing pictures In a moment we're going to have to learn how to do this without being able to draw pictures because we're going to go outside Things that we can visualize in two or three dimensions So let's just talk about how we could discover some of these things sort of purely algebraically Okay, how could we discover purely algebraically? For example an important feature of the polygon is that the boundaries of the polygon The boundaries of the polygon Correspond so why is some point inside the polygon the boundary corresponds to when y lies on one of these lines Right, so I want to know what corresponds to the boundaries Well, it's when y lies on a line like one two or two three in general It's when y z i z i plus one goes to zero these are these are the boundaries Okay, now again, it's obvious from the picture of the polygon that it's one two two three three four and so on But how could we discover that if we were blind? We couldn't draw a picture and see it Well, let's say we want to investigate whether a given z a z b is z a z b is Z a z b a boundary of the space The a z b being a boundary means that y z a z b Would have to be always have one sign Let's say always be positive or greater than or equal to a zero somewhere, okay? And so the boundary would correspond to it hitting zero So just by contrast if I look at this example of a square one two three four You see that why one two is always positive, but why one three? See why is in here that why one three is positive? But if y is here when y one three is negative and that's because as I move in the interior of the space in the middle somewhere Why one three goes to zero, but it's in the middle of the space. It's not on the boundary. So it can have either sign okay, so What it means for it to be a boundary is that you find some some line segment where y z a z b is always positive Again here, it's all trivial because we see the answer, but let's see how we discover algebraically that this is where the boundary is Um Well first why is there any hope that there are boundaries at all? Well, let's let's let's expand it So we said that y looks like c 1 z 1 plus c n z n So that y z a z b would look like c 1 1 a b plus c 2 2 a b plus dot dot plus c n and a b and Now you see there is a hope that this might be positive because the C's are all positive and Also, these brackets are all positive if the Zeds are ordered if the Z's are ordered right if they're correctly ordered So there is a hope that sometimes they might be positive But now let's see when they are positive and let me draw a picture this circle is now just a circle of indices one two Three four somewhere here. There's a somewhere here. There's b. Okay, and then we're back to n Okay, so the C's are all positive. That's good. What about these? Brackets well, so one a b is positive. Okay, that's fine if a is less than b. That's fine Well, maybe two a b is positive But you see eventually someone in here is going to be negative There are some indices that might be there between a and b If there's any space between a and b there's always some bracket in there That's going to be negative because it'll be out of order right if there's some k here There'll be something that looks like k a b which is negative a k b and so this is negative and If it's true that there's any one guy in there that's negative We're screwed because there's always the C that just makes that one more important than the rest of them And we can make y z a z b arbitrarily negative So we can see that the only way we can have a situation where this always has the same sign is if there's no space at all between a and b Okay, and that's how we discover that the boundaries are Of the form I plus one right which of course again we see from the picture So we discover that why z i z i plus one is positive and now now let's say we want to triangulate the space Well, what does the triangulation mean? It means we want to cover it with a bunch of triangles such that the boundaries of the triangle is just the union of all these i i plus ones Well, so then what are you going to do? You know that you have to have these i i plus ones So the simplest thing to do is to have some triangle that has i i plus one in it and Just has any other point just put any other point that you want in there and Algebraically, how do we see this is the boundary? Well, I just think the boundary of this region and This is just in the usual sense of boundary. This is the sum over i star i minus star i plus one plus i i plus one That's the usual anti-symmetry of the boundary operator and this sum cancels telescopically over i Okay So that's how I see that the boundary is indeed the sum of i i plus one So that's why that's such a simple canonical triangulation and the example where I put it at one Is just you know one where where the triangulation point I chose to be One of the points of the polygon to be as efficient as possible. Okay All right now, let's generalize Away from a polygon the first thing I'm going to do is I'm going to stay in projective space even I'm not going to go to a Higher space and I'll stay well actually let's let's jump first straight to the full definition Okay, so now I want to generalize this notion of the inside of a polygon to the grass manion So what I'm going to do is the following so instead of having So what we had here polygon Was something that lived? it was a one plane in One plus two dimensions This is a space of y Okay, why I which is just this is This is p2 which is the same as g1 3 and And and and we defined a region inside g1 3 which was the inside of the polygon and How do we define this region? It was that why I was ca Z a I summed over a right so I runs from 1 to 3 a runs from 1 to n so a Little bit more loosely. We are just saying that y is equal to c dot z Where c is positive and z is positive. Okay, so that's a little mnemonic for what what what we did External data was positive and the C's are positive So now we're going to generalize this we're going to have y's that live as k planes and K plus m dimensions m is just going to be any integer 1 2 higher Here was the case m equals 2 and k equals 1 And so we're going to have external data The external data is going to be just n points d a I in K plus m dimensions Okay, so I is going to run from 1 to k plus m And now I have to tell you how to carve out a region in the space of k planes in k plus m dimensions So so the y's there's going to be k K plus m dimensional vectors so alpha is going to run from 1 to k. I want this to be a linear combination of the positive external data once again that runs from 1 to k plus m So this is fixed this data is fixed But it has to be positive positive in the sense that z a 1 these minors z a k plus m are positive This is fixed data External data and the C's have got to be positive in the sense of being in the positive grass manian gk I so that So that all these ordered minors of the C's c a 1 through c a k are Are positive? Okay, so that's it this this this defines the analog of the inside of the poly of a polygon in the grass manian and That's the amputated. That's the tree amputated. So this is the tree amputated for Tree amputated and that's labeled by n K and this an m and it lives in the space of y with the z's as external data so that's what the object is just a Geometrically now for physics it turns out that actually all the m's are important But but but most obviously to begin with for physics. We have m equals 4 so I'm telling you that the external data is K plus 4 dimensional K plus 4 dimensional vectors now the 4 you saw a second ago that that was these bosonic Momentum twisters we're talking about are just that four dimensional Four dimensional vectors for each external particle There are these extra k directions here though, right? What are these extra k directions? We didn't have that before that's the part that I don't have time to explain But in this story, there's no supersymmetry obviously. There's no adas. There's no super variables Everything is bosonic the geometry is completely a bosonic All of the super stuff is encoded in those extra components of the Z's okay bosonically encoded in the extra component of the Z's and actually geometrically what's going on is you have this k plane So if you imagine you have this this is a schematic picture. You have a k plus four dimensional space Here's why there's a there's why is a k plane in this k plus four dimensional space and now the Z's are points, but Some pieces of these vectors are in the direction of the y and some are not and the ones that are not in the direction of the Y these are the momentum twisters That we talked about before and the parts that are in the direction of the Y's is where all the super stuff goes here But is encoded in a completely bosonic way So in the end it's really true that if you hand to me the kinematical external data I associate it with it canonically This configuration of points in the in k plus four dimensions Okay Now something else that I haven't told you so this is just a space We're how am I getting functions out of this scattering up to their functions. They're not the they're not just just spaces but Once you have this space if I go back to the polygon If I go back to the polygon What I have is what I have is here's why and here's this region. There's this natural region associated with it Now it turns out this is a non-trivial fact, but it turns out that for these kinds of spaces Once you specify a region there is a certain So here here I have I have a polygon There's a certain differential form in in Y space There's a certain differential form on Y that depends on p and this differential form has the property with the property that it has it's fixed by the geometry by the property of having poles or More generally logarithmic singularities when you start taking higher residues other than just the code of mention one singularities logarithmic singularities on all the boundaries p of the shape What this means is that in the neighborhood of any place where the form develops singularities There is a choice of coordinates where in that neighborhood in the neighborhood of any Locusts of singularities the form looks like dx1 over x1 up to dxd over xd if it's a d-form Okay, that's what it means of having logarithmic singularities What's important about this kind of singularity is that if you take residues if you just take one complex variable dx over x means that you surround it with a circle and when you take a residue you get one And this means that this generalizes so the singularities as a complex object This thing has singularities where you take tori and surround all the possible all the possible Lower dimensional boundaries of the object that you're talking about and all the residues are one It's a simplest possible singularities you can have But that's how associated with the geometry you associate given a geometry There's a certain differential form with logarithmic singularities on the boundary of that geometry and That's how given the geometry we get a function and that function is the scattering amplitude Why do we know it's a scattering amplitude because The properties of the geometry force this form to have exactly the singularities of the scattering amplitude has to have Which involves putting particles on shell and all the rest of those things that we give all these physics Interpretation to in terms of evolution Unitary evolution in space-time is interpreted here instead as a consequence of what the of what the boundaries of this positive geometry looks like Okay, yes That that's that no no it's a well a particle going on shell in this picture corresponds to moving y to the boundaries Okay, so so so exploring the singularities of the function means you see This this I don't have time to explain in more detail But in the usual picture you would take the data and you move the data until you encounter a singularity That's dualized here with the data staying fixed and you move y around until you hit the Until you hit the singularities and so the singularities of the amplitude corresponds to just hitting boundaries of the amplitude But the fact that the fact that the amplitude factorizes when you sit on a pole is the is reflecting the Geometric fact that if you take this c dot z space and all the magic is what those determinants those determinants being positive It has the feature that when you go to a boundary you discover the the Amphitheidron breaks up and on the boundary What the geometry looks like is the two lower Amphitheidra? glued together with the addition of one additional Piece of apparent external data corresponding to the internal line, okay But that's that's a feature of the that's a feature of the a geometry. Yeah Yeah That's right that the in this language the actual residue would would correspond as usual and we have an amplitude the residue would Residue of the amplitude would correspond to the amplitude that you made it multiplied by the amplitude that it decayed on the other side Yeah, okay, so So just in the so anyway, that's so that's just the definition of the tree Amphitheidron I don't have time to define the idea of hiding particles in the loop Amphitheidron All I want to do in the in the last few minutes is go back to the simplest case of k equals one m equals four Where it's we're not even seeing the fancy gross money and structure yet here This is still just going to be living in a projective space. So it's just just a polytope But where we can Where we can see how we make contact with the physics that I told you about so So we're doing k equals one m equals four so the y's are Just the sum of or just ca zai But these z's now are five dimensional and they satisfy z a one through z a five are positive when the a's are ordered and The C's are positive That's the space that we're talking about so now this is living in G one five or p four so it's a four-dimensional shape not a two-dimensional shape Okay So it's harder to visualize But let's try to figure out Where are the boundaries for the boundaries? So let's just mimic the exercise we did for the polygon We want to ask so the boundary is now going to involve four z's rather than two z's But whereas can I have z a z b z c z d can I have why is the ab this all be positive? Is that possible? Well, it's exactly what we did before it will be c one one a b c d plus dot dot plus c n and a b c d should be positive and Let's again draw the circle of indices one two now. I have an a a b a C and a d Then down to n okay, and Just like for the case of the polygons We find that it could be has a chance to all be positive because these minors Ordered or positive and the C's are positive But if there's any space between a and b if there's any room there that'll get something negative If there's any room in any of these places that'll get something negative There is only one way that it's always positive and that's when it looks like I I plus one j j plus one Only when it's the boundaries are when y z i z i plus one z j z j plus one goes to zero And then you can say that all everything is positive because there's no room Between you only ever have to jump over an even number of guys in order to reorder it And so it's always positive So that's that's quite striking because it's remember we're asking on general grounds We have a function of a bunch of z i's why would it care about i i plus one and j j plus one? That's that's locality right that's that the poles should know about when x i minus x j squared goes to zero Right why even though it's a function of the z's does it care about the lines i i plus one j j plus one Well, that's a consequence of just the of the of the positive geometry in this case so we discover that the boundaries so we discover that that the boundaries are Just the sum over over all i j of this i i plus one j j plus ones And so how do we triangulate the space? It's exactly the same So the space is triangulated By the sum over i and j of some star i i plus one j j Plus one and by exactly the same argument as for the polygon you can discover that the that the boundaries of this Everything that doesn't involve i i plus one cancels Telescopically and you're just left with with the boundaries that that we want so so So So that's that's the geometry in this case It's triangulated by exactly the objects that we're talking about for the amplitude and the final part Is if you just work out what is the form of logarithmic singularities? Which just turns out to essentially be the area of these the volume of these? Simplices it's the area of the triangles in the previous example That that you precisely get to the to the formula for the scattering amplitude so So that just that's a little taste of what it looks like In the more general cases in the more general cases, it's much more intricate because the geometry is not linear So the boundaries are not defined by linear equations. There's a much richer There's a much richer structure involved But here at least you you see the sort of first taste of how it is that that the positivity is enforcing these physical properties in this case Most clearly locality, but when you go to the more the richer examples You see both the locality and the uniterity at tree and loop level coming out of this coming out of this picture All right, I'll leave it at that. Thanks