Good morning. Good morning. Good morning. Happy Thursday. Happy Thursday. What's up Pepper? How's it going? Hope everyone's off to a good start to their day. First one just sold out before going live here. So number two has six left. We've got Cardinals, Panthers, Broncos, Packers, Texans, Colts.  One of my favorite rips. Beautiful looking cards. Some nice new inserts this year that throwback designs honoring a 2002 and 2012. So those are some neat ones. The Nebula and the neon marbles are looking awesome this year. And of course the red, white and blue mojos.  So definitely looking forward to this. Again we got 12 of these going there. 10 box half case breaks and then ending the day with Bowman Baseball. Those are already sold out. What's up Alvin? How's it going? How's it going? All right, let me get y'all the link here again. Cardinals, Panthers, Broncos, Packers, Texans and Colts. One, two, three, four, five, six. Six teams left. And contenders, number two. There is the link for you. Six teams to go. Some really cool all-time contenders autograph sets here. Jerry Rice, Dan Marino, Troy Palamalu, Drew Brees, LT, Big Ben, Reggie Wayne, Warren Moon, Champ Bailey, Wes Welker, and Coach's Ticket Autograph Checklist. That's awesome. Andy Reid. I would buy an Andy Reid Coach's Ticket Auto. Mike McDaniel, that's a cool one. Sean McDermott, Brian DeBall, and Byron Lefwich. And then the contenders tribute autographs are really cool too. Heinz Ward, Anquan Bolden, Wes Welker, DeMarcus Ware, Stafford, JJ Watt, and Derek Carr. You got that. So cool. Heinz Ward is a 1998 contenders tribute auto. What's up, Kai? He says, Go Giants. Pull some Willis for you. And then the inductions look sick. Those look sick. Peyton Manning, these are all numbered to 25 or less. Randy Moss, Ray Lewis, Primetime, LT, Warren Sapp, Jason Taylor. Really, really cool checklist. I'll drop the link here. So y'all can check. Check out the checklist if you like. There's definitely some sneaky teams in this, for sure. Ben Atari, some some really neat names in here. Player of the Year Contenders Autograph. Josh Allen, Jalen Hertz, Herbert Burrow, Tua. Nick Chubb, Derek Henry, Tyra Kill, Justin Jefferson, Micah Parsons. Veteran Ticket Autos, Herbert, Russ, Dak, Kyler, Jalen. First class ticket. Those are so cool. Again, the Legitness is a new short print this year. A lot of big names on there. Tom Brady's in there. Nice, Mahomes. Some some really neat stuff coming out of this. I am very excited to rip this for y'all. Very, very excited. Let's see. I've been scooping up some teams in 12. Let's see, Saints, Jags in 12 and Saints in 11. Very nice. Colts Broncos gone in number two. Some more mats coming in here. Again, Matt is currently live over on the LSE Baseball Channel. If you all want to check out some baseball breaks, he's got one, two, three, four. He's only got five teams left total in all of his breaks today. He is ripping. The only thing he's ripping over there is Bowman Baseball Jumbo Eight Box case breaks. And again, in 13, he's got one. 14, he's got two. And then in 15 and 16, he's only got one team left. Good luck, Rodolfo. Thank you for hopping in. There is his link to his live. So let me update these spots here as a few came in down to three on the second one. So we've got three, 13, 12, 14, 15, 18, 18, 19, 20, 21, 20, 20, 21. All right, that is the updated look. So now for number two, we've got the Cardinals, Panthers, and Texans. Three to go on the next one. We'll get started on this first one in just a few minutes. Looks like Dolphin's gone in three. Thank you, Jeremy. Josh taking the Packers in number two. Good luck on the cheeseheads. Accidentally ordered a frozen coffee this morning. Instead of iced from Duncan, it's actually very good. All right, let me pull on my teams here. So we're ready to go again. 12 different 10 box half case break. Pick your team style and then any of the day was some Bowman baseball that has already sold out. You got Forrest tonight starting off at five. He's ripping baseball for the first couple hours. Bowman baseball jumbos. He's got Archive Six series, some Dynasty Baseball. And then for basketball, looks like he's got three, four box breaks of prison basketball, pick your team style, and then he's going to get into contenders optic for the rest of the night. He's got random team style, tiered random teams and random division. So that is what it's going to look like tonight with Forrest. He got Personals Resume after group breaks here on the main channel with Forrest. So he'll be taking care of you there. Appreciate all the support for sure. We had a ton. I know the team had a ton of fun over at SGC. That went so well. It was fun popping in and out of that, watching that. And just seeing the cards come back so quickly from grading. That was definitely, you know, an honor to be able to do that for y'all. Dolphin's gone in 12. Thank you Alvin. So really, really cool. Excited for y'all to get those cards here in the next few days. Get those shipped out, sorted. But yeah, that went fantastic. Forrest did an amazing job putting that on for y'all. And I definitely appreciate the SGC team for having us. And then be sure to check out if you don't follow us on Instagram. We did just post a little recap video from that. And at the end, let's see if I can fast forward. If I can pause it right here. There is a special that we that SGC is doing for Bowman Baseball right now. $9 grading, no minimum order size, no declared value limits. At a five to 10 day turnaround time. So definitely be sure to check that out for sure. I put on a great special for for Bowman Baseball for sure. But all right, we are ready to rock and roll. We are on break number one to start.


This is a very brief PowerPoint because most of today will be in NetLogo. I'll just be demonstrating things there, but I wanted to highlight a couple of concepts. So the concepts and planning of experiments, a little bit broadly about the behavior space, which is how you automatically run experiments in NetLogo, a little bit about output and then basically that's the end of the PowerPoint presentation. There will be time for Q&A at the end of the session as well. So the first thing I want to say is that working with models and experiments is iterative. So you write code in the code tab in NetLogo.  This is kind of running experiments. It's more like bug testing and things like that. But broadly, you're getting a sense of how the experiments will work when you switch back and forth between writing code and running the model to test it. But then when you get the model pretty much the way you want it, you proceed to a wide sweep and then onto maybe more writing code if your wide sweep shows you that things aren't the way you want back to a wide sweep or onto a narrow sweep and then probably back to a wide sweep, narrow sweep and eventually when you've gone through the loop several times onto experiments at which point probably you'll go back to writing code. And I'll clarify this a little bit more as we go through. So just to talk about today in the experiment section, I wanted to find a basic model with very few parameters to show you how experiments go with parameters. Although the tram commute model that we looked at last time and some of the other models that we've looked at, they're very interesting, but they have so many parameters that it can be hard to get your head around what experiments look like. So I thought just pick this one. It's in the models library. It's super easy. Only three parameters as the number of cars, acceleration rate, deceleration rate. And this model essentially it highlights one agent, which is in a red car. All the other cars are blue and it tracks the red cars speed, but also tracks, you know, the slowest speed, the max speed, all the, you know, the various speeds of the cars. And what we're trying to find in a model like this or in our experiments, other experiments might try to find other things, but what we want to find is the point, the break point in behavior. So the set of parameters or the combinations of parameters where the red car behaviors change. So if it's on its own, it can go max speed continuously. There's nothing in its way. There's no reason to slow down. But when there's other traffic, it has to accelerate and break. And so it's not consistently going the same speed. So we're trying to find the combination of parameters where that behavior changes. How much other traffic is too much other traffic? How much acceleration and deceleration are too much acceleration to deceleration, that kind of thing. So why just sweeps? Let me just briefly highlight here. You get a brief example of how to set up the sweeps. So what you do is you choose the variables. These are all the variables in your model that can vary. And you set them up depending on how you want them to vary. So you can either set them at specific points. In this case, 0.001, 0.003, et cetera, through 0.009. That's basically its full range. Let me just say, all parameters, you start wide with all the parameters varying. You want a full or wide range of parameters to vary. You want big increments. You want few repetitions. And the goal is to find errors that are not found in basic testing, broad areas of interesting behavior and conceptual problems. Generally, wide parameter sweeps on models that you've written yourself are likely to motivate some major code restrictions. I'm going to have to turn off this rest break in a second because it's come at exactly the wrong time for me. So hold on just a moment. And I'll skip that. OK. So these are all the parameters. We have them all varying across all of their ranges. You can either set them to vary at a fixed list of points or within a range according to a certain increment. And this gives you instructions about how to do that. So you can have by adding the extra brackets, you can say, beginning of the range, increment, end of the range. You can do this as much as you want. Also important to notice is the repetitions. This is how many times the model will run an experiment automatically at each combination of these parameter settings. And also important to note is what you are measuring. So this is the output of the model. And you can choose to measure the runs at every time step. So it puts this thing that you're measuring into the output at every time step. But if it's unchecked, you only get the final time step. So that's very different depending on what kind of model you're producing. So you might only want the end. You set up this model. You let it run. Where does it end? That's all you care about. Other models, you set it up. You care about how it gets there. And generally you set a time limit. If there's a natural way that the model ends, like maybe when all the agents die or all the agents get infected and recover from a virus or all the agents reach a certain level of health points or something like that, the model might end naturally according to your code. If not, it's very important you set a time limit. OK. So that's a wide sweep. Now a narrow sweep has maybe not all of the parameters. Maybe you might decide, right, I don't care about acceleration. I'm only interested in the number of cards in deceleration. It's narrower ranges and smaller increments. So instead of the entire range of number of cars, I'm only looking at about half the range. And I'm moving in smaller increments. And there's more repetitions. So instead of 10 repetitions, I now have 50. And the goal of this is not necessarily to find errors. You probably will have found all the errors in the code run sort of switching or in the broad sweep. But you are really looking to clearly find the break points where the behavior changes or to find the ranges at which different runs break the break point and behavior. There is a chance that you will need some code rewrites. Probably they will not be major rewrites. They will be detailed rewrites. Finally, if you've gone through broad sweeps, if you've gone through narrow sweeps, you've done all the code rewriting that you're going to do. Essentially, experiments are really very narrow sweeps, you might say. There are many, many repetitions on a single or very few settings. So this one, the increments is even narrower. The list of different settings is much smaller. And the repetitions is much bigger. And the goal here really is to find statistically useful volumes of data on areas that the sweeps suggested were important. So the sweeps probably tell you that the very end ranges of your experiment are always, you know, the red car always goes max speed, no problems. Or the red car never goes max speed, and he's always stuck in traffic. So really you're trying to find the area in the middle where things aren't always consistently max speed or never max speed. And that's why you set the experiments to narrow in on the bits that are interesting. Now let me just switch over to the model real quick. Here we go. I'm going to set it up. Well, I'll tell you briefly. In the tools menu at the top is behavior space. You can also hit control shift B to open behavior space. This is where you set your experiments to run automatically. So I have these three experiments that I set, wide sweep, narrow sweep and experiments. And you'll notice they have the number of individual model runs that this set of, you know, that the sweep or the experiments contains. So the wide sweep, I'll just open this so you can see, has this wide range, big increments, 10 repetitions. And that comes out to 1,250 runs. Now the narrow sweep has fewer settings, you know, smaller increments, but many more repetitions. So it comes out to 3,200 runs. The experiments, even narrower ranges, even smaller increments, but many, many more repetitions. So it comes out to 4,500 runs. Now this is not impossible to do with Excel, but I really think I really encourage you to consider using R or Python or Julia or some kind of, or even Stato or SPSS to analyze your results because they are potentially much, much larger than this and they will break Excel and you will go crazy trying to get it to work. Now, if I wanted to set up even more narrow experiments, I could hit duplicate and let's say I just want to look at when there are 20 cars and 22 cars. So I've changed from a range with the increments to a fixed list. And let's say I just want one acceleration and two decelerations. Now I'll keep the repetitions at 100, you can move it to 200 or 500 or whatever and I'll call this experiments focus. Why am I typing focus? Today is not my day, folks. And hit OK, keeping everything else the same. So now it's got 400 runs because I didn't increase the increments and I narrowed down the number of experiments I was running. Do we have any questions so far? I don't see any in the Q&A. So that's fine. I'll just pause for a minute here and let you all, if you have your NetLogo open, you can open this model. I loaded this model with these experiments, the wide sweep, narrow sweep and experiments, the first one. I loaded to the GitHub repo. So if you want to see exactly what I've created, you can pull these changes from the GitHub repo. Alternatively, you can just go to file models library. Well, I'll cancel this. But within models library, it's called traffic basic. And you can recreate these experiments within behavior space, which remember is at the bottom, near the bottom of the tools menu. So what happens? I'll move back to the PowerPoint, I think. That's not what I wanted. I wanted the full. There we go. So those, that's what I have in my behavior space. You'll notice run that actually sets the experiments to run. Let me, there we go. But you will need to choose an output. Now it will give you, when you hit run, it will give you the option of a spreadsheet output, a table output. You can update the view. This is the view. And you can update plots of monitors, which is in the actual interface.  Simultaneous runs in parallel. This is good if you don't have a creaky old laptop and you can run lots of things at the same time. If you're concerned about computer power, just run one or two at a time. The difference between spreadsheet and table output is the orientation. So spreadsheet has the variable names going down in the first column and the output of the different runs going across in the y direction. Whereas table output has the variable names going across and the results going down. I find table output more natural to work with. But maybe you find spreadsheet and that's fine. And this is a little bit of when you hit OK, it'll ask you where you want to save it and what you want to save the output as. But when you pick a place to save it and a name to save it under, it starts running. This got to run 188 of 3200. And I think I ran all of the 3200 experiments in about 60 or 70 seconds because this model is very, very straightforward, very small and trivial. It runs fast. You can of course, where's my cursor gone? Here it is. If you have to, if things are going terribly, terribly wrong, you can just chuck the whole thing and start over. You can also, I mean, it's not very important for this model because this model is so small, you can change the speed. This kind of only matters if you're updating the view. It doesn't really matter for most things. That's pretty much the end of the PowerPoint. So let me get back to traffic. And do we have any questions? Let me just take a moment if anyone has questions. Okay. So I'll go ahead and hit run just to show you what this looks like. Gives you this little window. I've chosen table output because I think it makes more sense than spreadsheet. Hit OK. Gives me where do I want to store this? This is the folder that I have. I'll copy this up to the GitHub repo, but this is the folder where I'm storing my local copy of that repo. I'll hit save. And you see these 400 runs are done already. That's very quick. So what does it look like in that? Let me just see if I can open. Now, I don't want to open that. Let's just go with PowerPoint. I want Excel. I'll go with one of these other runs that I had already exported. It's taken a while to open. That's all right. Let me move it over. Excel. My nemesis, I feel. It's refusing to move the window. There we go. Okay. So let's make this big. All right. Let's decide to open a bunch. Okay. So you'll notice in the automatic export from behavior space, it gives you six lines of stuff about the model. So it was a metadata, basically, sort of like, you know, what was the version of NetLogo that ran this model? What's the model called? What's the experiment called? When was it run? What was the world? What are the dimensions of the world that were used in this model? Now, if you're analyzing this with R or SPSS or Python or something like that, you probably don't need these rows. You can delete them and re-save the file as something else. So I will delete these. I want to actually delete them, not just delete the contents and save as traffic, where is it? Traffic-wide tidy. That's what I'll call it. And it's going to tell me that it already exists. Why aren't you? Okay. Let's try that. I want to replace it. I basically replaced it with itself. That's how I created traffic-wide tidy. And let me just open our studio, because this will show you a little bit about how we can work with that. Now, you remember if you saw my previous workshop, the one about adding in real-world data, we talked about using the CSV extension. And we looked a little bit at the code for how to get the model code to export things, maybe not just at the end or on every turn, but when a certain thing has been accomplished within the model, you can ask agents to open a file, write something and then close the file. And that's very useful if you want more complex, more interesting sort of model-specific data, rather than just what is happening in basic parameters, like global parameters at this point in the model or at the end. So here's my art studio. It's taken its time to open, because that's basically how it goes. Let's see. Let's close PowerPoint, because I think I'm done with it. Close all windows. Wants to know if I want to keep my ink annotations. No, don't be silly. Okay. Let's close that. Why is this taking so long? Clearly I should have practiced a bit. So I've created a little bit of code, and this art code is in the GitHub repo if you want to use it. Basically, I install a few packages and make them available. I actually don't need the read Excel. That's not helpful. So you want to get your working directory and set it, if necessary, to the directory in which your code, your output is working. Read in this case, wide. Let's clear out the old variables. Yes. Why didn't you clear them? Well, it'll get around to it. I'm guessing it's just slow, because everything is slow right now. So you want to read in the variables. Is it not? Yeah. It seems to be working. Yeah. Slowly. Slowly. Feel free to ask questions at any point. You don't have to wait until the end. I am ready and able to clarify anything if I'm going to quickly. If you want another example of something that I've shown here, could not find function as tipple. You know why? Because I didn't install the packages. So install those first. Always good. This may take a little while. Sometimes package installation is a bit slow. Again, I could have done this before the workshop started. But then you wouldn't see the genuine silliness that is like working it with R. So we've got these libraries. That's good. Let's just get my working directory to make sure. I know where I am. Yeah, it's fine. Cannot open file. No such file or directory. Okay. Let's see what's going on with that. It's always a problem, isn't it? That's modeling. 22 training. Come on. Oh, it's not showing my... Let's try this again. Get working base modeling. 22. There we go. Traffic model. Okay. So traffic underscore wide underscore tidy should be there. Maybe it was just because I had it open in Excel Oops. Control enter runs the highlighted code. Nope. Cannot open the connection. I have no idea. Today is really not my day. Oh, right. I see. Because I want... I will change my working directory to source file location. Let's try that to see if the working directory is correct now. Yeah. Now it's in traffic model experiments. So the problem was despite having hit get working directory, I did not pay proper attention to what it said. Note to self, pay attention to self. It's time. Try it again. Run the code to read in the CSV as a table called wild. Wide. Sorry. We see over here it is quite a lot of observations of six variables. And it will view coming up here in a second. So the, you know, it's sort of automatically coming in from CSV. It adds this X dot thing. It puts dots in the middle of the names, but they are run number, acceleration, number of cars, deceleration, time step, and the speed of the sample car, the red car. So that is all the things I need, but you know what? Those names are a bit fussy. So I'm going to rename the variables to simply run acceleration, number cars, deceleration, step, and red speed. So now they are renamed. Great. Now briefly, I want to just do a little bit of summarizing. And this is going to find the average of the 10 runs in which all the other variables were held the same. So if you remember from our wide sweep, we had 10 runs and there's 10 repetitions at each combination of these settings. And what this code does is it finds the average of those 10 and stores that in a new table called wide summary. I think it's a table. So if I run this briefly, it finds there are now 10%, it's only 10% of the size, so it's dropped a digit essentially because it's fine for every combination of 10 runs where all the variables are similar. It's only found the average. And briefly, I'll just plot this.  I wouldn't recommend that you try and get a paper published with a plot like this. But you can see sort of, so this is when there is one car. And here is the acceleration. So when the car can accelerate very quickly and there is only one car, the red car very quickly gets up to max speed and then just stays there. But the lowest acceleration, the car never makes it to max speed. So we can eliminate one car as like an interesting combination of parameters. Now with 11 cars, again, low acceleration never gets up to max speed. In fact, if we look, even the low acceleration never helps anything. So this is why we drop low acceleration in our narrow sweeps and experiments. High acceleration, still pretty good. I mean, right up through sort of medium acceleration. That's not bad. But what's really interesting is when the traffic is big, when there's 41 cars in the model, the actual speed is hugely variable. So while acceleration was the only thing that matters when there's a small number of cars, even, you know, 11 broadly up to 21, like as soon as you get a lot of cars, there's other things are driving behavior differences. So some runs, the red car goes very fast. Some runs, the red car goes very slow. So we're kind of trying to zone in on that. And what we can do is replicate these same basic steps but with traffic narrow tidy. So let's call it narrow and then traffic narrow tidy. If we run this, we'll see another variable here. Eventually it's taken its time. It does not seem to have worked. Oh well. And then when it arrives, we will rename its variables. Right. It seems I heard not. Oh, right. The problem is I did not save a version without all of the headers and stuff. Now I could say do this in R. I could remove those rows, but that would make importing it more difficult. So this is another reason to do the CSV extension, the kind of detailed exports that I did in, yes, the previous workshop where I showed how to build reporting into the model operation is that the exports, the export file in a CSV extension like when the code properly, I've just renamed wide. Oh well. We'll go back, rename this. Sorry, trying to do too many things at once. When you properly export, say PaaS, when you try and export things properly through code within the model, you don't get these headers with the version of the model software with the dimensions of the world. You can if you want, of course, but you have to set that up. Whereas the export that comes automatically within the behavior space really wants the, it always puts that stuff on and I never want it. Like I find you can always just make that part of the file name or something like that if you want. I don't know. Or it would be more useful as a column of in the spreadsheet rather than like a heading that I don't know. I don't particularly like it, but what can you do? So let's change these things. And now we see here, we did finally get it imported. It is a huge number of observations because there are now 50 times 50 repetitions at each combination. So when we find the narrow summary, it's going to take the volume right down. And it does narrow summary is now the same number of observations we have here. So it's much, much smaller. Oh, that's because it's, let's try this again. That's still going to be small, but probably not the exact same size. Yeah, it is now even smaller. Okay, so that's very good. So now let's plot this and see what this smaller set of experiments looks like. Here we go. All right, we see much more interesting behavior here because we have a smaller number of, you know, car numbers to compare. And we see that the behavior is much more streamlined. It's especially streamlined when there are fewer cars, but 25 and 30 are very interesting, aren't they? Because we get good ranges of behavior. So there's still chaos. There's still individual circumstances within each of these experiments. But there are, I'll just post upon this, but there are interesting behaviors. So 25 is still, okay, you know, we'll see how it goes. Certainly the highest acceleration 0.008 and 0.07 reach max speed, but, you know, lower accelerations 0.006 and 5 don't. But when there's 30 cars, nobody reaches max speed really. Maybe some higher accelerations. And the ranges are very big. So let me save this code. Do we have any questions? I feel like I'm moving quite quickly. And I am surprised that no one wants clarification on any of the things that I've covered. Within NetLogo, within are about the logic of experimentation.  None of this is shocking to you and you just need an example. But I'm surprised. So please do feel free to ask questions. Okay. And again, we can repeat these same steps for experiment. But let's have a look at that experiment and think about is actually this where we want it to do. Now, our model suggests between 20 and 25. There's maybe not so much interesting. We think might be interesting is between 25 and 30. There's a big change here. But maybe we could go slightly beyond that. So let's go with instead of 18 to 30, let's go with maybe 26, 28, 30 and 32. And what about the acceleration? 0.05 is certainly, you know, it's not that interesting. 0.06 is just about interesting. So let's say, yeah, 6, 6, 5, 7. This is pretty good. 100 repetitions. That's it. That's because I forgot to take away these brackets. If you want a range, you need an extra set of square brackets. If you want a list of individual points to consider, you just put in the list. So it's now 3,600 runs. Let's run that table output. Okay. Yep. Let's just call it experiments. Save that. It will go pretty quickly, I think. Yep. It's moving through this very well. I don't think I'll have time to finish this micro break. Maybe I will. Let's see. Total elapsed time so far, 22 seconds. Let's see how far it gets. I think it'll finish. Yeah. Ooh, it's a bit of a race. Will the runs finish before my micro break? They did, yes. But I'll wait three seconds more. Thank you very much. I will go back to my folder. Let's see. 1602. That's today. No. Yeah. Is this the one I just ran? I don't know. We'll see. 13. This is not the same one. Let's try again. Excel my nemesis. Okay. So let's delete these. Delete. Save as traffic underscore experiments. We need to stop this so that our can open it. And let's experiment. Underscore one. All of these things in. Place. And we can rerun all of this code. To see what it looks like. Now again, no questions. Please do feel free to ask, you know, why am I doing this? Why have I chosen the parameters that I mean, what do I mean by break points and critical area? These are pretty important topics. And if you're not clear on them, please feel free to ask. You can of course ask anonymously. So nobody needs to know that you're the one who's confused. If that's what you're worried about. We name that. Find the summary. Now. You'll see the original that I imported had. 643,200 observations. But after summarizing it. After averaging all of the means. We get one, two, eight, six, four. Now you might think, why is that bridging the means important? Because that's, you know, that's. Removing interesting variation possibly. This is just a demonstration. We don't, you can of course. Focus in on these differently. So again, 15, 20, this is definitely not the setup that I had just exported. No idea where that went. So let's try it again. Hermann's tidy. Save. Yes. So it's going to run again. It's time. Probably again take around 30 seconds, which is very fast. Well, what one thing that's useful about all these mistakes that I've made and problems that I've had with files and exports and all this is that it's, it's really demonstrating how hugely fast it is to run thousands of Asian based modeling experiments. How easy it is to get the data that you want. How easy it is to change the experiments and rerun them. So, you know, we can really very, very quickly do a lot of work. The slow part is absolutely. Yeah. Setting up the model functioning. If you're writing a new model, that is a slow process. I mean, it doesn't have to be very slow if you're good at writing models, but getting it correct is typically very, very slow. Whereas the experimentation is super fast. Delete these. Save that. Close this again. Go back to our rerun these things. And we'll see. See what we get. Okay. Again, just, just not seem, I have no idea where. I don't know why it's not changing. It's tidy. It's just now. Oh, I see. This was my problem. What did I call this experiments? So, there you go. Everybody makes mistakes. Sometimes quite goofy, silly, obvious mistakes. The problem is now that I have misspelled experiment. Now, if you're not interested in using our, of course, you're not obligated. I, I like it because it makes these nice graphs quite straightforwardly. You can use anything else you want instead. All right. So here we see an interesting down slope at the beginning. It's not something we've seen previously. Go back to previous graphs. Well, there's a little bit of it down here. But if we go back a bit further, come on. Valid plot index here. The very tiny dip, but mostly, you know, with, with very small numbers, it's almost instant acceleration. Whereas in the later graphs, we do get these dips. What's going on? Why are there, why are people slowing down instantly at first? And they're spending quite a lot of time accelerating after that because they, they dropped so slow. So it's interesting. It's, you know, they're reaching pretty much zero speed in all of these quite quickly. It's an interesting one. And again, much more variation when there are higher numbers of cars. So it doesn't necessarily answer all of the questions. You know, like if our question was how many cars can we get on here that flow smoothly, we would have to revise because all of this suggests that even as many as 26, you know, that's too many that we're, they're not flowing smoothly because they're, they're dipping pretty much instantly. But maybe we assume there's, there's, you know, set it at 26, for example, cancel. Why is this so difficult sometimes? Okay. Let's set acceleration at 60. Deceleration, let's increase that just appropriately. And run this and see, see what's going on. Why, why is it? Okay. So let's look and we see, yeah, there's lots of start and stop. You know, the max speed is, they're never reaching the max speed. There's lots of starts and stops. Maybe this is just too many cars. So if we drop it down to 15, same, same settings otherwise. It is still starting and stopping, but much less so. Hmm. So if we, you know, we can just keep trying. You can try exploring. You can run experiments to see here what it looks like at the settings that your sort of analysis and averages suggest are interesting. There's a lot of different things you can do to find out the logic of what's going on. Now, if anyone wants to suggest another model that I can, you know, look at and maybe run some experiments on. That's fine. We can, we can look at other things. I can show you again how to set up more experiments. Maybe you're interested in knowing more detail about how to set up ranges.  I can, for example, by open, go ahead and save those changes. Oh, no, that's not what I wanted. I can show you the, let's see. If I go up, I can look at the GIS commute model, you know, and we can, we can look at this, how the CSV export works in that and why it's different to analyze that. Okay. Let's look at the file exports. Okay. So in this model, whenever one of the little agents arrives at a destination, it, and the model is set to export data. It opens this CSV file, which is created based on an input in the interface. So this is the CSV file.  And then it would print its own name, a comma, it's starting place, a comma, the local authority. Of that starting place, a comma, its destination, a comma, the local authority of that destination, a comma, and how long it took to reach that destination from its starting place. And then it would close that file. So in this case, the output will not have those headers about, you know, what, what version of net logo this was, what the dimensions of the world were, the date and time that it was created. I mean, you could keep those if you want, you would edit this, the file print to include those features if you wanted it. So it would need more commas and more, you know, variables. And it would include those in the file export, which might be useful to you. It might not. You could also put them in the name of the output file if you wanted to rather than just the input code from the interface, you could put code, the name input into the interface, plus the date, plus the version of net logo, plus, I don't know, something else, whatever you wanted. And that's why these files were created. And that's why these files are different in structure. Yeah. So these, these are different kinds of output you can get from your model, depending on what you think is important.  Notice here, there's no experiments. You'll have to hit new. It gives you all of the variables, all the things that can vary. You do not have to set them all. So you could set export data to true. Oh, it would help if I could spell true. If I could get a list of false number places could be 50 and 100. And that's it. Look, label trim stops false. That's number generated trim stops only. You know, if this only matters if the projection is set to random. So I could set the projection to both random. And I would have to set it to GM. I think it's GM Alice. So I would need quotes around both of them, but it would just list them here. And if it's using random, I could set it to 20 and 25. For example, chicken sandwich. I could also set this to, you know, you know, so it would run. This is an interesting one. In this case, you probably, it's not that helpful to have variables here, because it will run all of the other things with all of the repetitions for each output file. So you will get two completely different sets of output files. Some of them called chicken sandwich. Some called verbal, but they will have all the same combinations of statistics. And, you know, all these other things you can set as you like. Repetitions, you might want to increase. Now, when you create a new experiment, it sets it automatically to be count turtles. This is basically to show you what reporters, how you enter reporters in here. But you could set that to other things. You could set it to, you know, average speed of turtles, or like we saw in the red car. It was the current speed of red car, because it was measuring runs at every step. You could set, yeah, it depends on how your model is built. And then obviously you want to set a time limit. Maybe you want more time in this because it's a bigger model. And ultimately, if you know that you have export data set to true, and that's really the interesting sort of data output that you want, you don't have to use the behavior space automatic commands at any, at all. You can remove measure runs at every step. You can hit OK. And when you hit run, you just uncheck both of these. So you won't get any automatic output. You will only get the export data output that you create in the model code. I'm going to cancel that because I don't actually want to run an experiment with those things. All right. So we are into the Q&A session portion of today's workshop. Please do ask me any questions about running experiments, parameter sweeps, how you choose parameters, all of that. You can, in fact, also ask me anything else about agent-based modeling because so far we don't have any questions on experiments. And so you are welcome to ask me anything else about the upcoming workshop maybe, or how I got started in agent-based modeling, or how useful it is to create different kinds of export data commands, what kind of useful things happen on there. You can ask me anything you like. I am here to help. We have someone who's raised their hand. If you're OK with that, then with Sorsha making you audible, you can just ask your question. Sorsha, if you want to make Peter unmuted. Yeah, we'll do. OK. I'm going to share my screen again because I assume that Peter will ask a question that it's useful to have an example for. We've also got a question come in on the chat, so it's fine. Peter, ask yours first. OK, can you hear me? I can, yes. OK, thanks. Well, thanks very much for that presentation, which I'm really, really valuable. I didn't ask questions before because I've been using that logo for some time. I got kind of stuck with the time it was taking to run the models and got something to run it in Python and then on the supercomputers at Durham University. And so my question is this, that my main focus has really been parameterization, which seems to have taken an earthly amount of time. And we're essentially using experiments to improve the parameterization. And I wonder, I've got my terminology mixed up here. Sure. Running what you're doing, but not to see what the outcomes are in order to improve the model. Is that the right way of thinking? I mean, early on in the PowerPoint, I had that sort of flow chart with lots of iterative loops because you absolutely do have to run a thing and then see what it is and then run it again maybe with a narrower focus or a different focus if your first look turned up nothing interesting. And yeah, if you're going to, if you're getting absolutely massive quantities of data, it does make sense to book time on big supercomputers and get them to run it because as I said, when I hit run from experiments, it gave me the option to run experiments in parallel and that works fine to a certain extent on most people's computers, but it will work a whole lot better on a supercomputer. But yeah, you, I mean, we all go into experiments thinking we know broadly what we're going to get. It was certainly we have, if we're trying to recreate a real world situation, for example, traffic jams in this place or, you know, in our rate of some of this that or the other, if we're trying to model pandemics, we'll try and play around with the settings until we can replicate the things we see in the real world. So in that case, you kind of have something to aim for and you can set up broadly like the full range and say, let's look at the output, which is close, which settings get us closest to the real world thing that we think and then we'll narrow in. And you can obviously do things with like, you know, Monte Carlo sampling and different kinds of things like that. And if you're trying to get big statistical pictures of lots of variables when they're all changing, it can be quite difficult. I mean, that's not a feature that's true only of agent-based modeling, like sampling complex sets of data with lots of, you know, variables is a problem for any statistical analysis. The benefit of agent-based modeling is that you can increase the number of repetitions so that every combination of variables is statistically relevant. So that helps a little bit. In fact, that touches on the next question, that question that someone has asked in the Q&A, which is, how do you change the number of runs? And so I'll just show that one more time. Did I answer your question, Peter, or do you want to, like, repeat it again? No, that's fine. That's good. Very reassuring. Yeah, absolutely. You kind of just kind of, like, the wide parameter sweeps are just thrown things at the wall and see if they stick. And then, like, you get to, you know, whatever is closest. Okay, so if you want to change the number of runs, you edit either when in the, you can edit an existing or you can create a new or duplicate set of experiments. Give it a name. So you might call it experiments. Underscore two. Well, that's bar two. That's not what I want. Okay, there we go. Repetition this here below the variables. That's what you change. If you want to change how many times the behavior space automatically runs an experiment with this combination of features. So it could be 10,000, 10,000. I mean, it kind of depends on how many combinations settings you have and how good your computational resources are. In theory, more experiment, more repetitions is better statistically anyway. But, you know, certainly early on in, in your narrowing in on the interesting critical spaces, 30 at least, like 30 is statistically relevant. Yeah, and that's a good number to do statistics. So start there. I believe I hope that's answered your question. If anyone else has questions coming in. Yeah, I'm happy to demonstrate other things because I, yeah, I really did whiz through a lot of that. So I don't blame you at all. If you missed an explanation that I gave or, you know, an example that I could have written another way. I mean, maybe everyone just wants to leave early, which you are also allowed to do. You're not obligated to stay to the end. I am, but you're not. But you can ask questions about models, about exports, about experiments, about code, you know, about other kinds of modeling platforms. I can't guarantee that I have very useful answers for that because mostly I use NetLogo. But yeah, you can, we can talk about it if you like. The, yeah, the, to say a little bit more about sort of experimenting, it's not, it's not easy to know exactly where to narrow in, especially if you're, if your model is quite like fact-free. If it's not trying to replicate a real-world scenario very well or if it's not trying to explore a really specific kind of case, if you're just sort of trying to figure out like, what would happen if I did this? Then it's, it's not easy to know what interesting behavior to look for in your statistical analysis. So it's not easy to identify interesting features of the results that suggest you should look closer. But yeah, if, if you are trying to replicate a real-world situation, it's much easier to, to sort of fine-tune your parameters because you can first start out with finding, fine-tuning the parameters to represent genuinely what, what you think the real world is like.  On the other hand, if you're trying to see what would happen if 15,000 people were to suddenly start using the tram every day, yeah, you can, you can put that in as a parameter setting. It kind of depends on your research questions. I mean, I can, I can open other models if we want to look at those. I can certainly look at code examples like CSB extension. I mean, this is the one that we, that we looked at with the sort of writing to file, the little agents that were placed randomly. GIS extension, if you want to look at mapping. Lots of other things though, if you want, I really recommend code examples if you want to learn how to, for example, get your agents to move in a circular path. Or, you know, like create fully connected networks. You know, there's a file output example. So that's, that's another example showing file print. Okay. That's probably useful to look at if you want to give the standard basic behavior space tables and spreadsheets and go with something that's more specifically useful to your model. Yeah. If you want hexagonal cells instead of square patches. That's good stuff in there. I really, I think it's very useful the way that they've organized the code, the models library with the code examples. I think there's a lot of good, good examples of how you can get, you know, turtles to occupy a space and not let another turtle occupy the same space, for example, if you're trying to show how people move through a room to take their seats in a, in a classroom, for example, you want turtles to not walk over the top of each other because that's not what people do. You want turtles to find a place and sit down and occupy it. Arguably, if you were being realistic, you'd want turtles to find a place, sit down, occupy it, put their bag on one other place and their jacket on a third place and then occupy all of those until someone comes along and gives them the eye. So, you know, even this is not super example, super realistic example, but it's better than, you know, turtles who walk over the top of each other. Yeah. There's a lot of good network examples as well. There's not those random walks. Yeah. There are no further questions. I'm astonished. I'm astonished there are no questions.  But this is how it goes, I guess. Maybe I've completely overwhelmed you. But there's still 15 minutes left. So I'll give it a few minutes. If no questions come in in the next, like, 15 minutes, then I'll just say goodbye and we'll end this slightly early. But please do feel free to ask questions about this model, about other models, about exports, about experiments, about writing code in R to model to, you know, to have a look at your results. Let me know. I'm here for you. So let me know if there's other things about agent-based modeling that you'd like workshops on, because we can, of course, have more workshops in the future about, you know, if you're interested in building, I don't know, economic models or supply chain models or, I don't know, travel models. Like there's all kinds of interesting behaviors that people might want to look at. And we can, you know, maybe take a broad kind of question and try and build a model together about how to approach it. Julia? Yep. We've got a question that's come in from YouTube. All right. So I have heard that Julia is very good at agent modeling. Have you tried it? Thanks. Is that a program or? Agent model. I mean, there's agent spring. Yeah. Because this next question is, how is it better or worse than net logo? Okay. I would need more clarity about exactly which model that is. But I would direct people to my, the first sort of webinar in the series. We'll just stop sharing. So my face is big. In the first one I did, which I believe was January 19th, the video is available on YouTube. The slides are available in either our GitHub repo or on our UK DS past events pages. In that slide deck, I cover several different kinds of modeling software and some of the pros and cons of each of them. So I would say that the best thing that I could do for a repast, I think agent spring, is another one. Doing agent based modeling directly in Python. I mean, because it's an object oriented programming language. So it is logically compatible with agent based modeling, but it is obviously a very broad tool. And so it doesn't have things like an automatic visualizer. It doesn't have a behavior space to automatically run your code multiple times at different combinations. It's powerful, but it's not user friendly.  It has a lot of built in language that makes it very easy to develop a concept from nothing to functional and experiment ready agent based model quickly. So the downsides to NetLogo and the reason why another language might be better is that NetLogo sort of maxes out if you have a complex model. So when we try and run this tram commute model with the real world GIS and real world census data, that's tens of thousands of people with, I think, 98 tram stops or something like that. And it's absolutely at the limits of what NetLogo on my own laptop can do. NetLogo is obviously much stronger if you want to run it on a high powered computer through your university or some kind of remote server allocation. But ultimately even then NetLogo will max out a bit faster than some other languages. So if nothing else, I would say start in NetLogo, prototype your concept, make sure it works logically on the cheap, easy, fast NetLogo, and then rewrite it properly in another more advanced language after you've already worked out the basic logical bugs in NetLogo. Because more advanced agent based modeling languages like RePast or Mason or Agent Spring are very, very strong and operate under the same basic logical principles as NetLogo, but they are not easy to learn and they are not user friendly interfaces, at least not in the same way, not as user friendly. So that would be my advice there is that start with NetLogo, get a working prototype, move it somewhere else if NetLogo maxes out on you. I promise I'm not being paid by NetLogo. The person who asked the question has added, I mean agents.jl which is a Julia framework for agent based modeling. I haven't used it. Actually some members of my team are looking into learning Julia as a programming language. I would assume that like Python, because it's an object oriented programming language, it is totally compatible with agent based modeling, but that the interface will be not user friendly and that it will be much more powerful, much more effective, but that it will be harder to get started, especially if you are not already have like a logically functioning prototype. So that's very good. I can add that to my slides for the next time I run this agent based modeling series. I'll add that to the intro slide on, here are some options for agent based modeling and I'll put in there that people are asking about this. This is cool. This is interesting. I'm sure it's very powerful and very good, but I suspect it is not user friendly. I suspect it wouldn't, for example, have this sort of like automatic visualization where you can see the little agents moving around and that's very, very useful for bug testing, for like prototyping, for getting your head around like what has the code that I wrote actually done. But maybe I'm just a very visual person and someone who's a whiz with programming would find this a big distraction and waste of time. I don't know. Maybe. I do think, yeah. If you're interested, like contact me on Twitter. If you want. Jay has my complex. So it's just my first initial, my surname and the word complex all ran together and there are no spaces. You can, if you're interested to know more, I'll have a look at it and give you my impressions. And we can maybe have a wider discussion about agent based modeling in object oriented programming languages and pros and cons. Yeah. Sorry, I don't have any direct experience with it. Yeah. That looks like it. So thank you very much everybody for attending this workshop. Please do contact me via email or Twitter if you want to follow up on anything, if you want to start discussions, if you end up using agent based modeling in your research and you want to tell me how it worked or, you know, like that we're always happy to host, you know, to sort of platform or to share the results. If you publish a paper using agent based modeling, we're all cheerfully retweet that. Let me know. Otherwise, I will let you go and have a good one. It's been lovely.


15th Marine Expeditionary Unit Concludes Support to Operation Inherent Resolve Saturday, April 3, 2021. Indian Ocean, 15th Marine Expeditionary Unit, MIL, embarked on the Mackinac Island Amphibious Group, ARG, concluded operations in support of Operation Inherent Resolve, OYER, March 18. US Marine Corps, F-35B Lightning II Joint Strike Fighters with Marine Medium-Tilt Rotor Squadron 164, reinforced, 15th Marine Expeditionary Unit, conducted a total of nine Operation Inherent Resolve missions as part of Border Counterterrorism operations in the Central Command Area of Responsibility. During the missions, the fighter aircraft departed from the amphibious assault ship USS Mackinac Island, LHD-8, and conducted long-range strikes and counter-OYER missions in Iraq and Syria. Executing missions in and outside the Arabian Gulf shows just how versatile and lethal the F-35B platform really is, said Lieutenant Colonel Christopher Kelly, VMM-164, RAIN, Executive Officer. Embarked aboard the Mackinac Island ARG, our support to the OYER mission shows the depth and responsiveness of the MOE as a maritime force, one that can be called upon when needed. Amphibious groups with Embarked Marine Expeditionary Units provide combatant commanders with a responsive, flexible and forward-deployed asset capable of maritime power projection, contingency operations and crisis response, shaping the operational environment to protect the United States and allied interests in any threat environment. The Mackinac Island ARG and Embarked 15th Marine Expeditionary Unit have been deployed since January in the US Fifth Fleet Area of Operations, OE, supporting security and stability in cooperation with regional and coalition partners. The Fifth Fleet OE encompasses about 2.5 million square miles of water area and includes the Arabian Gulf, Sea of Amman, Red Sea and parts of the Indian Ocean. The expanse is comprised of 20 countries and includes three choke points, critical to the free flow of global commerce.


Hi y'all. Oh, we got extra water. Nice. All right. Goddess, I'm so excited that you're here today because we're talking about brands being bold and you are one of the boldest people I know, particularly in your career trajectory over the last like two to three years. So before we get into brands, I want to talk about being bold in your career and maybe for starters, you can just let us know how did you originally get into advertising and social media? What attracted you to that kind of work out of all the things you could have done? Oh my goodness. I like to say I tripped and fell and landed in it. I think it's interesting. I'm still kind of of that old school class where when I was in school, social media didn't exist, telling my age a little bit, hopefully I still look good, y'all. But I think it was really interesting. I wanted to be a journalist. I wanted to work for Essence Magazine. I always knew that what was important to me, yeah, I wanted to be Susan Taylor. I thought that that would be my path and what I loved the most was writing and coming out of school, again, being of that kind of strange group that came out of school in the middle of a recession, we all kind of had to be really creative and figure out what the heck are we going to do now. And I was like, oh, I know how to do this right. And then that can take you places. So I started as a marketing coordinator at a startup and then that kind of lended itself to who's the youngest person in the office? Turn that Facebook thing on. Turn that Twitter thing on. I think a few of us probably ended up in social media just because of that. And that's really kind of how I got there. But what kept my interest was the idea that I could connect with people from all over the world. That was so exhilarating to me and it still is to this day. So I'm happy that I tripped and landed here. So fast forward 2016, which was a really pivotal year for you. You're working as a social strategist for VML, YNR, the agency. And you sent an email to your manager that literally set off a chain of events that brought you to this chair. So can you tell me a bit about what was happening at that time and what that email was? It's so crazy just to be here today and think about that day three years ago. Being in advertising, what I loved about it is this idea that you get to connect with people, right? You get to represent people and culture. And for me, being in a very, I'll just say it, white male dominated industry at the time VML itself was very not as diverse as I know they would want to be. I just didn't feel seen. And that particular day I actually sent an email to the head of our New York office. It was the day, I think it was the end of the week that culminated the week that Philando Castile was murdered, Alton Sterling was murdered. And there had just been a string of murders and police brutality. That just wasn't sitting right with me. It felt like my community as a black woman, I was under attack. And to go into an advertising agency where our job is to know how to connect with human beings, what makes them tick, what is important to them. It just felt like nobody was seeing me. And I kind of just had a moment I said, I'm going to send this email. If I get fired, you know, at least this is what was important to me. And I sent that to the head of our New York, our New York office, which is Mike Wenty and Chris Verst at the time. And I heard later that it made it all the way up to our CEO and HR and there's a little bit of panic in there. But, you know, I asked, could we just talk about this? And how could we call ourselves, you know, representing culture and connecting with human beings? And we can't even talk about the things that our people are going through. And I got a great response. And we had a meeting just to talk about what was happening and what we felt we could do about it. Yeah. And something that's been a theme that I think has come up in a couple of sessions today is how difficult it can be to speak up. You know, there's a fear of retaliation. You mentioned it. What gave you the courage to send a note like that? Did you have any reason to believe it would turn out well? You know, and we were literally just talking about this backstage. But I think, you know, there's a little bit of privilege in saying, well, I'm going to send it. And if I get fired, you know, I have somebody to catch me. My amazing mom is here. My husband's great. You know, I think there's a privilege in saying, I can make this decision and be OK with whatever happens. So I recognize that. But I also think that there's something to this idea of having to follow and be true to who you really are. And I was really just at this point where if I couldn't be at a place where I worked that saw me fully, then what I was doing was kind of a lie, right? Yeah. How am I really connecting with people around the world? And I'm not even been seen at in my office where we create these campaigns and these these ways to talk to different people. So it just felt like to me it was almost a turning point for am I really going to do this? Is this what we say we're really doing or not? And it was just important to me. I just felt like I think if things had gone differently, I probably would have pivoted to something else where I felt like I could be free in who I am, be seen, and then also fight for what I knew was right. And at that point, they created a new role for you, the director of cultural resonance. And you had this interesting hybrid job where you were like half HR, half creative, sitting on pitches. And you became famous within the advertising community for a PowerPoint presentation, a PowerPoint presentation. So tell us about that. Well, I'm not actually that good at designing, though. But so that first iteration didn't look that great. But you know what, I'm still shocked by it. I felt like I always say I had the advertising year of hardy beat, right? It was like a Tiffany Haddish year. I just never expected this kind of reaction. But to me, it all goes back to my love, which my love is, how do we strategically connect with human beings around the world? And as a director of social strategy, that was what I was tasked to do. But I felt that I had to still rely on the skill and the mission of doing so. And so for me, I think also marrying that with my passion of my identity of being a Black woman and wanting to really try and elevate the voices of marginalized groups, I particularly for years was so interested in Black Twitter because, one, I felt very much a part of it and identified with it. But it was a really great example of showing how a historically marginalized group could really find a democratized space like Twitter and rise to the top, push culture forward. I mean push activism forward in ways that we didn't need to wait to be hired by a CNN or wait to get this journalism job. We had a microphone and a podium in Twitter and not just for activism, but for joy and celebration. And I think that was such a beautiful representation of what happens when you understand culture, when you understand technology, and also when it's important to you to connect with these groups. So I just kind of said, you know what? I think people should know this. I had a 13 person team I was a part of at VMLINR and I said, I'm going to do this presentation. I had a lot of gifts in it. It was funny. But then our CMO responded to the meeting invite. And I was like, oh, let me try and pause this up a little bit. There were some good jokes in there. I kept a few. And so I think that was what was important to me is that it was something I believed in, something I identified with. But it still lent back to my mission is that the more we can better understand different groups, different cultures, especially married with where we are right now in terms of what we have access to, what the world has access to at their fingertips, platforms like Twitter, how are we going to evolve where we are? So I think that was kind of what happened. I didn't expect it to catch on wildfire. And I'm grateful that it did. It's brought me a ton of opportunity. I'm still just as passionate about that presentation. And speaking of the head of Twitter, am I right, was in the audience at one point. You got to ask to do this presentation over and over again. You're not kidding about having a year like Cardi B. People were inviting you. Twitter ended up recruiting you. So how hard was it to make the decision to switch from an agency to the tech world? What made you make that switch? You know, I think for me, and we talked about just kind of my hybrid role, and it was a hybrid role, for me the hybrid was not something that I necessarily thought was going to happen because my background is not in HR. I don't do any recruitment. My background was straight strategy. And so I think when I kind of petitioned for the role, I knew that I needed a partner in HR because it's just important to see the work that's going out the door. You want to make sure that there's a team that's also inclusive. So I needed a partner in HR, and it was kind of like, why don't you try it? And so I was kind of doing these dual roles. But I think for me, what was kind of the enticing thing about the Twitter opportunity was that I get a chance to be at a platform that's agnostic. It's our job to serve the public discourse. It's our job to be this platform for all the voices that I talked about before. I think in the ad world, sometimes you can be a little bit compartmentalized. So let's say, I think Pepsi's in the audience, right? Pepsi and Gatorade was one of VML, Y&R's clients. Shout out to you guys, wherever you are. Had some good times, but I could never talk to Coke. You know what I mean? And so now it's one of those things where this mission that I believe in, when it comes to understanding intersectionality, when it comes to understanding people at their core, I can spread this mission across industries, across verticals, and that to me was freeing. And then also this role at Twitter is actually not in HR at all. It is purely strategy and experiential as we connect with people in real life, which we'll probably talk about a little later. So that was kind of what was core to my passion. And I think that we have some amazing people in our HR team at Twitter that do work on that piece of it, which is so important and integral. But it's not what I'm tasked with. I can really focus particularly on the strategic development of understanding and listening and amplifying culture and voices. And I want to ask you more about that. So what exactly is your mission at Twitter? What are you trying to do there? And this is a brand new role, too, I should say, right? Has it existed before? Yeah, I can get those brand new roles. Yeah, I can do that now. I think the mission for me is the same mission that I had as that 16-year-old who thought I was going to be Susan Taylor. It's this idea of connecting with marginalized voices around the world and amplifying them, aligning with them, and elevating those voices to be heard. And that is what has always attracted me to Twitter as a user for a decade, is this idea that there's this space where everybody has the same amount of characters. Everybody can be heard. And I think what I hope to do is really I lead the charge that connects with all of the marginalized groups on our platform so that they have an advertising, you say one throat to choke. I know we've probably heard that term before. But a person to go to, right? A name, a person who can talk to you about if you're having issues on the platform or something's bubbling up and you want to get more amplification, or if you need to understand the way things work. What we're trying to do is really connect with these voices and understand them in ways that I think our platform allows us as we have this incredible amount of public discourse. And what kind of things do people come to you with it? When they reach out to you, is it like that they're having a problem on Twitter, that people are ganging up on them, or they're trying to get a message out? Yeah, I think that right now, every single day at Twitter is different. And that's one of the things that I love about it. And so what we've been really trying to do is make these connections, I think, Twitter's 13 years old, so they are in this incredible space right now to really evolve who we are in a really great space and I'm grateful to be here. And I think this is very important from our executive staff level all the way down to really reach out to our voices on the platform and start to listen, what is it that they need? So we work with organizations, we work with individuals, and that's really kind of where I live. I work with the power voices who have something to say, who have equity, who have trust on our platform. And that doesn't mean a celebrity. These are the everyday people who use this platform to move culture forward. And so the kind of things that come through could be anything, could be learning more about our policy all the way down to wanting to get connected or having ideas about content partnerships and then kind of directing those people in the right way. But mainly right now it's done a lot of education and introduction because we want to build that trust. That's what's most important to me is my number one objective for this year. And let's get into brands a bit. So this weekend on Meet the Press, I'm not sure if you saw this, but the US soccer teams, Megan Rapino, called on brands, yeah. We're all familiar, no one here's been under a rock. She is the homie, it's been that cool. So she called on brands to quote, get comfortable throwing their weight around. And P&G actually followed suit by giving half a million dollars to the team. I'm wondering what is your take? Cause I know people have different thoughts on whether brands should be political. Do you think that brands should be part of the social conversation and if so, to what extent? I mean, absolutely. You know, right? I think we had this strange few decades where brands were just kind of fat and happy. And you guys kind of wrestling your laurels and these people who make this brand who it is, these consumers that are trusting you, buying your products, they are going through things, they are passionate about things, they are, you know, want to learn more about things. The idea that you could kind of separate yourself from those experience, I think is no longer. And I think what's really most important is that we're at kind of the first time where we've got a two way connection with brands. Brands have been talking at you for years. And the radio, the printing press and newspapers and TV, you can never talk back. Now you can talk back. And people are doing it in droves and they're doing it on platforms like ours where we see they're telling brands what they want. So this idea that you can kind of, you know, have consumers just be regular consumers who are completely removed from the things that make them who they are and their identity and their lived experience, I think is completely absurd. And it's always kind of been absurd, but nobody really had a way to challenge it. Now that we have that, I think that the days are kind of long gone of not wading into these waters that are important to people. Yeah, I love what you said about kind of the two way communication we're in now and how that's unique.  Like maybe there's a recent example, but to... It's so funny, because we did this tour today study with IPG where we really came up with, we saw that about, I think it came to 25% of buyers think that cultural relevance is one of the most important reasons that they make a purchase. And so I think that was a really interesting number to us which we started to look at how, how does this matter to people, you know what I mean? So I think for a brand wanting to get into it, you mentioned Procter & Gamble. I've been really interested to see and we've got a chance to work with them on how they are kind of approaching this. And I think it's, you know, it's scary a little bit. Nobody knows if they're gonna do the wrong thing, but I look at it as a brand being an individual almost. And we talked about this I think before we tied it on the phone and that's where I kind of start to make sense of things. When you look at a brand as this big conglomerate, it feels wieldy and scary to make a stand. But I think when we think about it from an individual level, you think about how human beings connect with each other. And I think Procter & Gamble's done a great job of that. They just launched The Look, which talks about the black male experience in America, which we know has a ton of different good and both good and bad experiences that come with that. And they looked at it as an individual. They want it to be a company that said, this is part of our consumer base. We want to not only talk about this particular experience, but also see how we can drive conversation around it. And that's something that like you and I are friends.  You would say, God, how can I help? What is it that we can do to better learn about this? So from an individual standpoint, it makes a little bit more sense to me. And I think brands have to get a little bit down to not overthinking things as a conglomerate and thinking of it as if I was an individual, how would we support the people that mean the most to us?  And I think that's because you've got a ton of different people making one brand. But I think if they think about it at that level, it's about brands standing up in what they believe in. And that sometimes means that people may leave or stay. You mentioned this. I mean, there's a real fear of backlash. I mean, just with P&G to use that example, as soon as they announced this, there are people on Twitter saying, I'm never gonna buy secret deodorant again. And they're saying, this is now my favorite deodorant. I will wear it for the rest of my life. Like, is there a class in how brands can deal with getting dragged on Twitter? How much time we got left? No, I'm saying three minutes. You know what, I kind of feel like, and I always say, will there be a day where I get dragged for something? And I think the thing about being dragged is that if you are confident and you stand for what you're getting dragged for, it doesn't matter. You know what I mean? If someone is dragging me, because I said Black Lives Matter, I could care less. Black Lives Matter to me, right? So I think it's one of those things where we've got to just convince brands to be bolder. And when you stand up in what you believe in, kind of that impact and that reaction, is it unfortunate? And will there be some bottom lines that could be affected? Possibly, yeah, but at the end of the day, I think that it's important for brands to understand that they have to stand up one way or the other and who they are. And sometimes it's not just such a hard political line. Sometimes it's something of sustainability or just things that are ethical. And I think when you talk about the bottom line, often what we see is that brands who are clear about who they are, we see that actually increase their bottom line because people know to align with that brand and they feel that they can see themselves in it.  So what is the best lesson that your mom passed on to you that you think is worth passing on to the women in this room? Oh my gosh, she's giving me so much, mom. I'm gonna say two, I think, that stick with me the most. Everybody suffers from an imposter syndrome, right? And I think sometimes even for black women, it comes even harder at times when you're literally kicking down the doors to these rooms that we've never been in. And I think my mom would be the one that's kinda cheering me on and telling me that nobody knows what they're doing. You know, it's such a simple thing, but you start when you get into these rooms where you've never been in, you just assume that you're this and everybody there is that, right? Nobody knows what they're doing. And I think that's, and it's true. We're all learning every day. And I think about Twitter and the charge that we have in being this platform for public discourse. We're learning every day and how to evolve this and make it work and to work at our executive level and have all of these people who are running huge companies. I've worked with CMOs. Everybody is trying to figure out what they're doing and to just evolve and be better and be smarter every single day. So I think that has helped me as well. But the other thing that she taught me is never accept the first offer. I love it. And that's something that has stuck with me as a negotiating tool. Yeah, yeah, good job mom. Good job mom. I might have time for one question in the front row here, yeah. Is it in or was it in? Yeah, so it was called woke, lit and ready. So it was actually about black Twitter and the history of kind of how African-Americans have communicated and how that has evolved to how we communicate on Twitter. So it was really kind of just understanding that group. And when I gave it to our advertising kind of that first audience, I talked a lot about how there's untapped talent there that should be valued, that often goes overlooked and how those are copywriters, those are strategists, those are organizers in ways that maybe the advertising agency wasn't traditionally thinking of. And then when I had to do this in front of all of Twitter, their one team event, I had to kind of change that. I'm like, okay, well they know this, right? They use this already. And I said, well, I kind of challenged them to say, well, how are you connecting with voices like this? And then, there you go, and this is how. Goddess, thank you so much for being here with us today. Thank you, thanks.


Where are you joining us from? Oh, I'm doing from Iran. Iran, right? Yeah, 4.30 in the morning. Oh, wow But it's good, yeah Thanks for joining session It goes goes 67 miles from from capital. Yeah Well, look at that only three of us Lots of people watching the YouTube videos, but not so many people joining us live. I guess it's a Hard to maintain the consistency. Well, thank you for joining That's a Certain thing you wanted to talk about or should we keep going? Oh Oh Yeah, see me. Both of you Okay, I Think you were about to finish the glyph. Yeah, I think you finish the glitzy. Let's go have a look and then Share my screen So I guess we're doing this one. Were we the rest Were we in the middle of something let's see I Guess not. Is this different to up shoe? Is this like a bigger version of it or something? Oh, it's up shoes That is up shoe. Yeah, we've already done that, right? I Think so, yeah. Yeah, I think so I'm just gonna type help just in case it's some weird different version of it. Oh What has happened that's strange, okay Never mind And if we definitely not done Iota under bar, I guess if we did it would have been probably here Okay dialogue language elements and try and remember what Iota under bar does where interval index Let's put it next to Iota. Okay, so I'm gonna guess that it is shift I correct Iota under bar. All right, we've already got row at this point. So that's good be Matt is two by three all right, so I Order under bar. I guess pretty clearly is showing us the locations of the binary trues And more than that it is replicating them the number of times according to what the number is Doesn't work on negative numbers Doesn't work on floats Okay It's easy enough and then for Matrices It's telling us the coordinates of each of the binary trues. It would be easier if this is printed out Row one column two two one two three. Cool. So Anything else I should add or anything that makes sense What's this one? We won't put it in our notebook, but Should we try if we can figure out what this is? Iota row omega. Oh Okay, so Oh, why is that here? Hi, Isaac. I see so I order I order a row of omega is the indices that Yeah correspond to that to that array and Okay, so That's very important That's a function. That's a function function operator Function, okay, so that's a function. That's a function. That's a function. So this is a book With three functions. Does that sound right? I'm right right so we go Row of This And then we go slash over comma Now there's no reason to use over right because it's monadic. So we could just use jot and get the same thing Yes So first so It's this the commas not necessary is it? Haha, it's not. I guess this is in case. It's like a scalar or something Okay, and then Can anyone remember what slash does The replicate I Guess that's the monadic great Overdue my addict slash is not defined. Oh So Wait, how is this working because one addict slash is not defined that's a syntax error. Oh No, no, no, it's dyadic Because there's a omega here Okay So it's dyadic Okay, we've got a okay, we've got a fork slash Iota row and a dyadic fork applies the outside functions to both outside arguments So Here. Yep. So this is a dyadic fork and it applies the outside functions to both outside arguments So were they just recreating the monadic Iota underbar Let's see. So this is going to be oh I see I think you're right So the reason this works is that slash means replicate and And The left-hand side tells you how many things times to replicate the right-hand side Omega slash omega Okay, so there's a mega slash omega. So that's doing the replicate and then you've got Row of Omega as we discussed Okay, and then dyadic Iota index of So wait, what did I do wrong? Oh It's It's This row is applied to the left down the right. So it's a mega row omega So, right, so I could put parentheses here, right? No, okay, so I'm parsing it wrong. Oh I'm parsing it wrong because of the amegas This is not in parentheses. It's not a fork. So I'm totally wrong So we actually have to be careful about precedence here Without parentheses, it's going to be there's actually no operators in this version. So we Find it most tightly on the on the right So this is actually simply going to be row of omega, which is four and Then Iota which is one two three four So it's just going to be that Which does give us the answer okay Gotta be careful of Precedence and I guess you could do that tacitly by doing Jot and then what a dumb cause selfie. Oh Look at that. Awesome. Yay Okay that was a bit of a Digression a single line in the docs to say This is how you can define Iota under bar. We've made that a bit faster, but I guess it'd be less fun Okay, yeah under bar. Thank you for your fast cable stuff. By the way, Isaac. Hey There's something I wanted for myself. I Think a lot of people are gonna find it really useful. Yeah, I hope so. I'm right now trying to Fight with get her back to him. I'm trying to get it scheduled so it'll Run update if any of the libraries are if any of the libraries are The great idea not the latest it'll just update. I mean then I can Have a big repository that people can just Point to so you don't have to upload their own once they want to Now this one I Remember doing with Claire. It's a bit of an odd one Oh, we're on Okay so Hey, are you? Well, there's two things. Yeah, what's it doing? Okay, so the number of our results is the same as the number on the right-hand side Let's start with a numeric one Because I think I know this one. So what the two four six does is it creates a number of groups? The first group it creates is less than two and Then two to less than four and then four to less than six and then greater than or equal to six And so less than two will become zero Two to less than four will become one four to less than six will become two Greater than or equal to six will become three So that's why one is less than two so it becomes zero two is Two less than four so it becomes one and so forth. So just tell to these to find breakpoints And then it just applies these to those breakpoints and gives you the results But be very careful because it's a bit weird that it Starts at zero instead of one which When I asked about this on the APL discord Marshall Lockburn told me he considers this an off by one error in APL Although somebody else pointed out it does have some convenient properties So you can think of it as like the first group number one is the bit defined by When you actually get greater than or equal to two so zero is like Less than the minimum value Does that make sense? Okay, so this is exactly the same thing That for letters these letters are in alphabetical order So D is between A and E so it's one Y is later than U so it's five Gotcha Okay rank two arrays Interval index works with major cells Okay, so this is one of these ones with broad broadcasting built in Wait, is it higher rank left argument? It compares sub arrays NY Which is the right argument With the major cells of X I'm not quite following What are sub arrays of this I guess three three is a sub array So it's not broadcasting Length just three or three three three would give errors right But what if we did Uh Two rows two columns of three three Three five Yeah, so it's This is sub array one. I guess this is sub array two. So this is the same as as This result concatenated with this result How does it decide that three three is one? I guess it's looking through here to find like a row that contains What exactly Would zero three give you the same result? No I think it's like it's trying to it's basically like looking for the first Okay, I think the issue is It's it's it's comparing column one and then column two And column two would only matter if there was a tie So I think if we went Um, okay one two three four three five So three three Is not bigger than three four Three four is But then the three five is higher still And then four five is off the end Oh greater than or equal to is always going to be there Yeah, so it's saying what row which would this slot into By first of all making sure that it's greater than or equal to the first column And greater than or equal to the second column. So this one Would have to go here because the second is not greater than or equal to But if either of them are greater than the second and it would slot them after right so yeah, but I think it's I think it's exactly the same as doing 12 34 35 All right, that's basically the same thing right same results And if people you want to learn more about this, they've got quite extensive examples Which I suspect means that this is fairly important I guess and you know anytime you're trying to Place a continuous number into a bunch of buckets You would use this kind of approach like plotting groups or something like that Molly, you're very quiet. I can't quite hear you properly Oh, uh, yeah, my mic was long ways away uh histograms. Yeah Yeah circle Okay, I wonder if circle should go in the basic math section Because that's circle functions. I can't imagine we're going to need other Stuff Much a bit of a big topic, I guess circle functions Uh circle here it is right Circle, how do we write circle? Hey, yeah, just remember I've actually installed the APL keyboard. I don't need this thing anymore. Oh, I guess it's useful to see what button to press anyway Okay, so it's Oh, so I can just press alt o. Oh, maybe I need to do this Alt o There we go. Cool Alt o Which I guess it's just called circle. Yep Circle monadic circle I times So I guess that means we could just go What at a circle one? And can we do like that? Yeah, okay Seems easy enough Okay, nothing weird fun oilers identity so I hi I A to the pi I Yeah, this identity Although at this point we haven't done any kind of Trainy things so maybe we should do that in two steps equals Oops, what did I do wrong? Oh, it's not exact. Is that what it's saying? Something weird going on here Have I done something weird? So this is Pi I This is pi I And then this is either the pair of that Yeah, that's definitely pi I That's strange Oh I see It's minus one plus 10 to the negative 16 I Which is basically zero So I think what's happened is My guess is that This is special case or something in APL dialogue APL To remove the annoying floating point residue left over bit here And basically this is close enough to zero I For some reason when we do it all in one go it's Gets it exactly right Right, well, maybe we won't show that then Okay There isn't really a short way to do this so I'm inclined to just provide a link Um, so basically the idea is you Put something on the left And whatever you put on the left to find what function you get And cos is two And presumably we can do both Yep, and can we do it to multiple things? What does that do and not quite clear on what happens when Alpha is An array Oh, I guess it's but well, no, okay, that makes sense. So If we do Pi Okay, so that's basically zero zero or maybe point five So then what happens if we And what does that do Because you kind of I was expecting four results I was expecting sine and cos because If we do sine and cos is zero Or sine and cos Of half pi So why do you So what are the two numbers been? I I think you're applying each function to each of the er sorry Each straight function to each of the ones in the right side I need better vocabulary You think it's broadcasting over the scalars Yes Yeah, I think uh Oh, right, that's exactly what happens by default in apl right is uh A scalar broadcast over an array But if they're both arrays then they just broadcast to each other. Yes Yes, yes, yes, you're exactly right Okay Great, okay Element of we should put with our kind of set related stuff Which I guess is where uh Up and down she were epsilon Why not epsilon Means enlist because ml for us is one I remember And it's just called epsilon Yes Epsilon epsilon Enlist And dyadic epsilon is member of Oh, okay, okay. See what's going on here is two by three Row I o to six So what it's doing is it's got zero that's got a matrix from one to six That's got an array with seven and eight that's got a scalar nine And it just Platons them and sticks them all together flatten can pattern out. I wonder if that does that for More axes Yes, it does Seems easy enough. Uh, okay, so this is interesting Yeah, so this is two by three But this is a single this is oh, sorry, and then this is Ranked this is a shape three So I guess this is um During the broadcasting thing kind of it's Applying it to each row all the elements in ravel order So this flattened that method arrays Yes, so Yeah, it is And also higher anchor arrays But yeah, this last one's interesting and that it's uh Combining these different shapes and actually We've ended up with less ones than we started with right So it's doing something slightly weird here Oh, no, uh, wait, am I doing this right? Okay. No, I'm reading it wrong So actually Yeah, sorry, I forgot I got confused by the precedence Yeah, so Even though there's no space there we have to remember this is actually that right Okay. Yeah, so it's just flattening it out. So it's not weird Yeah, so you got to be a bit careful to remember that although it prints this like a word nine It's actually These are just characters n i n and a that's why this shape is 13 Okay member of abc four Is that Is it an ella? Okay. So is abc an element? Of this set No, it's not Is for an element of this set? Yes, it is So we get zero one and then this is just saying is Each element of the matrix an element of Is set and it puts them in the same order. So One is not an element two is an element three is not an element so forth Makes sense. So if you If you kind of reversed that and did Like one two three epsilon mad Would it be searching like the major cells of if mat was on the The right hand side. Yeah, so it's saying Okay, so still looking cell wise. It's not looking at major cells Saying is six anywhere in that matrix And it's two anywhere in the matrix. Yeah, exactly So I think the right the the rank of the right hand side seems totally meaningless It's just traded as a set I I also tried with two matrices and got the same behavior that you would expect there. Cool Thanks molly All right epsilon underbar Just dyadic Okay, where is a in Oh is a no Is b and A and a is a and a is and I think it's going Okay Wait No, because n is in both and that would be zero And that's zero. So it should be So I was wrong Huh Santa compelling Yeah, worked for the first two It finds occurrences of x within y Okay The whole of x Anna is here And Anna is here And we're going to try to give the easy examples Okay Okay, how do we create this thing x is Okay x1 Is Two by two Oh one One oh two Four by four One one That's what we want. Uh, wait, what's oh, that's x and y Never mind x. All right Oh, I see so it's finding where is like this Oh, where is this like one one diagonal? And you can find it here Here and here Because they're the ones Okay, makes sense Yeah, I think the the the apl wiki seems to have a kind of a I think a better explanation of it, but The inquiry maybe you guys already talked about that. I've been I've been in and out of the discussion Um, no, we hadn't thanks buddy Great Why is there an n here? Is that actually the other end? Oh, no, let's see the other shoe on that one And till do we've done this first two might go under the math I think might go in math Yeah, I think there's the circle star is log and natural log And the domino looking thing is matrix division and matrix inversion. I think Okay matrix we're going to have to do after we've done rank Certainly log we can do circle star Oh, that's easy Do you know how to type it? I think it is The back tick star Oh Or alt star now to me Not a great rendering here. I think rather than writing that I'd rather write e to the power of one There we go domino quadtivo Matrix inverse and matrix division by Okay But I'll write them down and then we can put them somewhere And how do I take this one molly to know or anybody? I believe it's um The um plus So it's the same as division, but with the shift key I think That one I think so that doesn't quite look right does it that was sorry? Sorry, that was alt shift slash Uh, alt shift plus or oh Oh shift plus sorry Is there some way to get a What's a good easy way to get an identity matrix? I did that in a blog Oh Well, I used it. I don't know that I uh doing how I got here was um I'll put in the chat when my identity function was why is this not working? Have I got it the wrong way around? I think I do actually Yes Oh, there's a few things happening in the chat I noticed I guess it's fine. Anyway, we can see easily enough. It's That's the inverse. That's the identity Oh I'm not really sure what matrix division means anybody of you now Oh, I think you're using x instead of the dash How do I do that? Okay I haven't quite heard that expression matrix division before but it makes sense It's just the opposite of matrix multiplication Oh cool. There's pseudo inverses as well If there's more rows and columns the least squares result So I think that's just what's called the pseudo inverse It's neat Okay, linear regression on complex numbers. Why not? Oh lots of things to study there Probably a good time to stop I thought we've done dot Or if we only done it in this one situation What do we not do it at all? I'm not sure that we've done it No, that's bad because I've definitely referred to it Yeah, I think we talked about it a bit Um, okay. Well, let's pop it underneath a ray rank But I don't know that we Linear algebra Let's quickly do this then But we finish dot Rich is a operator. Oh, it's an operator So we can't do it here Let's put it over here All right Okay, so this is a dot product One times four plus two times five plus three times six This is Interesting So this is some and isn't it? So this is three equals three and three equals three and three equals three and three equals three If I did this would be zero That's cool For example Okay You know when I was playing with this inner product I was thinking that using it with the up and down styles for min and max was Could be useful as well Right Go on. I can see where you're heading Uh So you could um I'm reading my own blog post trying to figure out what it means. So Oh, let's maybe just put it in the Forms right after we do it And so then they had uh Oh, yeah, then they got the special case actually Didn't we do that in The competition thing Because we were doing that. Oh, or maybe we just played with it, but we ended up getting rid of it I remember we talked about outer product for that gene example But um, yeah, I think I messed it up and didn't quite get it working So that's fine. All right I think that's it Thanks all Good to see you All right, thanks a bunch. Bye Okay, bye