alternative subtitle, Unicode, dates, and names. So really quickly about me, I'm co-founder of a startup called Apex, based out of Charlottesville, Virginia. We do analytics for digital marketers, so those agencies that have lots of data that you heard a previous speaker talk about, we help them manage all that data and understand and try to make sense of it. You can read more about Apex at Apex.com. My personal website is the second link, and then my Twitter account is JXXF. Oh, also one more thank you to the organizers, Ben, Karen, and Jonan. Thank you for inviting me, and thank you guys for having me. So this talk is fundamentally a talk about assumptions, and we use assumptions all the time to understand and simplify the way the world works. Now, some assumptions are good, and they serve as well, so for most of the time, for most of the assumptions that we make, there's no problem. So eating a balanced diet is good for your health. I don't think many people would dispute that, but sometimes you make assumptions like I did on this trip that don't turn out to be true, and then you have consequences for those assumptions. And when our assumptions are wrong, we wind up with problems that we might not have foreseen to begin with. So for example, I'm often prone to the problem of believing that I know where the performance hot spot is without having to do any actual profiling, and so I just sort of be like, that's definitely the problem. We take that out, but usually I'm wrong. Sometimes people might think that for all objects that you could define in Ruby, the objects are equal to themselves, but that's not true because float nans are not equal to themselves by definition. And other times we make assumptions that are predicated on social or cultural constructs that wind up in software that turn out not to be true. And they're reflected in the structure of our software. So here's a database schema from a Massachusetts civil service that I worked for that specifies the gender of the people that are to be married in a marriage license, which doesn't work as soon as you allow same-sex marriage, and it's a very expensive process to reverse that. So we make assumptions about the way the world works, not just in terms of software, but in terms of social and cultural assumptions that are then encoded into software that we're also making a whole other set of assumptions. Sometimes our assumptions are maybe even more simple, so here's a string that says Noelle, either in reference to the term for Christmas or somebody's name, but this seems to be five characters long, but it looks like there's only four characters there. And if we reverse the characters, we don't wind up with what we might expect of reversing the individual letters in the phrase. And we'll talk more about that later. Sometimes we make assumptions about the way dates and calendars work. Sometimes we might think that very logically, the date that's after one day can be gotten by incrementing that by incrementing the number of the date by one, but that turns out not to be true. So as we'll see, I think this talk is really all about people and the assumptions that people make and what happens when they go wrong and how we can do better. So I hope to enlighten you on that subject. So first let's talk about time. I don't think people would dispute that time is a difficult thing to get right in software. This is the on-diss size of TZ data, which encodes all of the information about how time zones work across the world. So all of the offsets you have to know, all the times in history that those dates changed that takes up almost a megabyte just in terms of rules about how to organize time, right? Imagine writing one megabyte of Ruby source code just to deal with time zones. Forget about encoding time in terms of hours and minutes and seconds and months and years and calendars and so on. So we're always adding rules too. Every time a political body changes when daylight savings times occurs, for example, another entry goes into TZ data, one for every time zone that's affected. We're always adding rules so we can never reduce the rule set unless we want to be incorrect about some subset of dates. And, you know, I don't think it should be any surprise that time zones are tough because we seem like we ask a lot of questions about time zones. But why is it so hard? And the answer is that we made it that way. We made a lot of presumptions about the way time work that sort of accreted over time, so to speak, and then led us into this big ball of mud. So first, to understand what time really is, we sort of have to, in terms of software, we have to ask the more philosophical question of what time is. And this is sort of a perennial philosophical debate. Isaac Newton came down on one side and he said that time is sort of this object or concept that exists independently of whether or not humans exist. So it's not a mental construct. It's something that's real and it's something that's a property of the universe we live in. Other people thought that time was not such a property that it was a mental construct that we created to try to understand the way the world worked and would not exist, but for the fact that human minds perceive the world in a particular way. But for purposes of our discussion, we're going to say that the definition of time is like the definition of space. So space is what separates me from any of you. The reason we're not literally occupying the same place is because we have different spatial coordinates. Likewise, the reason that all events don't happen simultaneously is because they have different temporal coordinates or at least that's the way we perceive it. So we're doing pretty good, right? A few slides in and we've already resolved a major philosophical problem of the ancient philosopher. So hopefully we'll have an even better track record for the rest of this. So just pull out your cell phones or look at your watches and just take a note of what time it is right now and just yell out what time you think it is. Okay, so most people said an hour in a minute. A few people said p.m. But I think I heard most people say an hour in a minute. So we can tell you what time it is by giving you a really small amount of information or we could be really, really specific about the time. We could say it's two hours and 30 minutes and 12 seconds or 230 and 1200 milliseconds or any number of granularity all the way down into the minimum possible unit of time that we care to think about. So that's an arbitrary distinction about how we want to do that. But that doesn't always work. So think about what 230 means. So 230 really means that we have this continuum of time and that there's a range of possible moments on that. There's an interval of possible values that time could take where it would be correct to say that it's 230. So that is the moment 230, all time that starts when 230 is true and ends when 230 is no longer true. But moments are... And when we talk about a specific moment, we can be more specific and narrow the interval that we're discussing by adding more specificities. So instead of 230, if you said 230 and 12 seconds, you'd get a smaller interval, but still a continuous one. So that's the moment, two hours, 30 minutes and 12 seconds. The problem, of course, is that moments all by themselves are ambiguous. Just today, there are two different... There are two distinct moments for which the statement 230 as a time is true. So that's an ambiguous reference as to which moment we're talking about. And, of course, this repeats every day, all the time, so to speak. It's gonna be a lot of time related inadvertent puns, I think, but bear with me on that. Now, the reason that I can understand you when you tell me what time it is and I'm not lost in a maze of ambiguity is that humans have context. So I understand that when you say 230, you're talking about the... You're talking about some window that includes this afternoon, today's date, this year, the Gregorian calendar, this time zone, and so on and so forth. I've assumed a lot of things that make it possible for me and you to have a conversation about what time it is and still understand each other. But if we're gonna tell this to computers, we need to be a little bit more specific. We can't just be more granular. We have to be more specific about what we mean. So when we say what time is it, we can't just say 230. We have to add all the pieces that remove ambiguity about what time it is. So we have to say whether we mean PM or AM, we have to say what day of the week it is, or we can say what the actual date is and that will tell us the day of the week, we have to say what year it is, and so on. Now, that gets us to the point of we can talk about what a time is and reference it with respect to other times, but we're still gonna have some other problems when we talk about times between two people that aren't necessarily in the same room. So when we compare times, one of the first problems we might encounter is we may not be using the same calendar. We might not have the same words for months or even the same number of months or the same number of days per month. We may not even have the concept of months at all. There may be one big year or a Mayan calendar cycle that has nothing to do with months, but has everything to do with seasons and so on. So the first problem is that we have to agree on what calendar we should be using before we can talk about other kinds of time because otherwise we're not gonna have a good way to convert between different moments in time. So we'll say, okay, everybody's gonna use Gregorian calendar and that's pretty much what happened around the 16th century of all of Europe and North America and parts of South America switched to that same calendar. And let's also, by the way, remember that when we say that we're gonna switch to that calendar, we also have to remember the date that we switched calendars because if we wanna go back and look up what old dates were, we're gonna have to remember that we used the old calendar before a certain date and the new calendar after that date. Okay, next problem. Your time isn't going to be the same as my time if we're not both standing in the same spot. So if I say 2.30 p.m. and we're in Portland, Oregon at 2.30 p.m., that's not gonna be about the same position for the sun in the sky at 2.30 p.m. in Portland, Australia. So, okay, let's fix that problem. Let's give everyone an offset for their local time so that we can all look at the sky and experience the same daylight hours roughly as each other depending on where we live and that way when I memorize my offset, I'll be able to convert between a time that you have. So this guy named Charles Dowd basically did this for the U.S. He was a seminary teacher who proposed time zones to a bunch of U.S. railway operators and they liked the idea so much of having time zones that it was eventually adopted in the U.S. in 1883. So all of our time zones in the U.S. basically harken back to cross-country railroads. So he said, great. So now we've got time zones and we've got the idea of offsets. So we know where you live when you are talking about a time. So you've associated that time with a time zone and that time zone has an offset. Next problem. That time offset's not the same value for all possible values of time. So, for example, in the U.S., we have daylight hours and evening hours. I'm sorry, daylight savings time and not daylight savings time or a standard time and in Portland, Australia, they don't have that. There's all one time offset no matter what time of the year it is. Okay, next problem. Your local time offset isn't the same as my local time offset for at different times of the year like we already saw and it's going to be different for different years. So, for example, the Congress of the U.S. changed when daylight savings time started a few years ago. So we have to remember when the daylight savings time start for a given year. Okay, so we have a lot of exceptions to when those rules happened and when they didn't happen and when they started and when they ended and so on. Next problem. Some of the times that you want to talk about don't actually exist as moments in time in my time zone. So, for example, if you want to talk about one in 30 a.m. Pacific time on Sunday, November 2nd, 2014, that is not a time that exists because that's when we go back an hour for daylight savings. There is no moment in time that universally corresponds to a unique instance there. So, for example, if I try to convert in Ruby from that, if I try to say give me the UTC time for this local time, it's not going to work because there are two UTC moments in time that correspond to the same local time. Okay. All right. So we've got all that information we have to remember. Obviously, we have to store this somewhere and we have to store it in a way that everybody understands it. That's what TZData does. So TZData is a way of encapsulating all the rules and weirdness in human logic about when time started, when times ended, and so on. So for every time zone we have, we have to remember a distinct list of historical offsets and that's what TZData does. So this is what a TZData file looks like for Pacific time. They're all referenced based on the canonical city for that time zone. So in this case, the canonical city is Los Angeles. There's a bunch of rules about when Los Angeles or when Pacific time was started being observed or not being observed. Now, sometimes, remember, you have to remember every single offset change to the list of offsets we have, which is why Indiana has changed its mind a lot about what time zones apply to which parts of Indiana. So we have 10 different time zones that you need to memorize. You can't just say central time. You have to know which part of Indiana you're talking about. So in Ruby, the two big classes you probably care about if you're just using Ruby and not Rails are time and date. There's also date time, which is more of a shell thing that doesn't really do anything unless it's augmented. But the important thing to know here is that time, as in the class, is aware of its offset. So an instance of a time is aware of its offset, but it is not aware of its time zone. So you must keep track of the time zone that you care about, or otherwise you run the risk of converting between two times in a way that does not make sense. And date is sort of even worse. It's not really a completely specified time. It's just a day on the calendar. So it doesn't correspond. It's like a moment. It doesn't correspond to a unique enough point in time that we can pinpoint something that we care about. And when you convert to UTC, you get a new time instance from your time zone. But again, because it's a time instance, you also don't have a time zone awareness. Another big problem in Ruby is that we don't have an idea of period or duration. There's no way to add like two hours to a specific time. There's no object that we can instantiate that represents three days and two hours. There are ways of making a new object that correspond to an advance of three hours or an advance of three days and 12 hours, but there's no object that represents that fact. So the closest thing we can really get is, or I'm sorry, one example of why that's a problem is if we start doing additions in terms of seconds or other granular units, we'll wind up with the wrong answer whenever we cross a offset boundary. So here, for example, when I add, when I pick three times November 1st, November 2nd, and November 3rd, each at 1 and 30 a.m., that's right around the Daylight Saving Sign fallback for the fall. If I subtract the first two dates, I'll see that they're 90,000 seconds apart. So I'll believe that a day is 90,000 seconds long, but a day is actually 86,400 seconds long. A good way of getting around that problem is that active support has an augments time with a method called advance, and advance pretty much does what you think it should do, and in this case, we'll get rid of the ambiguous time error by being able to advance one day and get that next day after Daylight Saving Sign. But it's not always correct in terms of what you think it should do. So here, for example, we see that advance is not an associative operation. If we start with January 30th, 2014, and we say, hey, I want to go two months in the future from that date, we'll wind up with what we think might be the expected answer of March 30th. So two months from January 30th is achieved by incrementing the month counter by two. However, when we do that twice in a row rather than all in one step, what happens is that we go first one month forward into February, which only has 28 days, and so the day counter is reduced to 28. So if a subsequent month when we add one more month, we're only going to the 28th day. So we wind up with March 28th versus March 30th. So with times, they're really, really hard to get right. Make sure that you always store your data in UTC if possible. Avoid trying to roll your solution, trying to roll your own solution of any kind because that will only wind up in blood and tears. And be aware of the limitations of whatever library you have to happen to be using. And this is a special plea. If anybody has ever used Jota Time, which is the Java Time library that's very popular and there's a number of ports to it, I would love to see somebody finish the Ruby port and come talk to me afterwards. I would love to hack on that together sometime. Okay, next up, Unicode. Why is Unicode hard? Again, because humans. So in the beginning in 1963, IBM invented an encoding that worked on mainframes called EBCDIC or EBCDIC is the quick pronunciation. So you can convert strings into EBCDIC. You can represent strings like as you can with any encoding in EBCDIC by using the icon library. And here we see that we've turned Cascadia into EBCDIC encoded bytes. People didn't like that because EBCDIC had a really inefficient system of encoding how the bytes were stored. So everybody kind of switched to Assy, which used only the lower 127 values of each byte. And so they believed there was an inefficiency there. So if we imagine how characters are encoded, if we imagine this grid that occupies one byte and it's gonna have 16 rows and 16 columns for 16 times 16 will give us the 256 values. So if we imagine this chart represents all the possible... If we fill in a value in each chart here or in each box here, we'll be able to have an encoding system that fits in one byte. So EBCDIC's encoding chart looks like that. Lots of wasted space. The red boxes are assigned values and the empty boxes are values that are not assigned to an encoding. They're undefined values if you try to encode them. Ascii's encoding chart looks like this. We use all the lower 127 values and we don't use any of the upper 127 values. But there's still a lot of empty space there for each byte. We're not only using 50% of the space. We could probably do something with all that empty space, but right now it's being wasted. Sometimes we use them for box drawing characters. Have you ever played old DOS games or used old DOS applications? That's what they use, the upper 1.5.4 is the extra box drawing characters. So you can have lines and do sort of primitive graphics. Or you could use them for emoji. You could define your own encoding that had extra symbols in it. But probably a better use is if we took that extra space and we said, hey, what about those languages other than English that need to be represented in the world and need to exchange data with one another? What if we used that space for them? What if we supported more codes? So we had the idea of code pages which were basically mapping a special number to the encoding that you wanted to use. So I would tell you, hey, this string is being encoded with code page 472. And then you would go look up what code page 472 meant and now you would have a mapping between all the characters that are in code page 472 and all the bytes that are in that string. So the code page tells us what to put in that empty space, how we should render those characters. Turns out that most of the time for most of the code pages they all kind of agreed about what 0 through 127 and each byte should be. So that was still A through Z, 1, 2, 3, 4, 5, 6, 7, and so on. And some of those other languages, we don't really care about you. You're not gonna get code pages because you don't use computers right now and this is the 1980s and we'll figure it out later. So here's a problem though. What encoding is this string of bytes? Well, you can't tell because there's no way to encode the... There's no way to determine the encoding of an arbitrary byte sequence. And if you need to do that, you have to remember both the bytes that you care about and the encoding that you want to use. But the encoding itself is also stored as a sequence of bytes. So we have to remember that and now we have to make sure that we understand what identifier is being used for that. So this is not a great state of affairs. We would really prefer that there was some way to encode stuff that didn't require us to negotiate content between each other. We would rather just have one system that managed all of that. So enter a Unicode. So if you imagine the English alphabet for starters, A, B, C, D, E, F, G, and so on, we can also think of other variations that are essentially A, B, C, D, but which are represented visually in a different way. But you can still identify that these characters are A, B, C, and D, and so on. Likewise, you can see that these are still A, B, C, and D. So we understand that different representations of the same letter are still the same letter which is shown differently. So we think the way and the word we call that is a glyph. So a glyph is sort of the platonic ideal of what a letter is. So when we say A, what we don't mean the word of the letter A written in a specific font or with someone's handwriting, we mean the generalized perfect ideal of what the letter A is. And a glyph assigns each encoding that we might care about to a specific value called a code point. So in Unicode, for example, the Latin small letter A corresponds to encoding code point 0061. Actually, the extra zeros at the beginning are superfluous, but by convention, there's four of them there. Or this tie character can also get its own encoding. Or this emoji can also get its own encoding and so on. So Unicode's encoding chart actually works by giving you an unlimited set of values to choose from. All you have to do is be willing to commit more bytes per character that you care about. So each character does not occupy necessarily a fixed number of bytes. You have to look up what glyph it corresponds to before you can understand what the, before you can understand whether or how many bytes that character should take up. And that specific variant of Unicode we get as much space as we want. It's called UTF-8. So Unicode has, UTF-8 has unique unlimited numbers of encoding. So for all the characters that currently existed, anyone has ever written down at any point in time, anywhere in time, we have a way of representing it. So, you know, are we done? Great work. Well, not quite because Unicode and Ruby has a couple of gotchas. So remember before I showed you the example here where this value is five, but we might expect it to be four. And if we reverse this string, we don't necessarily wind up with what we thought. So I've broken apart the characters here. And you can see now that the diuresis, that's the little two dots above the, to the right of the E, that character is actually what's called a combining diacritic. That means it combines with the character of its two, its left, to make one single character. So I've underlined in yellow the fact that those two bytes are used to represent one character. So if you, so when we ask for the length of that string, we actually get five and not four because that combining diacritic counts as a character, even though as a glyph, they're represented as one unit. And that means we can have surprising results like if we reverse the order of the bytes, sorry, if we reverse the string and then ask for the bytes, the bytes corresponding to the diuresis are not reversed. They're kept in the same order. So in other words, it reverses based on character, not based on byte. So likewise, just because you think that your character should be atomic doesn't mean that your library will treat it like that. Make sure if you're performing unicode string transformations that you do so with a healthy awareness of what's going on under the covers. And again, as before, be aware of what assumptions the library you're using is making about how unicode encoding works. All right, last thing, names. Names are hard. Why are they hard? Because people are using them. Everyone should read this article by patio11, aka Patrick McKenzie called falsehoods that programmers believe about names. I won't go into it too much here except except to say I believe that 80% of the things on that list were true statements and it turns out that none of them are. So a fundamental problem with names is that names are a possibly empty set of strings that map to something, a person, a place, a location, a ballroom, etc. But almost no system that we invent for software models names that way. Many systems treat different parts of names especially like you may have a field for middle name and a field for last name and so on and so forth. But this will always lead to disaster whenever you're trying to accommodate a global audience. So who here has heard of the scunthorpe problem? One person, okay. So scunthorpe problem is named after a town in England called scunthorpe England or it's called scunthorpe and this contains an offensive word in it. But it also has about 100,000 people who live there. So every time they try to put their address into a filtered field they're going to run afoul of that problem. Linda Callahan in 2005 tried to register a Yahoo email account but couldn't because her name contains the string Allah and Yahoo was banning all people or all accounts with that substring in them. This domain name was not possible to register for the first 10 years of the internet's life because internet, the organization that was the registrar at the time prohibited domains with profanity in them and shatakeymustrooms.com contains a profane word. Craig Coburn that's pronounced Coburn couldn't register Coburn at hotmail.com his email also even after he was able to register it his email got caught by spam filters a lot and see if you can guess why because his title was software specialist which contains the string C Alice which is highly associated with spam email messages.  So Dr. Loki Sky Lizard a thoracic surgeon who by the way was on Google Plus before I was so explain that Google. Or maybe this University of Alabama football player ha ha Clinton dicks or some other real actual names of people that are all awesome but probably would not be allowed on Google Plus. So you can see there's a lot of really bad assumptions being made about names. I'd like to review a couple of them real quick for you. One of the worst assumptions that people make is that names don't change. How many times have you been on a website if it didn't let you update the name that you were assigned right? So what if you get married and change your last name for example? Another assumption people make is that names only change at predefined times and the number of those times is probably limited in some way and that you're probably your name will never change unless you go through one of those predefined things. I will bet any set of such events does not include someone going into witness protection and so therefore would be missed by that sort of scheme. Another assumption people make is that users have a single canonical name that they go by but that's not true because what if someone has a nickname or maybe you are called something different in college? Okay fine but maybe people have a single canonical name for financial and legal purposes but what about credit reports that aren't in sync? If you have two credit agencies that both have a different idea of what your name is one may have your middle name and another may not, different names. Another assumption is that names will be unique in the context of some system. There are 820,000 people with this Chinese name. There are about 46,000 people in the U.S. named John Smith. There's about 120 people named John Feminella. Another assumption is that names will contain capital letters. That's definitely not true. You only have to look at history and some famous literary people to understand that. E. E. Cummings was a poet and Bell Hooks was a prolific feminist. Names don't contain numbers but this is a real New Zealand child's name. Number 16, Bus Shelter, 16 numerals. Okay fine but they probably won't contain non-Alpha-Numeric characters so that's clearly not true. Jay-Z has a hyphen in his legal last name. Okay fine but names will always be unicode characters surely not prints. Another assumption is that people have names to begin with but that's not true. In the U.S. nobody is legally required to have a name. That's just a social convention. Your life will certainly be harder if you don't have one because you'll have no way to identify yourself but you're not legally required to have a name. So don't filter inputs for the sake of just trying to identify a specific real world person, place, or thing. And I think you should probably just have names as a unrestricted single text field of virtually unlimited length if you can manage that. Thanks very much for having me and thanks again to the organizers for inviting me.


Admiral Samuel Locklear, Commander of the U.S. Pacific Command, spoke at the surface Navy Association Symposium about the Pacific Pivot Initiative to realign our Navy's fleet. The underlying thing behind that whole pivot is that after two decades of really difficult work in the Middle East, we have to look globally at where our long-term national interests are. Our children and our grandchildren's where their interest is going to be most important. And the continuing vector, the consistent vector, is in the long term is to make sure we get it right in the Asia Pacific. This initiative will strengthen our Navy so we can continue to be a global force for good. From the Defense Media Activity, I'm Petty Officer Tyrell Morris.


Hello everyone, welcome to Options with Doug. Streaming live daily on Bookmap Discord and the Bookmap YouTube channel at 1.30pm Eastern Time. Before I get started, I need to go through the Disclosures. General disclosure, all Bookmap limited materials, information, and presentations are for educational purposes only and should not be considered specific investment advice nor recommendations. Risk disclosure, trading futures, equities, and options involves substantial risk of loss and is not suitable for all investors. Past performance is not necessarily indicative of future results. Here's my contact information. The best way to get in touch with me is through Discord. My name on Discord is Doug P, and then there is a chat channel in Bookmap Discord, Options-Doug. That's a great place to post your questions, comments, and your own content related to the topics of this channel, which I'll go over in just a moment. So Bookmap Discord, Options-Doug, chat channel. I'm also on Twitter, X formerly known as Twitter. My name there is at Doug Pless. And hello, Gray. Welcome, glad you're here. If you could just give me a quick check. Yesterday there were some individuals reported a problem with the stream on Discord. If you would just please let me know Gray or anyone else on Discord if my stream is coming through. Okay, great. Gray confirms. Kraken as well. Thank you. Hello, Kraken. Glad you're here. Welcome. All right, so it looks like all is good on Discord. All right, the focus of my presentation today and the focus of the Options-Doug chat channel is options, order flow, the impact of options markets on stocks and futures, and the influence of market maker hedging flow on price action. I have a two-step process for trading and the first is planning, and I use positional analysis. I look at how traders and market makers are positioned in the options market and how those positions change from day to day to develop a thesis regarding the expected trading range and volatility for the day, as well as the directional bias. And the second step of my process is execution. I look at real-time order flow in Bookmap and real-time market maker hedging flow and spot gamma hero to confirm my thesis and for setups for entries and exits. And when I talk about setups today, I will be talking about setups in an underlying asset. For example, and setups can be taken, setups can be taken any number of ways with futures, shares of stock or options. My analysis is based on the options market of an underlying asset again, but setups can be taken with futures, shares of stock or options. Questions and comments are welcome, and I will be watching both the Options-Doug chat channel and Discord, as well as the chat and YouTube for your questions and comments. Please feel free to post and I'll do my best to answer. All right, my agenda for today, what I want to cover, just a quick update on news for today. Then I'll go through my positional analysis, then I'll review some setups from this morning, and then I'll talk about the live market. And when I get to the live market, if anyone has any stocks they want me to take a look at, please let me know and I'll be glad to do that. All right, so news items for today, there was some minor news, 9.45 a.m. Eastern time, there a PMI data came out, manufacturing PMI was greater than expected, greater than the previous number, and then the services PMI was less than expected and less than the previous number. So, excuse me, not really much of a market mover. All right, let's take a look at my positional analysis for now. I'm going to start with the S&P 500. This is the S futures and book map. And before I take a closer look at this chart, I'm going to take a look at a larger time frame. This is SPX in a 30-day one-hour chart showing price and key levels. First of all, let me focus on price. So, this was the September, wrong tool, September expiration, the September 15th expiration last week, call dominated, positive gamma expiration, and you can see the move lower after that expiration. And then the SPX moved all the way back down to the low of the, right around the low of the August expiration, right at 43.35. All right, so a much, much quicker move lower than the move higher. All right, let me point out some levels on this chart. I'm going to start with the daily and weekly expected moves. So, the dash purple lines are showing the lower and upper weekly expected move. SPX is creating well below the lower weekly expected move. That's based on the options market. That information should be available in any trading platform with an options chain. All right, the next, the dash blue lines are showing the lower and upper daily expected move. SPX is trading within that range today. All right, so those are the expected moves. Now, let's take a look at the spot gamma levels. These are proprietary spot gamma levels based on gamma weighted open interest available to spot gamma subscribers. And I'm going to point out the key daily levels. So, the first level is the 4300 level. That is the put wall. That's the strike with the largest negative gamma that can be expected to act as support. And that is also the absolute gamma strike. That's the strike with the largest absolute negative and positive gamma. And those levels did move lower from yesterday. So, both the put wall and absolute gamma strike were at 4400 yesterday, moved down to 4300 today. And that is bearish. That is a bearish signal. All right, the next level up is the volatility trigger. That's at 4,480. That is spot gamma's proprietary gamma flip level. Below that level, market makers position on the gamma curve is negative. In a negative gamma environment, market makers have to trade with price to hedge their delta exposure. And that tends to enhance or increase volatility. And SPX is trading well below that level in an extremely negative gamma environment, gamma positioning at the beginning of the day. I'll go over that in a few minutes. All right, the next level up is the call wall. That's the strike with the largest net positive gamma that can be expected to act as resistance. So, the range today, there are the range of ceiling to floor 4,300 as the put wall, the potential floor. Again, that level did move lower opening up. The potential for a move lower, that moving that support level lower. And then the call wall, the ceiling at 4,600 not really in play right now. So, the 4,300 put wall is much more important. All right, and this chart is kind of scrunched up because of this 4,000 l4 level. All right, so let's take a look at the levels in play for today. So, I'm going to go to another SPX chart, one day, one minute chart. This is the August 18th expiration low. So, SPX now below that level. Let's just scroll down a little bit. So, the next level below is the 4,326 level. That's a combo l5 level. And then the put wall down at 4,300. All right, let's go to book map now. So, in book map, I have my own cloud notes and I'm showing the SPX levels. Yeah, I don't see. So, anyway, I've got the, there's a 4,350 level, 4,352, noted as resistance, the 4,350 level noted as resistance and the spot gamma AM founder's note. There's the 4,336 noted as support. Around that level did act as support earlier today and now the SP500 has broken below that level. Next level down, the 4,324. Note there is a difference in price between ES and SPX. And today it's about 41 points. That's what I'm using. So, SPX 4,324 is at ES 4,365. I've also got SPY levels on this chart. There's SPY 4,34 that did act as resistance. And now the SP500 is heading down toward the 4,31 SPY 4,31 level. All right, so those are the levels I'm playing for today. Again, the SP500, well below the volatility trigger for SPX, also SPY, volatility trigger at 4,45. And note for SPY, the put wall and absolute gamma strike also move lower. So, for SPY, the put wall moved down to 4,30. Giving more room to the downside for SPY as well as SPX. And then the absolute gamma strike is at 4,35. That also moved down and that is up above. All right, so those are the levels I'm playing for today for the SP500. And we'll take a look at setups in just a few minutes. All right, let's take a look at NASDAQ. Here in NQ Futures, upper daily expected move, right at NQ 15,000. Price chopped around that level for quite a bit. And finally found resistance around that level as options traders were taking negative delta positions in the NASDAQ, QQQ and NDX. I posted that on Discord earlier this morning, right around 10.49, and it took a while for that to play out. All right, this is the, actually before I, I do want to take a look at the QQQ chart. Let me do that in the NDX chart to see the levels and play. So for QQQ, there's the 360 level, that's the absolute gamma strike. That level did move lower from yesterday, from 370 yesterday. And the put wall for QQQ also moved lower down to 350 today from 360 yesterday. Again, opening up more room to the floor, to the potential move, for a potential move lower. Let's take a look at NDX. So NDX really only this combo L5 level in range today. All right, back to book map. So again, I have my cloud notes here. Again, as I pointed out, there's the upper daily expected move and price could not move higher than that. Here's the 360 level. You really can't see it there, but that's the QQQ 360 level. And it looks like today around this 359 level has been support more or less. All right, so those are the levels and play for NASDAQ. And both the ES and NASDAQ, S&P 500 and NASDAQ, not real clean adherence to levels today. All right, let's take a look at some additional information now that I use in my positional analysis. I'm going to start with gamma notional. This is market makers position on the gamma curve at the beginning of the day for SPX, SPY, NDX, and QQQ. These numbers are significantly negative. 1.5 billion negative for SPX, 2.779 billion negative for SPY. And this is about as negative as it gets really. And then for QQQ minus 1.059 billion. So market makers position on the gamma curve at the beginning of the day was very negative. Again, a negative gamma environment, traders are long puts, market makers are short puts, and they have to trade with price to hitch their delta exposure. So let's see what that means. We'll take a look at the VANA model. So I'm going to start with SPX. This curve with a very steep skew to the left, very typical of a negative gamma environment. What this chart is showing is market makers delta notional or delta exposure and how that changes with changes in price. There are two curves on this chart. So first of all, the light gray curve just shows how market makers delta notional changes with changes in price only. And the purple curve adds implied volatility to the equation. That shows how market makers delta notional would change with changes in price and applied volatility. And that change in delta notional with a change in implied volatility is the VANA effect. VANA is the second order Greek. Let me check for questions. All right, Redshift Trucking says after watching you daily for about a month, I have now subscribed to Spot Gamma. And thank you for sharing your take on the markets. You're welcome. Glad you're here. And hope you're getting a lot out of Spot Gamma. I certainly do. And VHU, hello. Welcome. Glad you're here. Yeah, I will take a look at NVIDIA when I get to set up review in just a few minutes. And Joel, glad you're here as welcome as well. Welcome. All right, so Mark says screens blurry. So well, I think at least as far as I see, the highest resolution that I can get on YouTube is 720p. If you join us in Discord, I stream at 1080p on Discord. And Bookmap Discord is free and open to all whether you have a Bookmap subscription or not. And hello, Mosconi. Welcome. Glad you're here. Good afternoon. All right, so the VANA model, let's take a look at where SPX is trading now. All right, so it looks like SPX making a new low of the day right around 43.31. So that's right about here. So again, this curve is very typical of a negative Gamma environment. So what this is showing is if price increases and applied volatility drops, market makers can buy back short futures hedges. And that's a put VANA rally as their, again, trader's long puts, market makers are short puts. As those puts lose value, delta drops due to an increase in price and a drop in applied volatility, market makers can buy back their short hedges. And this works both ways. So if price decreases and applied volatility increases, they have to sell futures to hedge their delta exposure. So they're trading in the direction of price. So this is showing here price increasing, they can buy back shorts, price decreasing, they have to sell futures. Again, remember, they always want to remain delta neutral. So that's all that curve is showing, but it gives us a good indication of how market makers may react to changes in price and applied volatility and that's a big part of my determining a thesis for the day. Looking for today, of course, I was looking for very high volatility based on the very negative Gamma in the market. All right, let's take a look at spy real quick. Spy trading right around 432, similar situation, very negative Gamma curve, market makers trading with price. Finally, QQQ trading at around 359, also near low of the day. It's about here. Here we go, 359. All right, so that's a Vana model. So based on this, my thesis for the day was for higher volatility, wider training range. We'll see if that plays out in the afternoon. Then also, my directional bias was bearish. That's based on the shifts lower in the put wall and absolute Gamma strike for SPX, spy and QQQ. All right, so Mike Gimm, I will ask, could you please show how I'm converting QQQ prices to NQ? He says he has the indicator I mentioned, but it only works for current price. All right, I'll show you in just a few minutes. Let's take a look at some setups and then I'll get to that. All right, so I'm going to start with take a look and see what options traders have been doing today. So this is the spot Gamma hero signal. This is hero stands for hedging impact real time options. This is available to spot Gamma subscribers. What this chart is showing is price for SPX with a white line. The purple line is showing the hero signal again, hedging impact real time options that showing options trades and market maker hedging activity for a combined signal for SPX, spy, XSP and ES futures. So if you trade any form of the SP500, this is really the signal that you want to take a look at. I'm going to zoom in on this and I'm really only going to highlight one setup from the morning. Let me zoom in on this just a little bit. All right. So what this chart is showing is right around 10, 15 options traders. So initially right around 10, they took positive delta positions that leveled off a bit. Then that around 10, 15, they started taking positive delta positions again and price responded higher. So when traders, when market, when traders buy calls, market maker sell calls and they have to buy futures to hedge their delta exposure. And I'm talking about calls and SPX, ES and spy. All right. Let's take a closer look and see what traders were doing. So what I've done is separate outputs and calls. Let me zoom in on this just a little bit. So it's, this is somewhat subtle, but it gives us a better indication of exactly what traders were doing when I separate outputs and calls. So first of all, the orange line, rising orange line, is showing traders we're not doing anything with calls. Then right around 10, 15, they started buying calls again. So the hero line for calls, the orange line is flat. Rising orange line, that's positive delta indicates they're buying calls. So right around 10, 15, again, they start buying calls 10, 17. And they were buying puts shown by the falling blue line and they start selling puts. So here in this case, this area right here, both the lines are going in the same direction up. So positive delta and spot gamma has noted that that concurrent signal of both lines going in the same direction is a very strong signal, either up or down. In this case, traders are buying calls and selling puts, market makers taking the opposite side of those trades, and they are buying futures to hedge their delta exposure. All right. So that's right around 10, 15, 10, 17. Let's go take a look at book map right around that time and see what was going on. So let's, let me zoom in. So by the way, I do most of my prep in the morning. So this is what I saw, and most of my trading in the morning. So this is what I saw. We'll get to the live market in just a few minutes. So we know that traders right around 10, 15, 10, 17 started taking positive delta positions. Now we're going to look at book map. So when I'm trading, I have hero on one screen and book map on another screen. So I'm watching hedging flow and order flow at the same time. So I know that traders were taking positive delta positions and that shift, both of those lines puts and calls both shifting up at the same time. That's a strong signal. At the same time, I see potential support levels. So this 4336, that's SBX 4336 noted as support. And the spot gamma M founders note also the spy 4333 level acting as support. Note the shift in order flow. Magenta dots showing aggressive sellers. The dots volume dots are by minus sell where there are more sellers than buyers. The dot is magenta where there are more buyers than sellers. The dot is green. So the order flow shifts at a potential support level as traders were taking positive delta positions. Cumulative volume delta shifts higher. And also its price starts to move higher. Bystop orders shown by the rising yellow line as well as these green dots help to fuel the move higher. So the initial move was up past this 43 resistance level, 4350 resistance level to the 4352. So that was the setup in the morning that I saw the bullish setup in an otherwise market that I was really looking for bearish setups. But anyway, this was a very easy read in book map and in with hero. Right. So that's the setup that I wanted to highlight on the SB500. All right. So Desiree says meta Microsoft went south after Euro close. Maybe some call gamma unwind potentially. This is not the type of environment that I'm generally looking for call gamma unwind. But we'll take a look at that. All right. So this is the that was the highlight that the setup that I wanted to highlight. And it looks like that move eventually ended up at the 434 level before reversing. All right. Let's take a look at NASDAQ. I'm going to go back to hero. Total signal. I'm going to take a look at NASDAQ. And actually let's most of this is QQQ. Let's just take a look at NDX. Well, there's some contribution from NDX. So we'll take a look at NASDAQ. So this is the combined signal for NASDAQ, NDX and QQQ. QQQ making up the bulk of this. But we'll include NDX for completeness. And this is the setup that I pointed out earlier posted this in discord took quite a while to play out this divergence. So initially from the open hero was trending slightly lower traders taking negative delta positions. And then right around 1030, they started getting more aggressive, taking negative delta positions.  Notice that divergence again took almost two hours to play out. But price finally responded lower as traders continued to take negative delta positions. So let's see what they were doing. So they were buying puts and selling calls. So in this case, again, both lines going in the same direction, both negative. So they're selling calls and buying puts. Now that could be because well, we'll just leave it at that. So that's what traders are doing in NASDAQ. So let's go take a look at book map to zoom in on this just a little bit. So let's take a look at the order flow items in order flow. First of all stops shown by the rising yellow line here by stop orders fueled to move higher. That activity levels off. Also note the cumulative volume delta really starts to shift lower just after 1230. Let me zoom out show one other thing. So sometimes liquidity in the NASDAQ is hard to visualize. But you can see the heat map here showing a history of the limit sell orders in the order book. So there were sellers coming in between the 1520 and 30 level as price made highs around that level around those levels. And again, remember, options traders were taking negative delta positions pretty aggressively from about 1030. All right, so that took a while to play out. But there's the bearish set up in NASDAQ move lower and hero starting about 1030 and the price response lower took two hours or so. All right, so that is the NASDAQ. And while I want NASDAQ, let me answer your question, Mike. All right, so what I do is let's go back to the QQQ chart. So what I do for NASDAQ, I have a simple little script in thinkorswim that displays this number in real time. So I'm just dividing in Q by QQQ price. And I look at it for a minute or two and just take an average. So today, after looking at this and it jumps around a little bit, I ended up using 41.534, maybe just slightly higher than that. So that's the number that I use, Mike. I use that number. And then what I do is let's go to my NQ chart and I have an add on called price lines. This is it right here. Price lines. This is available in the book map marketplace. So it's an add on that you can purchase and it will fill out this column here. It displays these labels and draw labels and draws the line, the horizontal lines. All right, let me show you the spreadsheet. So this is all this information comes from a spreadsheet. So this is the section for QQQ. So right now I've got QQQ 350 to 380. I'm displaying 350 to 370 that's shown by the yes over here. Then what I do is just enter this little formula right there. So that is sell C187 times 41.534. And that then calculates the correct NQ number shown right here that corresponds to 350. And then I just drag that all the way down. All right, so Mike, I hope that answers your question. Let me know. Let me know if that makes sense. Let me know if you got that. All right, so that's what I do. And I just leave that for the rest of the day. All right, so SK Gango, hello. He asked how to import spot gamma levels to book map. Tried several times, not able to. All right, so I, there are a couple ways of doing that. You can use the cloud notes that spot gamma provides. I like to use my own. So I'm just showing, let's see, I'm showing, let me see, let's go back to actually, I don't know. Let's open up. All right, so I go to this, the home page for the dashboard, go to daily index levels. There's spy. So I do the same calculation for spy that that comes again from a little, that little script in thinkorswim. So 430, I just note that in my, my spreadsheet, 430, the difference, the ratio between ES and spy 10.1345. And I do that calculation by hand so I can actually show that label. So let's go. So 432 is in 10.1345 times spy, that gives me the correct ES number. And that's shown right here. All right, and then also with SBX, it's just a price difference. Again, 41 points today. All right, so SK Gango, that's what I do. And you can also just go to spot gamma, go to the dashboard, go to the integrations, and you can add your levels for book map, trading view, thinkorswim, book map, etc. All right, let's take a look at some setups for stocks now. So the first, and let's go to, oops, we'll go to AMD real quick. All right, so initially traders were buying calls, also selling puts, we'll just zoom in on this. Sorry about that. All right, so initially traders were buying calls, selling puts, up until about 1040, call buyers take their foot off the gas, and then traders started buying puts. So price makes an initial move up, and then is, again, call buyers take the foot off the gas. Traders stop selling puts and start buying puts. Price moves lower. So let's take a look at AMD. We'll go to book map. So there's the move higher. These traders were taking positive delta positions, and then they level off, start taking negative delta positions and price moves lower. All right, the next meta, let's go to heroes, see what options traders were doing. So meta, pretty similar story. Initially, traders were taking positive delta positions, whoops, wrong tool, driving price higher. Then they take the foot off of the gas again, start taking negative delta positions, and price responds lower, separate outputs and calls. So initially they were buying calls, selling puts. So they stopped buying calls, stop buying calls, stop selling puts, then started buying puts. Price moves lower. So very good long setup in the morning. Let's go back to book map, take a look. If you're fast, good long setup, up to the 305 liquidity, and then price reverses lower as options traders start taking negative delta positions. Let's take a look at Nvidia, and then we'll go back to Microsoft. All right, Nvidia, there was a good long setup, right around 1015. Let's go to hero, take a look at that. Here's the long setup in this morning, right around 1015. Traders shift from taking negative delta positions to positive delta. Price responds higher. Note 420 is the hedge wall. Zoom out just a bit. And similar to the other stocks that we took a look at, as options traders take their foot off the gas, price consolidates, then reverses lower as they start taking negative delta positions. All right, let's go take a look at book map and zoom in on this. So remember 420 is the hedge wall. So that's a spot gamma level. And note, there's a quite a significant shift in order flow here. Magenta dots showing aggressive sellers. And then right around 413, aggressive buyers start to come in. They're shown by the shift from magenta volume dots to green volume dots. And a quick move up to the 420, 420 hedge wall level, and the high liquidity at that level shown by the the heat map and book map at the 420 level. All right, Mike asks, says, thanks, Doug. You're welcome. So as you replay the webinar, see it now. One more thing, does the formula and the spreadsheet you use ever change? It changes every day. So the the numbers, the ratios, and the price difference changes every day. So the ES to spy ratio changes every day. The NQ to QQQ price ratio changes every day. That's why I have the instead of doing any manual calculation, I have those little scripts in and thinkorswim. So you know, I did, and I just note these every day. So it changes a little bit every day. And note the ES to spy ratio will increase quite a bit at the contract rollover. So the the ratio increased quite a bit last week with a rollover to the December contract. And now it will gradually move down towards 10. And same for the same for NQ to QQQ. That ratio will gradually move down towards 41. And the ES to SPX difference widens out quite a bit at the rollover. It started right around 50. And it will ended up at the at the rollover for the September contract. It ended up right around three. So it'll start it started at 50. And I expect this ES to SPX difference will gradually move down towards a single digit number again. And Jerry asked, how do you calculate this ES to spy ratio? Again, I do that with a script. A thinkorswim script. It's very simple just to buy ZS by spy. It does it in real time. And I posted those scripts links to the scripts in the options-dug chat channel. Look for the pin icon in the upper right. All right, so that is NVIDIA. Long set up. And if you can't find it, let me know. So move up to the 420 Hedgewall level. Options traders start taking negative delta positions. Price reverses lower. All right, let's take a look at Microsoft. Let's go take a look at Hero. So with Microsoft, I just did not see as a clean setup. That was partially due to this very large block order here. That's straight line up in Hero. There was a large block of calls. So large institutional order, large block of calls. And note again, as the call buyers took their foot off the gas, Microsoft reverses lower. They're also buying puts that show by the negative notional value there. All right, so slow to sorry. There you go. There's Microsoft. And note for Microsoft, 315 is the put wall. So if you're looking for a spot gamma based or gamma based target, there it is at 315. And you can see all the high liquidity there. For those of you who may not be familiar with book map, this is showing a history of the limit orders in the order book. Very significant number there shown by the bright red shading. And note the liquidity comes in at the cash open, typically stays in there until it's filled. So now the the buy orders at 317 were filled. So all these below price, these are limit buy orders. And they come in typically at the round number levels for stocks with higher price stocks. Or let's take a look at the broad market now. Again, if anyone has any other stocks that want me to take a look at, let me know. And by the way, for those of you who are asking about the how I get the levels on the chart, the ratios levels, I do offer one on one mentoring. If if you need more, more assistance with that, I'll be glad to help in one on one mentoring. If you're interested, just let me know in direct message in discord. Doug P is my name. All right, so maybe the SB 500, maybe found support at this 4324 level, a level noted as support in the AM founders note. Let's go see what options traders are doing now. Then we'll come back to this. So price really reversed lower from the 434 level. Move back up to around 4352. All right, let's see what options traders are doing. Go back to the SB 500 signal. So overall today, options traders have had a somewhat of a mean reverting effect on price. So options traders are buying lows, selling highs, at least in the first half of the day, assuming on this. So it looks like at least for right now they are started taking positive delta positions again. So we'll see if that that reverses lower or not. So at that support level, options traders started taking positive delta positions. So even though negative gamma market makers position on the gamma curve very negative, the options traders are having somewhat of a mean reverting action effect on price with a slight bearish tent tilt. All right, let's take a look. Let's see what zero DTE traders are doing. So that's shown next expiry for the SB 500, SB X by ES, that would be all options that expire today. So next expiry for the SB 500 is zero DTE. Looks like that options volume, option notional value is about half of the total total. So the purple line is showing all expirations and the green line is showing zero DTE. So maybe a reversal or maybe just a pullback for further move lower. Let's go, oops, let's go to book map, take a look at order flow. So it looks like today there are a lot of large traders buying with iceberg orders that show them by the rising light blue line. You can also see the buy iceberg orders here. Also that number is positive for the day. CVD cumulative volume delta just about flat. All right, so based on order order flow here, this could just be a be a pullback for a continued move lower series of lower highs. Let's go back to hero. So overall since about 1245 that the overall trend for hero is lower, making a series of lower highs, although the total notion of value for the day is positive. Let's just let's take a look at a shorter lookback period. So just looking at the last 30 minutes of data, now that that notional value is negative. So overall in this 30 minute rolling window period lookback period, traders are selling calls and buying puts. Let's take a look at NASDAQ. All right, so now it looks like NASDAQ traders may be taking positive delta position. So they're definitely selling puts, still selling calls. Let's take a look at the total signal. So right right around 120, they started taking positive delta positions. Price continues to move lower. We'll go to book map, take a look at NASDAQ. I'm going to go ahead and shift back to the one day lookback period.  So still looks bearish in the S&P 500 to me. All right, NASDAQ looks pretty bearish too. By the way, this is what Spot Gamma was looking for. They were looking for traders to sell strength, especially with the shifts lower in the put walls. And then again, the put wall shifted lower for SPX, SPY, and QQQ.  I want to thank everyone for watching. Thank you for your questions and comments. And again, for those who are asking about how I display the, how I get the ratios, how I display them on my chart, I'll be glad to answer your questions in Discord. If you need some additional help, contact me through direct message in Discord, and I can set up a one on one mentoring. But again, just post your questions in Discord. And if, you know, if you're interested in one on one mentoring again, just let me know. All right, everyone, have a great weekend. Thank you very much for watching. Thanks for your questions and comments, and I will see you on Monday. Bye.


You know doctor lawyers take four or five twelve years to finally learn their craft and start making money You know and you can't just put in a couple months of hard work to learn this you want to get rich overnight You want to graduate overnight? So these these are things you have to consider guys anything.


Gweithio, mae'r athgorngu. Gweithio gweithio i fynd i gyd. Dyma gweithio, mae'n rhoi gweithio i gyd yn ymddangos. Mae'n gwneud o'r cyfrifio gyda'r Cardi Cymru yn yng Nghymru, mae'r ein bod yn ei wneud o'r gwaith ymlaen. Mae'n gweithio'n bwysig i gyd, David Gunning, ond ydych chi'n ei fyw i'w gweithio, ond, mae'r gweithio'n gweithio i gyd yn ei gwynhau. Fel ydych chi'n gweithio ar y gwnaeth. the presentation is on the record and the questions and answers aren't. It's really great pleasure, David, to welcome you here, and it's of honour to the Institute. David is going to be talking to us today about something that we probably all have thought about but haven't had the luxury of working on about explainable artificial intelligence. David yw'r pryddoedd Cymru yn mynd i'r bywch â'r byddwyd yma'r defnyddio cyffredinol yma o garfent rhagor pethau yw'r Abod, a yw'r pethau�arddedig sydd wedi'i gael y ddweud roedd ni'n gwybodaeth ar yr Abod, elwodraeth, mae'n addysg fel nawr yn yr unig o ddweud y gweithredu yw'n ddegwyddol yn roedden nhw, a wedyn byw yw edrych i ddigon o'i this. The thing I heard, David, is that, I think at the age of eight, you saw the sputnik go over and that was the beginning of your interest in technology. I also remember my dad bringing me out to see the sputnik, but our careers went in different directions. David has over 30 years experience in the development of AI technology and at the moment manages the explainable AI as well as communicating with commuter's program. This is David's third tour of DARPA and I can explain that DARPA has tours for four years so David has had three tours and it's there to get the excitement, interest and development on each tour. They're all different but he's also worked, as you can imagine, in startups, in industry and research. So he's had a very distinguished career and he's previously managed the PAL or the personalised assistant that learns the product or the project that produced Siri. He's at the intersection I think of the human mind and computer science and I say that because maybe there's some similarity between us David that you did cognitive psychology but then went on to do a master's in Stanford and computer science. I think perspective within that area is very valuable and I think it defines a lot of the work that you do. So we are asking the question can AI be taught to explain itself as machine learning becomes more powerful. People working in the area increasingly find themselves unable to account for what they know or what they don't know. And I suppose today we're really, really honoured to have you David here to talk to us about explainable artificial intelligence. Thank you very much. Thank you. Hi, good afternoon. It's a real pleasure to be here. This is my first trip to Ireland and really enjoyed Dublin. My ancestors came from here in the 1850s from Sligo County so it's a real pleasure to see my homeland.  This is a project I'm running now at DARPA. We started about a year and a half ago. It's a four year program so we're just beginning to get some results but I can give you an idea of what the program is about. So here's the premise, the need for the program which I think we're all aware of is just to say we've now had this explosion of new AI technology. A lot of applications of AI, especially of machine learning and deep learning that are behind it and so that's being applied in medicine to do radiology. It's being applied to do sentencing in criminal justice. It's being applied in a lot of places in defense, especially in intel analysis and in the development of autonomous systems. So this new technology is much more powerful than what AI technology could do previously since 2012 is when this kind of revolution started with deep learning. So it's very effective but the models are very complex and difficult for an average user to understand. So that's the problem we're trying to address. So here's a little, most people now are beginning to get tutorials on deep learning which is really the heart of the technology and this kind of simplified version of it will kind of give you a basic idea of what's happening in these systems. So these machine learning systems can always learn to do a function, a classification or prediction. That means you give it one set of data as an input, it's going to give you an output that's given you a classification or a prediction. And this is done by feeding huge amounts of training data to these systems and that's what has to be done to make them work.  You said, oh, this is a cat, this is a chair, this is a house, this is a table and you do that over and over again. And what happens in the layers of this neural net, you feed it the input at the top, you give it the classification at the bottom, it learns how to weight the input. It's calculating numeric weights in every node of this neural net so it can do the correct classification. So that's where the magic happens and now since 2012 this technology has actually been around since the 80s, the basic idea of a neural net. But in brown 2012 several things came together. One, there was enough data, plenty of data out there on the internet. We had more computing power and we could use GPUs and new computing infrastructure to do computing on these systems like we couldn't before. In minor changes to this architecture and then suddenly this stuff was doing much better at image recognition, speech recognition, kind of orders of magnitude better than what had been done before. So that's what came about. Now in terms of explainability there's kind of good news and bad news in these systems. You see this picture on the left or on your right. These systems have layers and they can have thousands of layers with millions of these neurons. As they inspect what's happening in the layers it tends to be at the early layers they're detecting simple features like edges, colors, textures. At the later layers it's detecting more abstract parts or features or concepts till at the very bottom it's learning to classify this picture as a wolf or a dog. When people inspect these systems they find out it's actually detecting features in these inner layers that people would understand. Just like our visual system we naturally organize information we see in the world and recognize concepts that are reusable that are important for us to understand. The bad news is the net has no idea what to call this feature. As far as the net is concerned it's a particular equation to calculate a weight at one of these nodes. So when the user gets an answer out of one of these things he has no idea why it did it. The only thing he gets is the numeric weight of the node at the bottom said more likely this is a dog that was the highest weight. So that's kind of the challenge we're trying to work with. How can you take these systems still take advantage of their power but find a way to pull out the features, the concepts, the logic of what's happening in the net and make it more explainable. So this is a kind of a simple version of what we're trying to do in the program. The top level, the top row here is the standard machine learning deep learning process today. You have a lot of training data, you need thousands sometimes millions of examples of training data to train these systems. You feed that through a learning process that then learns a learn function or a model. You can then take that model and feed it a new piece of data like this picture of the cat and if it's trained properly, a pretty high probability it's going to get the right answer, it's going to say I think this is a cat. If you say why it'll say the node for a cat had a weight of 0.93 and that's about all the explanation you'll get. You can't say why isn't a tiger or tell me more about why you thought that was a cat. You just don't get that out of these systems and that's the problem. So the row on the bottom is what we're trying to do to remedy the problem and that's really to do and make two changes to this process. One is to change the machine learning process so you get a more explainable model. So you identify what those features are inside. You somehow get more semantic information out of the net. Maybe you change its architecture so it's organized in a way that makes it more explainable and there's a wide variety of techniques people are trying and I'll show you a few examples of those. The second thing is going back to my background in cognitive psychology is to make sure you put together the right explanation interface so you can generate an understandable compelling explanation for the human user. And in this case our target is an end user, not a machine learning expert, not someone who's an engineer who understands the technology, but the end user, a lawyer, doctor, a soldier, whoever's depending on this system to make decisions and see if they can get an explanation that's useful for them. So here I believe is our characterization of the goal and I believe there's an inherent trade off between the performance of these AI systems and how explainable they are. So it's much like with people. There are people that are incredibly brilliant but can't explain anything. There are people that make great instructors but may not be the most brilliant researcher at the university. So there's something going on here in these systems. If you have the largest, most complex deep learning system that's learning all of these internal features it may not be possible to make that completely explainable. And there are other machine learning techniques like a decision tree or linear regression that are much more explainable to people but they're not quite as high performing. Now there is some debate about this. I have machine learning people that argue if we really get the architectures of these things right, if we really get explainability right, we'll actually improve learning performance. But so far that hasn't been shown to be the case. But that's kind of an open question if eventually we may get it all, right, make these systems more explainable and more higher performing but right now I think we're stuck with this trade off. So the goal of my program is just to move this curve up and to the right. These orange yellow dots are kind of where we are today and I want to have a portfolio of techniques. So any given person that's developing a new system they can look at how important is performance, how important is explainability for their application and their new techniques that they can use to make that trade off better for whatever purpose they have. So here are the two application areas, challenge areas that we're working on that are really generally applicable in a lot of areas but they're certainly important to US Department of Defense. One is if you have an AI system that's doing data analytics on masses of multimedia data. So we have Intel analysts that are flooded with images, photos, overhead images of all sorts of situations in the world. They have to pour through all that data and find items of interest, find where there's new activity in North Korea, what's happening in Syria, that sort of thing. But there's too much information for them to look at manually so you have to use AI systems to pull up the most relevant targets for them to look at. They need explanations so they know why the system is pulled up a particular item. We started the program, I was at a workshop with some Intel analysts from the US government and one of the ladies there talking about the workshop was basically a lot of machine learning people trying to pitch their new technology to the analysts. And she said, well, this doesn't solve my problem. She already had big data, data analytics systems that were giving her recommendations. But she's saying she has to put her name on the recommendation that goes up the chain. And if that's wrong, she's blamed, not the AI system. And so to her having that explanation was kind of the most important facet of the system. The second area is autonomy, right? Of course, self-driving cars, you know, we're in the middle of beginning to develop all sorts of autonomous systems for the defense department, right? So, and this is a more, still it's not quite here, right? A lot of these things are in development. You don't see a lot of these autonomous systems out in the field yet. And the ones that are coming soonest are not really very heavy in AI technology. They're programmed with much simpler control logic. But the people in the research universities are now using this deep learning technology to train robots, to train autonomous driving systems, to train autonomous air vehicles. And when that's done, if an operator sends one of these systems off on a mission and it comes back, he'll want to know why did you make the decisions you did? Why did you turn around? Why did you succeed or fail? Unless there's more explainability built into the deep learning versions of these systems, he won't be able to get that information. So those are the two challenge problems we're having our researchers work on. Probably a little small for you to read because it brings up the question of how do you know if you have an effective explanation? What makes an explanation better than another? And we'd love it if we could find an automated way to measure that, but I don't think, as far as I know, we can't do that yet. Because the quality of the explanation depends on a human user reading that explanation and getting a better understanding of how the system works. So the only way we know to measure this is to actually build some of these systems, put them in front of a user, have him use the system, get decisions, get recommendations. But then measure everything we can about it to see if he has a better understanding of the system as a result of the explanation. So he could ask them, are they satisfied with that, with the answer. One of the most important aspects is, does it give the use of the right mental model of what the system is doing? So you want the person to know, have an intuition on when the system is strong and when it's weak. When should he trust it? When shouldn't he trust it? If you gave the user a hypothetical situation, can he predict what the system will do? So that's some of the techniques we're using to see if the system gives the user an accurate mental model. And it won't be a model that's completely the same as the complex math model inside the AI system, but does it give him the right intuition or the right kind of analogous model so he understands what the system is doing? A second can measure their task performance. Of course, if the human and machine are doing a task together, they should do a better job if you have the explanation. Do they trust it appropriately? Is there calibrated trust? It can be relatively easy to trick a person into either blindly trusting or blindly mistrusting one of these systems. So the goal here is to not do either one of those, but to give the person a better idea when they can trust it and when they can't. There are some experiments done in psychology where they'll show people two different computer systems. And in one case, they put a smiley face on the computer and people will actually trust that one more. So there's just all sorts of quirks of human nature that make us either trust it when we shouldn't or mistrust it when we should. So we're hoping explanation will help correct that. Is it a little more technical diagram? This Venn diagram, which you probably can't read, is a description of all the different machine learning techniques. There's neural nets and deep learning that's so popular, but there are other statistical techniques like support vector machines. There are these graphical models like Bayesian belief nets. There are simpler systems like decision trees. So we are looking at that whole portfolio of techniques, although deep learning is the most important and the hot topic these days. So this is where I get this curve, where some of these techniques are high performing, but not very explainable. Others are more explainable, but not as high performing. This is going to show us three broad strategies that we're employing on the program. First, what I'll call deep explanation, which means do whatever you can to make deep learning more explainable. So that's the most severe problem. It's where all the activity is these days. So more than half of the performers on my program are trying to do something there. I'll show you some examples of ways they can make these systems more explainable. The second one in the middle is what I'm calling interpretable models. So this is to say don't use deep learning, use some other machine learning technique that will learn a structured Bayesian graph or Bayesian belief net, or learn a decision tree, or something that has more structure that makes it more explainable. In the last technique I'm calling model induction, this is a little bit like what people do with rationalization. We don't always know what our deep nets, why they've made a certain decision, but we can make up a plausible story that kind of fits our history and the situation. So in this case I'd have the explanation system will not try to understand what's happening internally in this black box system, but the explanation system can experiment with it, run millions of simulation examples, try every input combination and see what's the output and see if it then can infer some logic that explains what the system is doing. So I have different teams working on all of these, often combinations of these. And these are the 12 teams working on the program up on the right. There's one team that's not developing a system. They're just a group of cognitive psychologists that have studied the psychology of explanation. There's a lot of research and education in decision making, et cetera, on when is one explanation better than another? When does an explanation contribute to a good decision? When does an explanation contribute to learning? So they're just there to dig through all that literature, find the most useful nuggets out of all of that and give it to the other members of the team. The other table you see here are 11 teams who are working on different versions of how to make these AI systems explainable. Three are working on both autonomy and data analytics. Three are working only on autonomy. And five are working only on analytics. And they're pursuing a wide variety of approaches. Some are heavy and deep learning. Others are only doing a model induction or rationalization technique and different combinations. So here, in terms of the deep nets, here are four different approaches that people are trying to make these systems more explainable. And I'll show you some follow-up examples of a few of these. First is tension mechanisms or salience maps, as they're called. And this is a technique that's even in use today, a lot of our researchers are trying to refine it to make it more effective. And that's basically, if this case, you've given this image to a deep net, it said, this is a picture of goldfish. And if you say, why do you think they're goldfish? It's highlighted exactly the pixels around the goldfish in the image. You can trace back through the net and see what in the input was it paying attention to. What data, what pixels in this case made it, contributed to it making that decision. So that can often be a very useful explanation. Like one of the researchers who did an early version of this did a test system where they trained it to distinguish wolves from huskies. And then they could ask the system, why did you make this, why did you call this a wolf? And in that particular case, they trained it in a way so the system highlighted all the snow in the background. So it, which is a kind of mistake these systems often make, right? It happened in all the photos it was shown. Every time there was a wolf, there was also snow. So that was a much easier feature for it to grab ahold of. And so you can siege from just this kind of salience map, get a lot of information about what the systems are doing. So that's one. The second one at the below that feature identification. And I'll show you a more detailed example of this is really at the heart of a lot of these techniques. As I mentioned, these deep nets are often discovering features on their own that are very useful and meaningful, but it just doesn't know what to call them. So we've got a variety of people in different ways. We're trying to find out, recognize useful features inside the net, find a way to associate those nodes with language or with labels or with some kind of description that they can give people. Modular networks at the top. This is, in general, because of all the emphasis on deep learning, there's a huge amount of work throughout the research community on how do you have better architectures for these? How do you set them up in a way so they might be more modular or so the architecture is more organized? So we're very much like in the early days of software programming in the 80s. People would write code, they used to call it spaghetti code. It would just point all over the place and it was a mess. Nobody could understand it. But eventually we learned how to write software so it's much more modular, much more understandable. Something like that is happening with these systems. Right now the deep learning systems are one mass of interconnected nodes, but eventually we'll find much more regular organized understandable architectures, I think, for these that'll make them more explainable. Finally, the most interesting one I'll call Learn to Explain. This is where the deep learning people are saying the way out of this is more deep learning. So if you've trained a deep learning system to make the decision, let's train a second deep learning system to generate the answer. In a sense what it's doing is being trained to find the useful features inside the first net and generate language or images or something to explain that to the person. So those are the four broad techniques. I can show you, oh did I? I didn't mean to have that slide in there. So here's an example of the salience method. This is one of the people, a group at UC Berkeley. What they're doing, you see this dynamic image on the left there. They have the image of the goldfish. They're randomly obscuring different pixels in the image and seeing how that changes the classification. So you're trying to find what are the fewest changes to the pixels that have the biggest change in the decision the net makes and that ends up giving you kind of a very precise heat map on what the net is paying attention to. So that's one technique. Let me see, did I? Well, I'll jump to this one. I thought I had a different slide in here, but it's an earlier version of the charts. Here's, again the Berkeley group, they're doing both the heat map and the way out of it is more deep learning. So what you can see here, you get the picture of the elephant. They're testing this system by showing users an image and a question and saying can you guess will the net get the right answer to this or not? So in the no explanation case, it just shows the person that picture of the elephant and says does this elephant have tusks and now the user can guess whether or not it'll get the right answer. In the explanation case, they get the image of the elephant. They also see the heat map so they can see the net is paying attention to it exactly the right spot. It's looking exactly where the tusk is. But then you look at the language generation that comes from the second deep net, it says because there are no bones sticking out from its mouth. So now you know it may be looking in the right spot but somehow it didn't pick up the right feature. So you can correctly guess it's going to get the wrong answer and that's what happens. In the second case, this is one where it gets the right answer. Let's see, is this a professional sporting event? It both highlights it's looking at the person's uniform and the verbal description is because they're wearing official jerseys. So things line up, you get an intuition that's going to get the right answer and it does. So this data at the bottom is just showing when they show these different situations to people if they don't get the explanation they can guess what the net will do about 57% of the time but if they have both of these explanations that increases the 70%. So it really does give people a better idea of what the system is doing to get a right or wrong answer. Now here's the same group. Now this is an application of a similar technique but to an autonomous system. In this case they've trained the system to mimic a human driver so if you will they've got a simple autonomous vehicle if you will. They can then watch what that system pays attention to as it's driving down the road. You see this red, green, blue heat map. You can see what is it paying attention to when it makes a turn. Independently they've crowdsourced with a lot of people on Amazon Turk. What features of a similar image do people think are important? Which parts of the image are important? Characteristics that are meaningful to the person. And then they can take the union of these two. They can see okay what did the net pay attention to. Which of these are features people thought were important and then generates a verbal description of why the system has made a decision. So now I'll just show some very simple examples of how this works. So here's the explanation. It's saying the car slows down because it's making a left turn. You can now see the heat map. It's a little obscured because of the heat map but you can see the car did indeed make a left turn. So in that case it was a good explanation. These go on a little bit. I'll show you a few of these. The car accelerates because the light turns green and the traffic is moving. So again you see this. Closer you can see the green light there. So it's picked up that feature that people thought were important and gives that as the explanation for what it's done. The car slows to a stop because the car in front has stopped. Pretty easy explanation indeed. That's exactly what happened. So this is fairly primitive. Just the first year of research but there's a lot of interesting ideas here on how you can dig into these nets and pull out explanations that are going to be useful to people. This is one where it actually made a mistake. I think it said it turned because it saw a stop sign but it was actually a do not enter sign so it kind of had the right idea but misrecognized one piece of it. I see I've got several. Let me... These are several other researchers that are working. Here's a group at UMass in a CRA that's learning to learn causal models using model induction. Here's a group at UCLA that learns something called a statistic and orgraph which is a more interpretable model. Here's a group at Oregon State that's working on an autonomous system. Another group at Park. They're interesting. What they're trying to do is they're taking a cognitive model developed by psychologists that try to mimic human cognition and they map what the learning system features into this cognitive model and then use that for explanation. Here's a group at Carnegie Mellon is learning a physics-based model of a system that will play Atari instead of just an uninterpretable deep learning model. Here's a group at SLRI doing more feature, a salience map. Here's the group at BBN that's using work at MIT where they're recognizing reusable features inside the net. Here's a group at UT Dallas that's using an interpretable model called Attractable Probabilistic Logical Model. A group at UT or Texas A&M where their test problem is actually detecting fake news so they not only detect it but then generate an explanation of why they think that was misinformation. Rutgers, this is an interesting one that is just a psychologist who's worked in an area called Bayesian teaching which is basically if you're trying to create a tutoring system for a person you want to select what's the next best training example to show the person that will correct his misconceptions or inform him which problem is the next best one for them to work on. So in the explanation case they don't try to understand internals of the model but they can look at all the training data so they try to pick out which piece of training data is the one that really caused the system to make this decision and often showing the training examples can be very informative. Here's our program schedule just say we're just about a year and a half through this program all these teams have just finished their first big evaluation we're kind of collecting all that information you know have a big meeting in February to kind of summarize all that and kind of see where we are and then we have another two years of the program to continue to develop the technology. This is how we're doing the evaluation in every case the teams are show their system to a user without the explanation show it to the user with the explanation also with partial explanations and then measure all those parameters I showed earlier to see if the explanation actually improves the user's performance and their trust and that's it. Thank you.