So today, well, probably more accurately, last week we released JRuby9000. Yeah. For those people who don't know the story, we couldn't come up with a version number, so we came up with a joke and the joke became reality. It's over 9000 was what Vegeta said when he couldn't believe Goku's power level. And we'll be over 9000 in the next week or so when we put out our first point release. So the big bullet points for 9000 is that we're compatible with Ruby 2.2. We have a brand new runtime, which we'll be talking about. In this release, we're bypassing the JVM a lot more and going down to C because the JVM is not quite giving us enough power and through a big effort, Oniguruma's transcoding support's been fully ported. So on our roadmap, we essentially have two branches that we support today. The first is what we're talking about today, which is 9000. Probably the week after we get back, we'll put out our first point release. But it's important to point out that JRuby17 isn't going away. I think there's plenty of people that still depend on Ruby19 support. So it'll be there as long as you need it. And every four to six weeks, we should put out a point release. If we don't, and there's something that's fixed, prod us a little bit and we'll get it out. This is probably the most contentious slide in the deck. So minor and security are fairly obvious and I don't think anyone would disagree with those. But you'll notice that we have two numbers now after that. So 9 really will be the major number. If we do something that's not going to be backwards compatible or if Ruby puts out a release of Ruby that's not compatible, we'll have to rev that number. But with that said, Ruby is not changing the language semantics to such an extent that we shouldn't be able to... Your code should still work when 2.3 comes out. So we're hoping that we can just make 9.1 the next release when Ruby 2.3 comes out. And we're not going to create another support branch. So if you have a strong opinion about this, come and tell us why you don't like it. It's not set in stone. We just don't really want to support more than two branches at once. So like we mentioned, the biggest amount of effort that we put into 9000 was to get compatibility that finally caught up with CRuby. And we feel like we've mostly done that modulo a couple of gaps. But it'll mean that we're now... We can mostly release in lock step. When Ruby 2.3 comes out later this year, within a couple months we should be able to do a JRuby release that has the same features. It's going to be much easier for us to keep up. We are leveraging both RubySpec, which is now an official Ruby Core project. I don't know if somebody saw the announcement. And they're test suite. They're normal mini unit, test unit test suite. We run as many tests as we possibly can to make sure that we're compatible. There are a couple known gaps that we wanted to talk through just so people know that there's a few things that aren't quite done yet as far as 2.2. So refinements. Is anybody using refinements or playing with it? That's good. So refinements are basically lexically scoped monkey patching. You can monkey patch only within a given file or within a given class. And it's very tricky to support both in JRuby and in standard Ruby. We didn't get it completely finished for the 9000 release, but it should come in an update soon. And for folks that aren't familiar with it, say we have a method add name that's just going to take presumably a string and add some additional text to it. Normally, this would be string plus, and do what you expect it to. But you can also define a refinement that will monkey patch that. So we have our string add scrub refinement here that refines string by redefining the plus method to do something slightly different. So in this case, we're just going to scrub both of those two strings before we add them together. And actually it occurs to me now this may actually be recursively a bad implementation, but you get the idea. We can monkey patch this so that it will throw a stack overflow. And so then you can throw this using line into a script body or a class body, and then only within that script or class will that monkey patch take effect. So to understand why it's a little difficult to support this, here's the logic for a normal method call. We've got a target object, we need to make a call. So we look at the class, look at its method tables and its parent's method tables, and find the method in there. Presumably it's there, we call it, and we continue on. To do a refined method call is considerably more difficult. We need to first grab all the refinements that are in scope, which has been used into that file, look for the target class in the list of refinements, look in that refined class for the method we want, and then if we find that, we can use the patched version. If we don't, then we fall back on the original logic. So it's very different from normal method dispatch and getting the semantics of this right have been challenging both for the core Ruby folks and for us, so it's not quite done. And the interesting thing about this or the unpleasant thing about this for me is that in the future, you're not necessarily going to be able to look at this code and know what that method call is going to do. You're going to need to look around in the same file, look around in the class, and see if those other monkey patches have been using into your scope. So it helps localize monkey patches, but it is still basically monkey patching and has a lot of the same issues that go along with that. Keyword args is the other thing that didn't quite get finished, although it's really, yeah, it's really only one problem, and we'll show you what it is in particular. You'll notice in Ruby spec, we actually have 13 failures. Ten of those is us not responding the proper number of times to two hash, which might be an optimization, but it's not related to two-two. And one is just we don't handle block binding. I'll talk about why block binding is so complicated later. So here's a typical keyword argument call, and we work. We replace the A2 keyword with one, because we're passing that. Yay, rest arguments is empty. If we switch this, and then we pass in a string key instead of a symbol key, we're broken. We should be doing this. We should be populating the rest argument with that hash and not changing the keyword argument. We actually change the keyword argument and have an empty rest argument. Now, what does this do? Just happen to notice this while looking into this problem. It's a really weird looking call. We don't work with this either, but it's the same basic problem, and MRI does this. It takes the left and right side, puts it into the rest argument, and then updates the keyword arg. So basically only the symbols actually go into keyword args. If you're mixing symbols and strings, you get interesting behavior like this. And so there's this extra splitting logic that we just need to add. All right, I talked a bit about this last year about how we've gone below the JVM to make better compatibility for IO, for process, and the transcoding stuff. I'll go through it just quickly this year a little bit. So one of the problems we have building Ruby on top of the Java platform is that Java's intended to have the same behavior across all platforms. And that unfortunately means it hides a lot of the platform specifics, and unfortunately a lot of the features we want in Ruby compatibility. On the other hand, Ruby's IO, Ruby's process, those really embrace the platform. They're mostly thin wrappers around a spawn call or read, write calls at the native level, and they will work like you would expect a C program to work. So if you're expecting the native platform behavior, it's a little difficult to get that on JRuby or on the JVM, and that's what Ruby needs to be able to do. So for this release, we went through the extra effort of using mostly native calls for most of IO and pretty much all of process. This gives us the best possible POSIX behavior, UNIX behavior of any JVM language. No one is going to be as compatible with standard UNIX process behavior than us. And it allows us to do things like this, which just are not possible with any other JVM language. So here we spin up three pipes in, out, and error for our child process. We're going to spawn just the cat program, the cat command, with the client side, the child side rather, of those in, out, and error. Then we've got a PID that represents that running process. We'll close the client side on the parent side, because we don't need that side of the pipe anymore. Write something to the in side of the stream that goes to the child. Close that out, read from the child's output, and the cat does what it's supposed to do. That's the line number and the text that we sent into it. And then the wait PID call at the bottom does everything that you're supposed to do, that you'd expect it to do. And most of these calls in here actually are doing a real native call down to the equivalent C function that MRI, that CRuby would use. So you get the right behavior. You can share these file descriptors in ways that other JVM languages can't. And there's a lot of work to make this all work nicely. We're going to do a little bit of video stuff much more compatible with the way Ruby does it now. The transcoding, Tom mentioned this was a big porting effort that's gone over many years. In JRuby 1.7, we basically emulated the CRuby transcoding encoding logic on top of Java's built-in support. And that worked okay, but it unfortunately meant that whenever we had to convert from one encoding to another, we had to pass it through a character array. Because all Java stuff goes to and from 16-bit characters. In JRuby 9.000, we actually finished the porting process of taking all of CRuby's transcoding encoding logic and ported it over to the JVM. It works great. All the compatibility is significantly better. And it should match the same behavior on CRuby and JRuby for even the weirdest encodings that they support. So we're really excited about that. It also plugs into I.O. properly, so as data is read off the line, it gets decoded into the right characters format, and there's a lot less overhead than our old JVM version. So we have a new runtime. Uninventively called internal representation. The seed of this came from Sastry, who's been working on it and working with us for the last five years. We knew that we wanted to make a change. We wanted to work with semantics and not syntax. This is the abstract syntax tree for the code on the left. And JRuby 1.7, we basically, when we start interpreting, we just start bouncing around through that tree. When you generate Java bytecode, we start bouncing around through the tree. So what we're doing now is we've created a set of instructions, like a virtual machine, and now these things represent our semantics. If we look at line 8, you can see we're calling the plus method on the receiver A and giving it C, A plus C. So this is fairly easy to read. We also wanted to make things more approachable. So if you've ever taken a compilers course, nothing here is going to look unusual for us. We have a lot of change on the left side. We have the typical phases of a traditional compiler design. If we look at the right, we see a lot of common vocabulary, abstract syntax trees, control flow graphs, basic blocks. You get the picture. So we do actually have an example of this. Sean, who's been contributing a lot to Rails, found a performance bug in both JRuby and MRI and JRuby. C, Ruby and JRuby. So this is the method. It's in Rails. It's an active record. When you use a block parameter and you go and pass it on and you don't use it, and JRuby before this optimization and MRI today, it'll actually create a PROC instance, even though it doesn't use the PROC instance. And this has some overhead. Oh, actually the numbers are right there. So in two or three days, we worked through the issues with Sean. He made a compiler pass and we got a nice bump, like six million more things in the micro benchmark, which happens to be MRI's actual performance. So that was very cool. And the evidence, if you go and look at this file in active record, you'll see that there's a version guard or an implementation guard. And we have the nice idiomatic Ruby and MRI has, like, this extra block to get rid of this overhead. So that's quite cool. Another realization that we've been making over the last couple of years is how much easier it's been to debug performance and correctness issues. When we find a problem, we just start it up in the interpreter, we step through it in the debugger, we solve it, and miraculously it fixes it in both the interpreter and the JIT. We're spending a lot less time looking below that opaque barrier of Java bytecode to figure out what went wrong. So now we'll it's where the wheels hit the pavement of reality here. How we're doing versus JRB17. The first one is the dreaded topic of JRB startup time. We knew in JRB9000 that we had a challenge because in both cases we make an abstract syntax tree, but in 9000 we additionally make a set of instructions. So this is clearly more work and it's responsible for us starting slower. We knew this years ago and we're going to fix it. We're going to fix it with IR persistence. So when we go and compile some Ruby code and generate IR we'll just go and save that IR away and then if we load the same file again we'll just load that IR and save all that work. Problem solved. Well, as it turns out the amount of data that we save in IR is quite a bit more than the original Ruby source and so it works out to take about exactly the same amount of time as just parsing it over. IR persistence itself we still use. We use it for all of our ahead of time compilation so we dump that blob into a Java class file. So it's still useful and we haven't given up on it, but for startup time this was kind of a dead end. So plan B, let's make this make IR thing as cheap as possible and in that case most of that time was spent doing compiler passes to optimize the code so here's where we stand today. We run IR builder which just makes one big long list of instructions we don't run any compiler passes optimizations on it and we had to develop a special interpreter to be able to run on that big long list of instructions. So that mostly solves the startup issue. If that method runs for a while then we run compiler passes and then run a better interpreter or full interpreter. Most of you will be running in mixed mode and so instead of running the full interpreter it'll just go off and generate Java byte code. So the actual numbers if you're running something that takes longer you won't really notice any difference at all. For things that are shorter it's usually about 10 to 15% slower. We're going to keep chipping away at this, but it is what it is. Memory is something they were getting a lot more aggressive about. This slide shows you that we're making continual progress on this. This is just running Rails console with a nearly empty app. You can see in JRB17 we're using about 68 megs of memory. When we put out JRB9000 we're using 88 megs and today on master we're down to 80. By the time we put out 0.1 we should be down about 75. So we actually will be using less memory in 9000. A few point releases from now, but not not yet. We'll talk a little bit about straight line performance as well. We didn't spend a great deal of time on this during the 9000 cycle. Our primary goal was to get compatibility where it should be, make sure the run time was stable, fix as many language bugs as possible, but we wanted to at least be roughly on par with JRB1.7 before we did our .zero release. So it should improve rapidly after this point. Now that we've got the main release out we can iterate on performance and should be able to go much faster. So here's kind of a mixed bag. Here's a few benchmarks where we're slightly slower. There's some benchmarks where we're slightly faster. Anything that uses a block dispatch heavily is a little bit faster in 9000. We do a little better job of optimizing it. So overall most things should be roughly equivalent to what they were in 1.7. If there's anything that looks bad let us know it's usually not a difficult fix. And in general most folks that are going to be running JRuby they're going to be scaling this stuff horizontally anyway. So you still can take your one JRuby instance and get as in the case here we'll be hearing more about this later today. Getting 900,000 records per second we've heard many folks saying that they're getting much better performance on certain sorts of applications. Nobody's reported anything that's severely slower. It's usually on par or better in 9000 for the most part. And of course we also have lots of folks that just get better scaling from JRuby either way. One group had several, I think it was 15 extra large AWS instances that went down to like 10 mediums when they switched from MRI to JRuby. So just being able to actually use all of the hardware you end up scaling out a lot better. Okay so this is back to you. So some things are going to be working on in the near term. The first one is block parameter binding represented in IR. What I mean by block parameter binding is in this first simple example we're passing one and two into this PROC and A gets one and B gets two. That's the binding. But of course blocks are more complicated and can be much more complicated. The next example we're destructuring a list we're stuffing a couple of the elements into a splatted variable and we borrowed from Rubinius's code base just because they represent all this block binding logic as a single method so it's a good way of showing the general complexity of block binding. So by the time you you can start to see some equations here. These equations are typically like figuring out where a certain class of variables are getting put along that thing and I I removed some comments and shortened up some lines so that would fit into four slides but the takeaway here is that there's a lot of logic. Like I pointed out earlier we're not even passing 100% of our block binding test cases. But what we want to do when we compile a block we compile that block we want to go and emit some code before that block to do the block binding logic in IR as instructions. Then let our compiler pass a churn on that and then pass it off to hotspot and those two phases are going to go and get rid of a lot of that complexity and fold away some branches and it basically works because we're manually inlining that logic into each block. Right now we have a helper method everything calls to that helper method and never inlines. IR also has an inlining system itself enabled. I'll mention that but we need to actually be able to emit this logic in IR if we want to inline blocks or more complicated blocks. And we're not going to boil the ocean we're not going to put four pages of logic in the front of every block because that's probably not going to work out. This is how things look today in particular all the compiler passes we run they're static and they're conservative we never have any reason to ever de-optimize this code but because of that we can't do more speculative or aggressive optimizations we have a profiler mode if we turn this on it starts to collect information and then we can start doing more aggressive optimizations. If those optimizations pay off we get much better performance if they don't then we have to back off and do something else. And some of those optimizations inline methods and blocks we actually have code for this the blocks that we support are incredibly simple but it did bit route a bit in the last few months as we were trying to get 9,000 out the door we want to be able to handle unboxing we have experimental pass for that as well this just means instead of when we make a Ruby fix num instead of making a Java object that wraps a field with a Java primitive long in it we're just going to interact with the Java primitive login by itself and get rid of all that allocation overhead invoke dynamic is fantastic tool for performance we found out that once it gets to a certain size the warm up is just too slow so once we can start profiling information we'll just only add invoke dynamic to the hottest parts of our apps and still get that great performance because of the changes to the startup interpreter our plans for how we deoptimize has to change this is probably our most significant thing we need to work on next and we want to reduce the cost we have an experimental profiler but it I think it takes like 4% of the runtime to collect the statistics and that's just way too expensive and figuring out what code is actually hot is surprisingly challenging so then on the JVM side of course we're going to continue to try and use what the JVM provides better use of invoke dynamic reducing the overhead structuring the calls a little differently so that they optimize well at the JVM level there are a few different libraries coming around the corner now the nashorn guys the oracle javascript implementation they did a big project to make it possible for integer code to just use integers and float code to just use floats and turn that into a library that ideally we'd be able to use to get the same performance the same specialization so that when you write a numeric algorithm in JRuby it would optimize down to the same numeric code if you wrote it in Java VM boiler is another little utility library kind of experimental at this point but lots of JVM folks are working on that problem of getting rid of numeric objects and turning them into numeric values for Java 9 and beyond I'm going to be working with some of the oracle folks on the value types JSR there will be a value type in the next version of the JVM this basically means it's not a full on object it's just part of the call stack it would save a lot of overhead doesn't put any extra hit on the allocating on the GC it's pretty close in performance to what you would get with unbox numerics and then there's also going to be a JIT interface hopefully it will get into Java 9 that allows the new Gral JIT to plug in which will work with the truffle stuff we're going to have someone talk about in just a little bit the last thing that we really want to do is continue to improve how we present our continuous integration how our disk artifacts our nightly artifacts are available one of the big areas is that we would like better with windows support is anybody here using windows that's usually with you okay so there are a couple we want to have our tests run just as well on windows as they do on the Unix systems it's mostly a matter of getting it set up spending a couple weeks working through the failing tests and then having continuous testing on that platform we have a nightly disk but right now it doesn't have all of the different release artifacts the windows installer it's missing the complete jar with all the standard library in it so we want to get that improved but you can go to ci.jruby.org and every night there will be an updated build of 9000 and an updated build of the 1.7 line and then because we're using so many native calls we need to do a little bit better job of testing that native support on various platforms we have it running in Travis on Linux we try to run it periodically on OS 10 and a couple other platforms but there's more areas that people are using jruby that we need to test against and at this point we're going to bring in our special guest to talk about a bit more distant future work in jruby and I'll introduce Chris Seaton from Oracle thanks so much for lending me some of your keynote time so I'm Chris Seaton I'm a research manager at Oracle Labs and a PhD student at Manchester in the UK and we'll talk about the truffle backend in jruby I'm going to focus on going from a research project more towards being something that's a product that could be used Oracle wants me to warn you that this is just a research project so you shouldn't buy any Oracle stock or products based on what I'm telling you this is kind of the point of the talk about going from research so what is truffle truffle is a new backend for jruby it's quite technical I could spend a lot of time explaining how it worked but in general the idea is to take an AST interpreter for ruby like jruby used to have in the 1.7 branch like MRI used to be in the 1.8 times but we take that AST and then what we do is we gradually type it as it runs so as your AST runs we see the types that involve such as string or double or integer and then we type the tree so as the program runs we replace nodes in the tree with these specialised versions this is an algorithm we call AST specialisation allows us to improve the performance of your program as it's running but the really clever bit is the next thing we do is we take that tree after it's been specialised and we compile it to machine code unlike the way that jruby currently works we have our own JIT compiler written in Java and because it's written in Java we can use it as a library we can take all the nodes involved in your AST and produce a single piece of machine code at the end of it this means that we're no longer having to negotiate very carefully with the JVM we're trying to trick it to doing things by using specific byte codes trying to see what it's doing we can simply tell it exactly what we want to do we can generate exactly the code we want and this produces a really high performance implementation of Ruby we've been talking about it for a couple of years and generally these are the results we've been showing synthetic benchmarks so we've got classic benchmarks here you might have heard of such as n body, pi digits, delta blue and these are reasonable benchmarks but they're not really very realistic Ruby code so fankov is a program about flipping pancakes n body is an astronomical simulation and delta blue is a constraint solver for logic programs so we do really well on these programs the implementation, this is all relative to MRI so the numbers off the side how much faster is than MRI jruby is in blue rubyneus in silver topaz which was the pi pi implementation of ruby is in green and with purple so sometimes we're doing 40 times faster often around 10 times faster and that's great but these aren't the sort of programs you people are interested in running we've produced tons and tons of research from this so these are some of the papers we've produced while writing this so if you want to know more about the technology this is where you can go and look we've got papers on how we support C extensions how we support object model how we do all sorts of stuff and hopefully this is going to be summarized soon and my phd thesis when I manage to finish this I'm going to show you the whole thing but we want to start talking about running real things instead of talking about research you guys aren't interested in research papers so much as you want to be able to use it if it's fast you want to be able to use it that's great and so far you haven't really been able to do that we're building a large team to work on this there's now four people working full time implied by Oracle on jerry wichoffers myself, Kevin Menard who's an existing contributor Benoit de Lowe's who's an existing jerry wichoffer and Peter Halubu's based in the Czech Republic and then these are other people who are occasionally helping us on specific parts of it or who are working on object compiler so Oracle's putting a lot of resources a lot of research power a lot of time and money into this implementation I think we're now the biggest employer of Ruby implementers anywhere in the world so we're really putting lots into this and we're moving towards making it work so we do now support a significant proportion of Ruby we support 93% of the language specs according to Ruby spec and 89% of the core library specs obviously there's a lot of work still to be done there in that last 10% or so but we're getting to the point now if you've got a normal Ruby program it will likely run if you've got lots of libraries it probably won't run but your general Ruby program is likely to start to be able to run what do we actually run so as I said we run Ruby spec we run most of the standard library including things like JSON, test unit it started to run some of the infrastructure of the Ruby ecosystem now such as webbrick, rack, RSpec the Redis driver we run tilt, it'll be part of concurrent Ruby we even run things like Sinatra and Roda and some interesting libraries like Chunky and PSD when I say we run it I'm using quite a broad definition there I mean the core functionality that you want to use those libraries for to do something that runs so you can boot up Sinatra and you can serve a page not everything's going to run but we've got that core functionality that's a big step towards doing that what we're going to use to do is start to do this to set up some demonstrations of applications you could run over time so you may think that the performance gains we had on the simple benchmarks would go away when we start running real code and we've really found that hasn't been the case this is a composite so across lots of different benchmarks we've taken from Chunky PNG and PSD to RB now these are extreme cases we're not going to get this performance on everything but these benchmarks exercise what we do well really really well so here we've got across 43 different benchmarks you can run them yourself from these links so this is again relative to MRI and all the other implementations of Ruby don't do much better than MRI here that's because these benchmarks heavily do allocations they do small little data structures used temporarily they do metaprogramming stuff like that no matter how much effort is put into them they haven't yet be able to see and optimise through this stuff and we are so across all these benchmarks we're running almost 40 times faster than anything else and this is real code from Chunky PNG and PSD it's not artificial benchmarks and I think this is where the promise of our performance is going to be applications like Rails do lots of metaprogramming they do lots of small data structures and that's where we do well and that's where we get these performance numbers from but Chuffle's much much more than simply being able to run JRuby a bit faster than he did before we have always enabled features such as Set Trace Funk although I know Thomas is working on that for normal JRuby now and object space each object they've both always enabled we've got revolutionary performance of things like metaprogramming I'll show you an example there in a second high performance C extensions we're creating a new implementation of C extensions to bring that back to JRuby we do it by interpreting the C code and it's actually faster than running the native code in the experiments we've done we're looking at interoperability of JavaScript, R and C in other languages so we've got a team doing JavaScript and we're looking at being able to call Ruby from JavaScript and JavaScript from Ruby the same of C the same of R and it's really really fast we can inline between the two languages we're looking at revolutionary debug capabilities the debugger in JRuby Chuffle is always enabled you can stop the program at any time and debug it in just a second and we're working on a statically linked binary builder JRuby Chuffle so there's no JVM dependency this will solve the startup problem we had a build a couple of years ago it's sort of bit rotted since then but we can start in about 14 milliseconds to run Hello World similar to whatever I can do so you don't need the JVM, you simply get this one big binary that is JRuby Chuffle that is a research project still that's not coming anytime soon so this is an example of some of the optimizations we can do that are particularly impressive you can run this yourself if you go to this Google link but it tells you what we can constant fold constant folding is one of those operations that removes a lot of the work in your program so this demonstrates the power of our constant folding so the example first value is simply 14 and yes of course we can constant fold that because it's a constant we can also constant fold 14 plus 2 we can do that by speculating that the method won't be redefining constant folding the arithmetic or eval so you can eval 14 plus 2 giving a string and we can constant fold through the eval statement noticing that the string is constant and we can remove that we can also constant fold if you met a program and then send the operator yes we can constant fold through that as well we can even do it if it's a string and get this we can even do it if the method name is calculated by doing L minus exclamation from the ord and then call it yes we can constant fold that as well we can see that expression is constant you may have heard about proc binding being the worst feature in Ruby and it's impossible to optimise nope we can optimise through that as well so if you see x is 14 create a new proc and we don't even reference x inside that proc but then we get the binding of it get the local variable so again there's a meta program we look up to and yes we can constant fold that to 14 of course we can't constant fold everything but we can do it ourselves we'll find some things work some things don't but this is sort of optimisation level doing this is world leading research in dynamic languages other dynamic language implementations including things like v8 don't do this this well so now Ruby is getting the best research there is on these sort of implementation issues frequently asked questions when will truffle run rails probably in the next year to two years we're working on it very hard we've got about half of active support running so far performance worse than it before it is at the moment again we work on that it's not something we've put time and effort into because our benchmarks and our targets we're looking at our peak performance but we'll work on that and with things like the static build that will go away will grail be available in a proper open JDK release as Charlie mentioned there's a JEP 243 I think which looks to put the interface that grail needs into JDK so hopefully in JDK at some point grail will be available are we using code for rubinus? yes we use the core library implementation from rubinus that's not any kind of political statement we simply find it easier to optimize ruby code than we do java codes we're using the rubinus code from that should you try truffle your application isn't going to run using truffle as simple as that at the moment but if you run a library or you've got a very simple application you'd like to work with us to look at if it'll work then we'll be interested in that but if it's too slow normally or if it's too slow normally is truffle going to replace normal JRuby? we're not out to form a coup and take over JRuby Oracle is working with the JRuby people it's entirely with the JRuby projects leadership's consent we do have a lot of people working on it and we do put a lot of commits in but we're doing greenfield development whereas the rest of the team isn't my personal technical opinion is that within a couple of years we'll have enough to replace the current back end in JRuby but I'm not advocating back at the moment and if that happened it'll be because of the community and the leadership decide that's what to happen so we're not looking to force it upon anyone and as I said it is just a research state problem so please don't buy any Oracle stop or products based on what we're doing thanks very much alright again thanks everybody for coming out we're really excited that you're here we're going to just jump right into the next talk and try to get things moving along I'll hand it off to our MCs here


Hello, everyone. Welcome to Options with Doug, streaming live daily on Bookmap Discord and the Bookmap YouTube channel at 1.30 p.m. Eastern Time. Before I get started, I need to go through the disclosures. General disclosure, all Bookmap limited materials, information, and presentations are for educational purposes only and should not be considered specific investment advice nor recommendations. Risk disclosure, trading futures, equities, and options involves substantial risk of loss and is not suitable for all investors. Past performance is not necessarily indicative of future results. The focus of my presentation and the focus of the Options-Doug chat channel and Discord is options, order flow, the impact of options markets on stocks and futures, and the influence of market maker hedging flow on price action. I have a two-step process for trading and the first is planning, and I use positional analysis. I look at how traders and market makers are positioned in the options market and how those positions change from day to day to develop a thesis regarding the expected trading range and volatility for the day as well as a directional bias. And the second step in my process is execution. And I look at real-time order flow in Bookmap and real-time market maker hedging flow in SpotGamma Hero to confirm my thesis and for setups for entries and exits. And just to be clear, I will be talking about setups today. And setups can be taken any number of ways. I will be talking about a setup and an underlying asset, and again they can be taken any number of ways with shares, futures contracts, or options contracts. And questions and comments are welcome. And I will be watching the Options-Doug chat channel and Discord and the chat in YouTube for your questions and comments. And second wins in YouTube says hi Doug, hello. Is there a way you can turn up your microphone? And sorry, I don't know what the issue is with YouTube. There's never a problem in Discord. And I have the gain on my microphone turned up to max. So if you're having trouble in YouTube, I invite you to join us in Discord and Bookmap Discord. It's free for all whether you subscribe to Bookmap or not. And there's often interesting discussion going on all the time, a variety of channels in Bookmap Discord. So again, I invite you to join us there. I haven't heard any complaints about audio. My agenda for today, I'm going to go over news items. Actually not. There are no news today. And as far as I know, no news for the rest of the week. Then I'll go through my positional analysis. And then I'll review a couple of setups from this morning. Futures and stocks. And then we'll go and watch the live market. All right, let's get started. Start with positional analysis. Let's take a look at Bookmap. This is the S&P 500 Futures. ES Futures and Bookmap. And before I dig into this chart, I'm going to take a look at a larger time frame. This is the SPX in a 30-day one-hour chart showing price and key levels. Let me point out some levels here. First of all, here's the lower and upper edge, the expected move for the week, shown with the dashed purple lines. And so far, it looks like for the week that there is SPX is trading right in the middle of that range. And then the dashed blue lines are showing the lower and upper edge of the expected move for the day. And again, SPX is trading just about in the middle of that range. And there are also some key gamma levels on this chart. First of all, the put wall. That's at 4,000. That's the strike with the largest net negative gamma that can be expected to act as support. And that is certainly not in play today. And then the next level is the 4195 volatility trigger. And that is an important level, that is spot gamma's proprietary gamma flip level. Below that level, market makers position on the gamma curve is negative. In a negative gamma environment, they have to trade with price to hedge their delta exposure. And that tends to enhance or increase volatility. And then on the other hand, like SPX is trading now above that level, market makers position on the gamma curve is positive. In a positive gamma environment, they have to trade against price to hedge their delta exposure. And that tends to subdue volatility and can lead to more range days like today. And the next level is the 4300 level. That's up here. And that is the call wall. That's the strike with the largest net positive gamma that can be expected to act as resistance. And that is also the absolute gamma strike, the strike with the largest absolute negative and positive recall input gamma. Right. So those are the primary daily levels as well as the daily and weekly expected range ranges. Let's take a look at one other chart. This is SPX, again, in a one day, one minute chart. And the point here is just to show a closer look at SPX and the levels that are in play. And here's the, here are the levels up above the call wall. Absolute gamma strike at 4300. And then this L one level, large gamma one, that's an important level at 4299. And that did act as resistance yesterday. And the, again, SPX is trading just right in the middle of the range expected for the day. Let's take a look at book map. And book map, I have the SPX levels shown in my cloud notes as well as the spot gamma notes. And the again, here's the call wall at 4300 and the large combo one level at 4299. And then the SPI call wall is just above that at 430. And in my cloud notes here in this column, I'm showing key SPI levels that have been in play for today. And then the SPI call wall, here's the, so the SPI levels, the 426 did act as support this morning. And note, the SPI is trading just above the 427 volatility trigger. And this level has acted as support today once price moved up above. Right, so those are the levels that are in play. And again, in my cloud notes, I'm including the SPX levels as well as SPI levels. Let's talk about shifts in levels now. And there were some confusion in the spot gamma data this morning. So the, I think the only valid shift for the SAP 500 was the SPI volatility trigger shifted up one point from 426 to 427. The data was showing that the SPI put wall shifted up from 400 yesterday to 425. That may be the case, but the absolute gamma levels do not show that. All right, so that is the SAP 500 again, the only shift that so there were no shifts in the SPX. And no shifts, especially important in the call walls. The 4300 for SPX remains the call wall, the upper, upper limit. And then the 430 remains the call wall for SPI. All right, let's take a look at NASDAQ now. And let's take a look at QQQ levels. And this is showing that the NASDAQ also trading a narrow range today. QQQ trading between the 354 volatility trigger, which is a new level shifted up from 349 yesterday to 354 today. And then also this 355 combo L, L5 level. So price right now trading between those two levels in a narrow range. And as we'll see in just a few minutes, gamma notional is positive for SPX and also QQQ, leading to a thesis of it, a range day today. And that's what appears to be happening so far. All right, so here's SPX. And again, I have the same cloud notes here. And in my cloud notes, I'm showing QQQ levels. And there's the 354 volatility trigger, also QQQ, NQ round numbers, and then also NDX numbers. And there is the, assuming this number's right, there is the 14,480 football for NDX that did act as support today. And we'll talk about setups in a few minutes. So again, for NQ, a choppy range day as well. All right, let's take a look at, take a look at the Vana model. And this will give us more clue as to how market makers are positioned. So this is SPX, the Vana model. And this is something that I look at closely every day. What this chart is showing is how market makers delta notional, shown on the vertical axis, changes with changes in price, shown on the horizontal axis. And that is shown by this light gray curve, showing how market makers delta notional changes with changes in price. And this is showing that as price increases, market makers will need to sell futures to hedge their delta exposure. And again, that tends to subdue volatility. And this is typical of a positive gamma environment. And then the second curve, the purple curve, adds implied volatility to the equation. So that's showing how market makers delta notional changes with changes in price and implied volatility. And that change in delta with a change in implied volatility is the Vana effect, hence the Vana model, the name, the Vana model. And what this is showing is that price increases, market makers will have less delta notional to hedge, slightly less, as predicted by the delta only curve. And then on the other hand, as price decreases, they will have much more delta to hedge as price decreases. And that's typical of a negative gamma environment. All right, let's take a look and see where SPX is trading right now. And give me just a moment. Take a look at a watch list. And I've got SPX at $4273. So let's locate that on this chart. So that is about near the bottom of this curve. So showing that if price increases, there's really not much of a tailwind, a Vana tailwind. And as price continues to increase, there will be a Vana headwind, $2273. And then as price decreases, market makers will have to hedge pretty aggressively to hedge their delta exposure. They'll have to sell futures to hedge their delta exposure. So this is how market makers were positioned on the gamma curve at the beginning of the day. And this is showing how they're going to be expected to react with changes in price. And let's take a look at some data now. Let me just refresh this to make sure this is correct. All right, what I'm looking at here is gamma notional for SPX spy NDX and QQQ. And this is showing that gamma notional for SPX is quite positive 916 million. And for spy it's slightly negative. And that was a pretty substantial shift lower from yesterday. So yesterday, gamma notional for spy was positive 563 million. And now it shifted to minus 92 million. So a pretty significant shift lower to negative for spy. And then for QQQ, it is still positive. All right, so that's how market makers were positioned on the gamma curve at the beginning of the day. And when we looked at the QQQ chart, we saw that QQQ was trading slightly above the volatility trigger. And then spy is close to its volatility trigger as well. All right, so based on this, my thesis for the day was looking for lower volatility and a trading range. Definitely not a trending market. All right, so let's take a look at some setups now. We'll start with the S&P 500. So here the S&P 500 is trading just at the 427 spy 427 volatility. trigger. All right, let's take a look at a hero. What this is, this chart is showing, this is spot gamma hero, HIRO, hedging impact real time options. And this is showing price with a white line for SPX and options trades and market maker hedging flow or hedging pressure for a combined signal for SPX, spy, XSP, and ES futures. And so far today this number is positive. The notional value is positive 678 million, meaning that's delta notional net for the day. Traders are taking positive delta positions in SPX, spy, and ES futures. Let's take a look at the individual components. First of all, SPX, and now that is slightly negative. Spy, that's positive. Traders have been taking positive delta positions throughout the course of the day. And then for ES futures, that's negative. So net net, the total signal is shown here. Let's zoom in on this chart. All right, second wins ask what happens at the volatility trigger? All right, so the volatility trigger is for index products like the S&P 500 and NASDAQ, the SPX, spy, NDX, QQQ, RUT, IWM all have a volatility trigger. What spot gamma assumes is that for these products traders are long puts and short calls. Market makers take the opposite side of the trade and they are short puts and long calls. So when the short put area of the gamma curve below the volatility trigger, their position on the gamma curve is negative. Negative gamma in negative gamma environment, they have to trade with price to hedge their delta exposure. And then above that level in the call region, their gamma is positive. That's above the volatility trigger. In a positive gamma environment, they have to trade against price to hedge their delta exposure. So the volatility triggers for index indices like the S&P 500, NASDAQ and Russell 2000 and that's the gamma flip level from negative below to positive above. All right, so let's take a look now at, again this is the combined signal for the S&P 500 and we're going to do a setup review. So here I'm going to zoom in on the morning session and there's a very strong correlation between hero and price action. So options traders were having a strong, it appears to me that options traders were having a very strong influence on price action. Hero rising and price rising. Let's go take a look at ES futures now. So this was the setup from the morning confirmed by hero and so we'll take a look at order flow now. So second wins asked, so if the price is under the volatility trigger then downside momentum is likely to happen. It certainly could, yes. We looked at the Vana model for SPX and as price starts to move lower and implied volatility increases, price moves under that volatility trigger then price can accelerate to the downside. On the other hand, it can work both ways. If price is rising and implied volatility is decreasing, market makers delta notional is decreasing and they can buy back their short hedges. So I think the way spot gamma looks at it is they assign a little bit of a bearish edge for price below volatility trigger and a bullish edge for price above. Alright, so we're taking a look at the SP500 right now and so we know from looking at hero that traders were taking positive delta positions and I'm going to review the long setup from this morning. Very clear setup, a check of the 426 level and note the shift in order flow. All these green volume dots, aggressive buyers coming in shown by this rising dark blue line that's cumulative volume delta that steadily increased from 426 all the way to 428 and also this rising yellow line, buy stop orders helping to fuel the move higher and these are shown with these small green dots, buy stop orders. So aggressive buyers, options traders and buy stop orders all fueled the move higher. So that was the long setup from this morning. Very clear in hedging flow and order flow. Alright, let's take a look at NASDAQ now. There's NASDAQ in book map and let's go to take a look at hero and see what options traders were doing in NASDAQ and just like the SP500 this is also a combined signal here for NDX and QQQ. I'm going to zoom in on the morning and this was actually a divergent setup this morning. So the NASDAQ, notice the price drop quickly shown by the white line and just after a couple of minutes after the open, options traders started taking positive delta positions and that led to a reversal or helped contribute to a reversal right there and let's go take a look at that in book map and there it is. I'm going to zoom in. So here's the reversal at the QQQ 353 level as well as the could be the NDX 14,480 put wall level if that is the put wall. Again, there was some confusion with the data this morning. So anyway, just reading this in order flow bit here. So the clues in order flow, first of all, notice the iceberg orders here, iceberg orders as price moves down that's shown by this rising light blue line, two orders for 155 contracts executed and then here right at this reversal level it looks like 481 contracts executed in 14 transactions. One thing I like to look for is this gap here. Iceberg orders executing, we know that options traders are taking positive delta positions and then order flow shifts, a lot of aggressor buyers start to come in, you can tell by the green volume dots and then CVD starts rising as well as the yellow line showing buy stop orders fueling the move higher. So the move higher confirmed by options traders with hero and order flow and book map and price targets at the 354 and 355 level. So nice long setups in the SMB 500 and NASDAQ. Let's take a look at a couple of stocks and then we'll go to the live market. First was AMD. Let's take a look at hero and we'll see the call buyers were active. There's AMD and I've separated output and call transactions. The orange line is showing that traders are buying calls, market makers sell the calls and they have to buy stock to hedge their delta exposure and that can cause a stock to move a lot higher and notice how price levels off when traders stop buying calls. So that's AMD. Let's go back to book map and that led to a sharp rally this morning with a price target up to 125. Note all the liquidity at that level. That is the call wall, yes. So that is the call wall for AMD and that would be the target if price had continued or continues higher this afternoon. If traders start buying calls again then price could go up to that level. So that's AMD. Let's take a look at Google and note that Google was trading below the 126 put wall this morning. I think there was an alert. Here it is. Google put wall breached at 4.10 a.m. So early this morning, pretty much soon after the trading session for stocks began, the pre-market trading. But now Google, let's go take a look at book map and we'll just see that traders are buying calls in Google as well. Let's go take a look at book map. We'll take a look at Google. So remember 126 is the put wall. I don't have it shown on this chart but that was shown on the hero chart. And price made a series of higher highs and then found support at the VWAP and 126 put wall before moving higher. So that 126 level did act as support as expected. So that's Google and let's take a look at Meta now. And I'm just showing a few setups from this morning just to highlight and we'll take a look at the live market in just a moment. So here's Meta. Strong rally in the first hour this morning. Take a look at hero. And again, call buyers, driving price action. Traders are buying calls shown by the positive delta notional there. They're also selling puts but more aggressively buying calls. So both numbers are positive. And second wins ask, can you look at Boeing and Spotgamma? Yes, we will. I don't have it in book map. So we'll take a look at Spotgamma and just a moment. So there's Meta. Again, strong rally this morning as traders were buying calls and selling puts. And note the 280 call wall up above. And looks like their price would have to go about five points higher to get up there. So that really does not look like a feasible target today. Let's go back to book map. So there's Meta. 280 call wall well above. Now price is rolling over and the discord Bob Lee asks what type of chat is that? And I don't understand your question. I'm sorry. All right, so let's take a look at let's go back to hero. And we'll take a look at Boeing. And it looks like there was news. So I don't that there was what a this appears that there was no lead effect from options trades. So this just appears that apparently some news came out. So a drop from the just around the 210 211 level down now to just above the put wall. So a 10 point drop and Boeing again must have been news. I'd really I don't follow Boeing, but that is a big drop. All right, those are the stocks that I want to highlight. If anybody else has any stocks, they want me to take a look at. Please let me know. Otherwise, we'll go to the live market here. So this is the SB 500 first. Let's go take a look at book map and a book map. Now the SB 500 is trading below the 427 volatility trader. And remember the spy was already negative gamma, slightly negative gamma to begin at the beginning of the day. Order flow also looks somewhat bearish here CVD sloping down sell stop orders helping to fuel the move lower. It looks like some iceberg orders. Large traders are coming in with iceberg buy orders shown right here also by the rising light blue line. Let's see what options traders are doing. So this is a one day rolling windows. So this is cumulative data for the entire day. And it looks like I just like to look at the trend here and the trend of hero is down. And let's see if we can gain any insight by looking at the zero DTE traders and the number looks pretty small compared to the traders trading all expirations. Let's turn that off. And so we're going to look at all expirations. Let's change the rolling window period. So right now if we look at 30 minutes of data, like a 30 period moving average, it looks like options traders are now starting to fade this move. Taking positive delta positions. And it looks like price may be responding. And remember we saw in book map that larger traders were coming in with buy iceberg orders. So let's go take a look at book map. So hard to say if this is a this is just a pull back in a downtrend or reversal. So now some sell iceberg orders coming from a small about 100 contracts executed. CVD and stop orders still look bearish. Look at NASDAQ and the insight and pretty similar reversal higher at the QQQ 354 volatility trigger. Again, we're on a 30 minute rolling window period here. So it looks like options traders started taking positive delta positions just around 130 135. Right? So we know that traders are now taking positive delta positions in the S&P 500 and NASDAQ. One thing to point out is when I'm trading I don't have to I'm here I'm presenting on one screen. So I have to jump back and forth and it can be a little bit confusing. When I'm trading I have multiple screens so I can watch a hero and book map at the same time. So NASDAQ just back in right in the middle of that QQQ 354 to 355 range ES just above the SPI 427 volatility trigger and VWAP shown by this light blue line could be resistance. It acted as resistance before back to hero and see what options traders are doing. So continue to take positive delta positions for NASDAQ. Same for the S&P 500 and this type of environment. One thing that works pretty well is trading something like an iron condor with short strikes just around the lower and upper edge the expected move for the day and then wings further out whatever your risk tolerance is 5, 10 points you know whatever with SPX trading right in the middle of the daily expected range. Right? So now it looks like options traders starting to either take profits or slowing down price responding. Let's go back and take a look at book map.