Today, as a first session, we're going to hear about RADAR. This is a reversing framework that has been gaining an immense amount of traction over the past few years. It even managed to get itself on to Dr. – sorry, Dr. – Mr. Robot. I hope you guys saw that series. It's great. It has a reputation, however, of being kind of fiddly and difficult and just hard to use, which I think is unjustified personally. But to dispel all of these myths, we have the author himself, Sergei Alvarez, also known as Pancake, also known as Truffaut. And he's going to tell us all about it. And please give him a really warm welcome. Hello. My talk is about domesticifying RADAR. I will try to explain a little bit what's RADAR and what are the reasons behind this title. As he already presented me, I'm Sergei Alvarez. I nickname other people in the internet knows me as Pancake. I'm actually working at Nausicure. I'm doing research and development on mobile security. I'm the author of RADAR1 and RADAR2. And also many other tools like ACR, Balabind, and all of them open source. I've been working in many things. I've been messing with Bluetooth. I've been coding assembly for video codecs, optimizations, doing IoT firmware development. I've been also participating in CTF DEF CON in the sexy pandas team. Also, I've been doing forensics, working as a sysadmin. I'm mainly doing stuff at low level C, but also I manage to do things in the website. What's RADAR? It's an open source service engineering framework. I tried to focus on being portable, extensible, and expressive. It's a hobby project. I don't really get money out of this. It's something that I started just because I had some personal needs. I had to recover some files from a hard drive. And I decided to, at the time I was working as a forensic analyst, and I decided to implement a simple accessible editor for looking for some patterns and dumping the results. After this, I started doing some crackmys. So I decided to implement the disassembler, the debugger, et cetera. And I was extending the tools for every time that I needed something. I made Fulton write three years later, mainly because it was like a big blob. And I decided to make it more modular, implementing different libraries and supporting different scripting languages in a better way instead of just launching the interpreter from inside the project, just exposing the APIs and allowing other libraries to use the project itself. There was really few contributors until three years ago. That's something that actually there are a lot of users, many contributors. Every day I have like three, five full records to review and merge. There are like 500 users in Telegram and IRC. More than 5,000 users in following in Twitter. And this year I organized the first congress about the tool in Barcelona. And this is the third year that we are organizing the Assemble of Code competition. So the project is really active and there are so many people behind. What's the name stands for Row Data Recovery? The reason is because I started the project as something for N6, not something for static analysis or debugging and everything like that. So it comes with an accessibility editor, which is the basic functionality, but also supports assembling and disassembling. There are different vacancies for this. You can implement new plugins for each of these functionalities. It supports a lot of file formats. Mainly, if they are broken, they are also supported. So it's pretty safe right now. It supports mainly static analysis, but there are also capabilities for doing dynamic analysis. It's also able to do hash check zooms, compute entropy, and look for differences between different files. It's able to debug. It also comes with an emulator. And there are APIs for writing plugins to extend. It also comes with an emulator. There are APIs for writing plugins to extend with other emulators, like Unicorn and things like that. It also comes with functionalities for exploiting, like RobGadget's Finder, Payload Generator, Debrushin Patterns, et cetera. It supports scripting and there is like a package manager, because actually there are so many extensions and plugins that people use and I don't want to bring them all together inside the core. So it's better if I can just provide a simple way for other people to install these plugins. What can you expect? Well, this is some examples of things that people is already messing up with, rather. There is Pokemon, ROMs for Game Boy, Street Fighter, same for iWords, DNA Sequencers, Apollo 11, CPU, et cetera. But wait, don't lose the rail. We try to focus on myth, so we should explain a little bit which myths people think about this project. Many people think that this rather is not stable. Other people think that it's difficult. There are so many comments that are difficult and hard to remember. Other people think that it's buggy. It's not able to compile. The bugger is broken. There are so many complaints like this. But first, let's try to make a poll. How many of you know this tool? I guess everybody checked the website before coming into this talk. Know how many of you already use the tool? Half of the attendees. That's pretty good. Okay, so let's talk about the first myth, which is it's difficult. This is a graph about the learning curve, starting from the point that you start opening a file and the point that you start doing something useful with it. In fact, it's pretty hard to learn this and use this tool, but you can compare this learning curve to other projects like JIT, Beam, Perl, et cetera. The reason behind this is because there are so many comments, but there is a logic behind them. If you understand the logic, it's pretty easy to start learning and testing things for your own. If you understand the logic behind them, you can create more expressive comments. You can make different comments and create new functionality with it. You can easily hack into the code. It's pretty easy to find the place that you want to modify and just change anything in there. You can create plugins. You can write any script in any scripting language for extending the functionality or just creating a specific analysis for your target. You can also build tools on top of it. So it's pretty easy to hack on top of this, but let's focus on the difficult part of the comments. Comments are mnemonic comments. This means that they are constructed by letters. Each letter means something, and they are structured in a way that the first letter means what's the reason for the comment, like, for example, the letter P means that it's for printing. The second letter means that what do you want to print. So you want to print in xdump. You want to print the disassemble. You want to print the octal, et cetera. It has a lot of Unix-like concepts behind this shell. So you can use pipes. You can use redirects. There is an internal grep. You can also less. You can indent JSON. All these things come inside the tools. You can also pipe to the shell, but the reason for this is because you can just ship the router to binary in any target device, and you will have a full Unix shell in there, more or less. You don't really need to ship less comments. You don't really need to ship other tools like JSON-PP, et cetera. It tries to be orthogonal. This means that you can mix different comments and get new functionalities out of there. So let's make a quick demo about understanding which comments are the basic ones. You only need to remember five comments. After this, you can start learning other modifiers or comments that can be helpful for your target. But the five basic comments are these. We can open the file. We can seek any place. Can you read it? I can make it bigger. We can seek, for example, this address. We can print this assemble of this address. We can seek back. As you see, all the sub-comments are in the help. So you can just append a question mark at the end of the comment and you get the help of the sub-comments. And you can also make relative seeks like this. The other comment is a P, which is for printing. We can print in different formats. You can print an exam. You can print a disassemble. You can print jazz instructions. And then the other comment is a double B, which is for writing. I will enable the catch because I didn't open the file for writing. So if I want to write something here, I will have to enable the catch. So I can patch, for example, the first instruction and we will see that the knob appears in there. So the double B command allows you to write things in there. You can write experts. You can write assembly. You can write the contents of a file into the current seek, etc. And then the last comment, and not for this less important, is the queue for quitting the program. We can see that there are some modifiers here. We can use, for example, this is something like this. We can grab. We can also grab for some specific strings. We can just grab for the calls. We can also count the number of calls that are here. If there is a huge or a long list of things, we can first ask if you want to print all these things. But you can also use .dot, which means internal less. And you will get scrolling in there. So that's more or less the first logic behind all these comments. We will have more comments after this. The other two are structured into different models. There is a liber directory, which contains all the different models. There are libraries, we can see them in here. There is like an air core, which links against all the other libraries. And each other library, it's linking against other ones. And RUT is the basic library on top of libc. So if you want to port to any other platform, you just usually need to patch on the non-portable parts, which usually is the RDA back that takes all the core functionalities for the bagging and for the specific platform. But you can also build without the bagging, so it's easy to port. This is more or less the structure of all the libraries. And each of these libraries allows you to create plugins out of them. They are on the P directory. And there is also the R directory, which contains all the binaries. So we can check this. And we can also see this into the liber. So we see all the different models in there. Okay. There is another myth, which is that it's useless for forensics. In fact, it was the first aim of the project. So there are some functionalities, and there are strong points for doing forensics on this. But there are obviously other tools which only focus on forensics, which is probably better for doing forensics. But you can do many things with it. Like, for example, opening these devices, you can open a SliceDep, SliceSDA, or a Windows device. You can also open the physical RAM device if it's supported by the kernel. There is also an R2K module which works on Linux and Windows. It allows you to read physical and virtual memory from the kernel. It's a kernel module, and you can interact with it. You can find port for patterns and do the results. You can mount file systems, and it's also able to understand the partition systems. This code was ripped from Group. So I also fixed a bunch of bugs in Group. It's also able to carve for known file formats. You can show structures, understand and include files in order to show the contents of a specific memory dump with a formatted memory. It's also able to compute incremental per block checksums. This is handy because if you want to compute the checksums like MD5 every megabyte on a gigabyte file, you will get a file with all this, and you can also compare this with another dump and then identify which part of these files have modified. You can change the plugins for the Iowa. I mean, as I said before, all the input and output is pluggable, so you can create plugins for anything. And there are plugins like, for example, GZIP and EWF for loading encase images. You can get a list of all your plugins with this common. And this is more or less the list of plugins that can be used for opening local or remote targets. The uppercase minu-cell flag can be used with RASM2, which is handy for getting the list of assembly and disassembled plugins. And the same thing goes for RABIN2, that gets the list of plugins for binary formats that are supported by the project. Okay, so let's go for the first demo. Okay, so we have a file which seems like it's trash. I mean, there is nothing interesting here. We can see the entropy computation here. And there is, like, pretty huge. There are some zeros in there. But more or less entropy is pretty high. So what we'll do here is just look for known file format using the slash m, which looks for magic. And it looks like it found something in this offset. So we'll see in there. And we will dump. We will write the file. This created a new file, which the contents are starting at the current offset. So we can open it. And we see that this is a gzip. So we have to open it with gzip. And now we see something in there. We can use pm again, and you will see that this is a file format. So we'll use the mcommon in order to mount this partition. Using the mcommon, we'll specify that we want to mount this in the root using the file format and the offset zero. We can specify a different offset if we want to mount a different offset inside the file. Now you can use the ms with each shell inside the mount file format. And we can see the contents of this file format. We can show the contents. We can see, like, what's in slash bin. We can get this file. And then we can just open it. And see that this is a Linux ELF common LS program. So there is a bunch of functionalities that can be handy for N6. Another complaint is that analysis is slow. It's true that analysis is a regularly slow operation. It's blocking mainly because it takes some time and you need to grab all the information before doing anything useful. You can do this analysis in background, but it will still take a lot of time. Doing it in background means that you have to put a lot of mutex on the logic of the analysis, so it will slow down a little bit. It doesn't work really well for big binaries, mainly because it takes a lot of memory and CPU time. It's sometimes not able to find all the functions. There is a rule of adding more As at the end of A, so if you use the A common like this, you use AA, you will analyze all the symbols. If you add another A, you will do more analysis, and you add more As, you will get more things. So people at the end start adding more As in order to get more analysis, and this is probably not the best way for making analysis. So there are so many comments and configuration options for doing this, and there is this blog post that I wrote explaining some of them. So if you're interested, you can just read it or just ping me a little bit, I will try to help you. But the idea behind this is not waiting a lot of time in order to finish the analysis and start doing something, because if the analysis takes a lot of time, it means that after doing it, it will be really slow in operation, so you will have to think another more straight way for solving the problem. So what I usually do is that 90% of the problems to solve, at least from my daily problems, is that I can just analyze 10% or part of the program and grab the information that I really need, instead of analyzing the whole binary. Usually you just look for a string, you look for references to this string, you look for the functions that are using these references, and then you start analyzing just 5, 10 functions, and after this you don't really need much more information. So you can just use the comments for analyzing the target information that you want, you can analyze much faster because it's much less condensed. We are improving every release, so if you're updating frequently, you will get fixes and faster analysis on every build. Well, you have to understand the logic behind all these comments and options, and we will see some of them. So this is the first demo. I will open a Hello World in Go, if we make full analysis of all the symbols. Go binaries are pretty huge because they study the link of all the libraries and symbols, etc. So it took, well, we can compute this with this prefix. This is like time in the bash. So it took like 8 seconds. And after this we can seek to the string. So we see that the Hello World is in here, and we see that data reference in there, so we can just seek to the reference, and we see that the Hello World string is referenced in this layer instruction. But it took some time. It was 8 seconds. This is a simple example, but if you try to do this in a huge, bigger binary, it will take more time. So let's do the same using this script. This script basically looks for the Hello World string. It seeks to this address, and then, well, it surnames the flag. It defines that this is a string. It changes the block size for speeding up the search, and then look for references to this offset. If we run this script, it took only 2 seconds, almost 3. And it half-identified the same offset and the printf of the instruction. So as you can see, you can speed up a lot the analysis. This is a simple example, but there is much more difference if you try in a big binary. Another thing for analysis is that sometimes the references are not that clear. This is a Hello World for iOS in ARM64. And we can seek to the Hello World string. We see that there is no reference because we didn't analyze anything. So we will make, like, all the analysis options. And we see that there is a data reference in here. But if we try to analyze, like, in a simple way, we'll see that there is no reference. The reason behind this is because in ARM and also architectures, the references are computed in more than one instruction. So you need to emulation in order to understand the correct reference out of the... So if we go to the main, we will see that the printf is getting the address of this symbol in here. So it's getting the base address and then zooming an offset. And we can enable the emulation. And using the emulation, we will see that every instruction is getting emulated, and then we get the values of the registers on each instruction. So after this, we will see that it's creating... in ADRP, it's getting the reference to the string section, and it's incrementing the offset to get the Hello World string. So there is a common AAE, which is analyzing using emulation. And for... depending on the architectures, for example, in MIPS, it's really handy. There are other ways for getting references using AAV, which looks for pointers to data or strings or long pointers inside the same other space. You can see that there are so many comments under the AA and AAE subcomments. Okay. Another complaint is that rather it's not documented. And well, that's not true. It's documented in C. And there is inline help in every comment. You can get inline help without having to get to any browser or any book in front of you. There is also a book that I wrote for rather one, and Magin updated it for rather two. And there are already a lot of talks, slides, blog posts, and YouTube tutorials for understanding how to solve some crack myths and things like that. So it's not really true. It's so that it's complex, and sometimes it's hard to find the help for something, but there is an IRC and telegram channels that you can ask in there, and you will get help pretty quickly. Let's talk about the compilation. The compilation is not able to compile. It's able to do things that are similar to the compilation, but that's not really the strength of the tool, because it's hard. The compilation, it's not something easy. So we try to delegate these to third-party tools. Niterman wrote the plugin for Reddeck, which is an online service for the compilation. It's basically a plugin written in Node.js that uploads the binary that you are analyzing to the service, and you get an API for getting the disassembly of the compilation of different functions that you are analyzing with Radar. So you get comments and the C code, C like code, into the Radar shell. There is also the project Radeko, which is a Rust implementation of the compiler, which is pretty academic, and it's not yet available. I mean, it's working progress, and there are some starting of the project, but it's not really solving any real-world problems yet. I hope that maybe in one year or two it will be an option. There is also Boomerang, which was supported in Radar1. I think that Boomerang right now is not really maintained, so there is not much interest in supporting it in Radar2, but it will be pretty easy to port the scripts from Radar1 to Radar2. Last week I ported the Snowman, the compiler. This is much more updated, and it supports ARM and X86 for 32 and 64 bits. We can make a quick demo for this. We'll try to come back to that after it's compiled. So let's talk about disassembly. Disassembly is probably one of the good points of Radar because there are so many options for disassembly. You can colorize the instructions depending on the type of instruction, which is pretty handy for reading code, because you can easily identify where are the gems, where are the mathematical points, where are the parts of the code that are doing crypto, or doing traps or some syscalls, etc. There is also support for analyzing variables and arguments, so you can identify where these variables are set in the disassembly. There is also support for the set of disassembly. This means that it will parse things like add.exe, and convert it to something more human-friendly, like the expression on the right. This is handy sometimes when you want to get some more C-like code, but it's not really a way to compile. Okay, so we get the R2Snow in there. Okay, too bad. Let's talk about the disassembly. It was working yesterday. As you can see, the disassembly, it's highlighting the instructions in different colors. You can change the color scheme. You can also enable the cellulizasm, so you get things like this. The thing is that there is pds, which shows us summary of the disassembly of the function if you analyze the function and you get the pds. You will see all the references of C-strings and calls of this function, so you can rate a summary of what the function is doing. You can also use pdc, which is a pseudo-decompilation, which is using the asm pseudo and doing some logic of basic blocks in order to nest the different blocks. You can put the comments on the right, put them at the bottom, etc. All these things are interactive in visual mode, so if you press uppercase B, you get this and you can scroll around, see where the destination of the jump is going on, etc. Finally, there is a nasty art graph for all the functions, so you can get analysis. You can also... If you are in this basic block and you want to fold it, you can also fold them. You can switch to this assembly and you are moving back to the same point that you are in the graph, or going back to here. There is no user interface, but the command line is pretty handy. Another complaint is about stability. People say that rather to it's not stable. The main reason for this is because they are using an old version of it because they are not using the JIT or the last releases. We can talk about the stability depending on the amount of crashes that the user is experiencing and the amount of changes in the comments and APIs. Right now, we are after 1.0, so I tried to put some stability on top of this. The comments are pretty stable. Most of them are already used for many people and they are quite clear, so this is not going to change. The comments are quite stable. We try to refactor all the time. It's a project that... I don't really care about breaking something that was wrongly designed in the past, so I don't want to keep compatibility for years for something that was wrong at some point, so the API can change, but it's quite stable, so you can still do things like that. If you want to pass the output of a comment, you can just append the J letter at the end of the comment and you will get the JSON output, so you can do things like this. The I letter, you will get the information about the header binary, you can get the symbols, you can get the sections, you can also get sections like this. And the thing is that if you append a J, at the end you get this JSON, which can be indented using this and you get the indentation of the JSON. This is pretty easy to pass because all the programming languages, the modern programming languages have libraries or they support parsing JSON to native objects of the language directly, so it's pretty handy for scripting. And the thing is that this is much faster than using FFI or trying to deserialize all the binary structures from C into Python or any other language because it's allocating a lot of objects in a non-properly way because parsing JSON parses are much faster than any other parser right now. So all complains that some specific functionality is broken. Many complains or most of the complains that I receive every day in IRC are already fixed in JIT, so please update your version of RL2. There are security, if anybody reports a security bug or any crashes phone, they are fixed in less than one day. Usually in less than one or two, depends on how far the laptop is from me. I try to follow the rule of you see it, you fix it, so if you see a problem, I try to teach you how to fix it before just saying it's broken and I try to fix it later. This is because the committee grieves too much and I try to teach the community to be self-aware of the problems and how to solve them and how to report properly and how to just report a reproducer, say at least pass the backtrace something that can be useful for me to understand what the problem is. Mainly, you should also post the version of RL2 that you are using to see that if it's not the last version, I will not fix it because I'm only fixing bugs that are right in JIT. We try to make a release every six weeks. This is something that I decided after seeing how Rust project works. I think that six weeks is pretty nice time schedule. Mainly because having one month is really predictive. You expect the first day of every month to update something and every six weeks it's something more random, so at the end I'm rounding up down and you never really know when the next release will be, so that's kind of surprise and funny. The thing is that if you release... Some years ago I was releasing twice a year and this was not really good for the project. Mainly because if you are releasing to few... There was like 2,000 commits to review. There is a lot of things to test. There was no test suite at the time. Right now this is solved by making shorter time releases, so we have a test suite and we test everything more frequently. So it's hard to make a JIT clone and get a version that RL2 is not working. But there is also the problem of Debian. Debian is a distribution that tries to be stable and by stability they mean that they are not updating really frequently. The thing is that the current version of Debian is stable is five years old. If you can imagine that every six weeks there is like 400 commits, you can see how many commits are in five years. So I would recommend you to not use the packages from Debian. There is packages that are shipping in my repos and you can also use the packages from SIT which are a little bit more updated. But they are still old. So as I said before I tried to use the test suite. I used the RDD pattern which was invented by me and it's mainly a regression development pattern. And the thing is that after you find a regression you write a test and then you test that this regression will not happen again. This is because it's too late for doing TDD. This test development this means that you write the test before writing the code. So as long as the code is already there you have to write the test after this. And as long as we are doing continuous factoring of everything in order to improve the stability and portability and the reliability of the code some bugs can reappear. So it's important to test everything. Right now the test suite takes 15 minutes to run on Travis and half an hour for OSX and it's too slow for Apejor. So the Windows test suite is not running yet. We try to impose the fuzzing inside the development process so before any release or during the development of every release there is some people doing fuzzing on different file formats, the command line shell different inputs into the program and we use different tools like green sun, scan code which are the parts which are more buggy and which are the places to solve the bugs. This is important mainly because there are so many contributions many of them are from people that are starting to code. We try to follow some standards. So there is some coding style and this requires some education to the community. Our complaint is that this is not written in Python. I know that C is not the perfect language. It is easy to make mistakes but Python is not the solution. Maybe at some point there will be a language that can replace C maybe a Rust or maybe Swift or I don't know but maybe a Rust is the one that fits better inside the philosophical project. But if you want to use Python there are three different bindings for the native APIs. There are two different bindings for L2Pype supporting different transports. I will explain later what is L2Pype. You can write plugins for EO, Erbin and Asmin in Python already. The reason for not choosing a dynamic language for this is because language allows you to catch errors at compile time so it is easier to identify problems before running the program. There are so many tools available in order to profile the bug and optimize or port to different platforms. It is faster. It is native. It is a smaller footprint. There is not really a run-time library that ships inside the final binary so you can put a static link and put it in a router or any other device. And C also runs spiles to JavaScript so you can run rather in the web browser. And that is fine for me at least for the 90% of the problems. And for the rest you can just use any scripting language. OK. Let's talk about the graphical user interfaces. The main complain is that there is no graphical interface but that's not true. Terminals are scary. People are scared about the terminals. They used to like to use the mouse and try to click and do things like that. So the real problem for RADR2 is not that there is no G. The problem is that there are so many of them. There is a visual mode. Which is DWM-like. There is a different web user interfaces. One of them is this which is material-like. This is the default for Android. There is the other one which is a domain jQuery mobile and it's more desktop friendly. I also brought the Blesser which is Node.js and courses like interface with windows and things like that. I also brought RADR which was written by RADR1 and this is basically a VTE terminal with a GTK interface that there are some buttons and menus that you can just instead of typing comments you can just run the comments on the menu. There was a complete interface written in Vala and it was working pretty fast and pretty nice but it was up and down because it was boring to write graphical interfaces and I think nobody cares about that. So I never released it but it's there. There is also Boken which is a Python GTK interface that it's right now not really maintained. I think it's working in the last version of Vala too but it's not adding new functionalities and it's just for static analysis. Now there is Guy Mandingo who is writing new user interface in .NET but he's focusing only windows. The thing is that the differences of this project are depending on the explorer widget and also the MFSE API so it doesn't really work on Linux or OS X so it's not portable. Maybe someday it will be portable. Three days ago a guy made a mention to me in Twitter so I had to add this slide and he's writing a Qt5 interface. You can see a screenshot here right from the GitHub but let's talk about real user interface. The author of Boken has been working for a year and a half something like that in a new user interface written in Qt and C++ which looks like this. This is the main user interface it supports actually only something that it will support debugging and emulation all those things like that. I will release early next year. Let's talk about scripting. It's something a topic that is complicated and really if you focus on the problem you only need to understand the comments and you only need to understand all the APIs so if you know how to use rather you can script rather pretty easily mainly because I was trying to follow different paradigms like trying to make the bindings more language friendly so at the end you really need to have bindings specific for its language. The reason is that this is too much work and APIs are changing so at the end I wrote a tool which is Balabind which transpiles BAPI interface into different interfaces for different languages so I create a single file that interfaces the C code and it creates Python and Node.js bindings for all this but this was not really stable at all because there are some memory problems because sometimes it's hard to manage the references and things like that so at the end I got the idea of implementing R2Pype which is basically a pipe on top of these four comments. The API provides an open method which allows you to open a different file using a specific transport to open a target file and then you have the CMD common which basically runs a common in R2 and then you get back the result and there is the same with the JSON output which basically returns an active object of the result and then there is quit so it's just only one method that you have to remember for using this API. Let's make a quick demo for this. This is a script for getting the configuration file of the Mirai malware this is a botnet and this is a script it's written in Python and it's using R2Pype it's basically extracting the configuration file out of the binary you can use it like from Python like doing it like this there's a config file in there you can also use it from inside rather so you can just connect it like this or inside the shell so if you are inside the shell you can interpret the file just using the dot so it's importing the file and running it there are so many examples in many different programming languages in fact this is the list of all the languages that support R2Pype not all of them support all the transports I mean that you can use R2Pype through HCTP you can also use it through TCP, you can also use it with pipes just spawning the binary and writing the standard input standard output these works in old platforms in Windows Linux 06 there are examples inside the directory so if you go into the R2Pype repository you will see that for example for Node.js there are examples in there you can see for example there is an IRC bot there is also a syscall emulator which is basically implementing the syscall handler in JavaScript and you can run this this hello wall which is basically this code and this will have to run install the npm but I can change this it's just a stupid thing anyway the Python example there is the same in Python so it's running the syscall handler in there and it's basically saying that you have to use a seal for emulating code and then you get the R2Pype script executed after this ok so the bugger is confusing, many people think that the bugger of R2Pype is strange mainly because it's a low level bugger the main reason for this is because I don't try to replace source the buggers there is gdv, ldv and the B-source Studio which are pretty good source the buggers but R2Pype aims to work for binaries that you don't have the source code or different assets so when you're starting the bug with R2Pype you are starting the bug into the dynamic loader of the system not inside the entry point the reason for this is because some file formats can exploit parameters inside the dld and execute code before the entry point or there can be more than one entry point also there can be changes in memory that will not be applied into the disk so some people complains that they are patching the binary in memory and then execute the program and the problem is not patched the reason for this is because you are patching the memory and not the binary itself if you want to create a specific environment you have to use R2Pype which is a tool that creates a profile to specify for example the user ID the ch-root the different directory that you want to run common all the arguments if you want to change like a file descriptor like standard input you want to be a socket it's listening in a specific port etc this allows you to create a profile for running the program every time this is handy if you are trying to make a crack me or things like that so the debugger basics are like this you can spawn, attach you can create plugins for all these things and there are plugins as long as I said before there are plugins for all these so you can use a native debugger there are more than one native debugger there are people writing their own native debugger for windows and things like that you can use attach to a gdv you can also attach to windvg, box etc all the subcomments of the debugger are under the D letter and you can also do local remote debugging and you can inject code there is a dx command that allows you to inject a bunch of bytes and then it will get back the result of the state the gdv backend is not working on the platforms mainly because the gdv protocol is crap it mixes binary, plain text and xml in single and with assets actions which is a really bad decision and they try to reimplement this thing for every single platform so every time that you try to connect to a new platform you have to use different solutions for getting or writing the registers or reading, writing memory or doing steps and setting breakpoints and things like that so it's kind of hell so it's working progress and right now it supports Intel mainly so you can use it for debugging windows or linux kernels attaching to QM or things like that and it's working progress for MIPS, ARM and AVR there is R2LLdv which is a project that I wrote now secure and it's mainly an R2 pipe script that allows you to you are inside RLdv and you run this Python script and it will allow you to touch from rather in there and you can use all the functionalities, read, write memory from rather to and just have the shell in there this is nice because if you are working with Apple things like Apple Watch or iPhone devices you can use R2 to the back binaries in there without having to do anything like that there is also R2Frida Frida is a dynamic tracer that injects library which contains a JavaScript interpreter in there and you can put the scripts in there or just use the core libraries for running coding there this project is written by Ulle Andre also working in my company and it allows you to it works in many platforms, can be used in iOS, Linux, Android, QNICs Windows and it's pretty fast, maybe thinking about JavaScript and all these things make you think that it's slow but it's really fast and it allows you to make introspection inside the process so you can do debugging inside the process let's make a really fast demo so I have Node.js running in this terminal and I have rather R2 in the other one and I will use Frida to do this process. Right now we can get the information using the backslash and we can see the list of comments from the Frida plugin. From here we can get information from the binary we can get information from the export for example the symbol that you can read write memory from the target process you can modify it or do whatever with it, you can also analyze it and get the graphs up to there yeah this should work okay it's not so what I'm going to do now is use the DTF comment which allows you to trace a specific symbol and use a format string for tracing this function so know every time that the Node.js process is writing using the right symbol it will print something in there we can also show the backtrace inject JavaScript in there etc okay let's talk about the seal the visual machine that comes inside the other two it's an intermediate language that basically it's translating every instruction into a string we can see this in there so if you press uppercase o you will see a string and this string represents what the instruction is doing behind the scenes so it's a forth like language it has two different stacks it's text based the reason for this is that you can easily read and modify what the instruction is doing you can also change the instruction in itself without having to recompile or other too you can change the expression and this is used for my thing not just for emulation it's used for analysis it's also used for assisted debugging so if you want to specify a specific string like defining I want to continue this execution until the specific register have a value that is in this range or you want to identify if the specific jump is going to be executed or not or find references such expressions that match some memory like for example I want to find a specific version of memory that contains some d-word and then some 0 etc you can use the seal for this these are some of the comments that are used for emulation it's basically the same for the back but using Ae as prefix instead of d and there is also support for unicorn but it's not as complete as the seal 1 so here is a crack me it can be solved using this script it can be solved using this R2Script which is basically seeking to create a symbol that you want to emulate ruining the code in there stopping at the point that is comparing the two strings and then dumping the registers values so if we do like this you get the password and the same thing goes for the Python so basically doing is calling the check password function this function it goes for looking for the string compare and then getting the value of the registers if you do this you can see for example in the debugger you can continue the execution in the entry point and then you can use drr to get the not in the debugger so I'm continuing to the entry point and then I'm using drr to use telescoping in order to get information of every register and where they're pointing to so you can also do the same in the stack to get all the pointers of the strings and many things like these are useful and used from there so about exploiting which is the final slide there are so many functionalities in R2 that can be handy for exploiting it's you can patch code in memory you can do things like that so I'm right now into the debugger so I can just put the cursor in anywhere and I can press uppercase A and I can just patch the code for example with this this is handy for preparing some specific environments there is simulation of the debugger and there are some handy functionalities like generating and finding offsets of diverging patterns you can do resistive telescoping like I showed before you can do heap analysis this is only working on linux but it will be ported to our sixon windows soon and let's make a quick demo about the rarop thing which is a frontend of R2 written in Node.js that basically allows you to create the rub gadgets so you have all the rub gadgets on the right side that are found in the slashv and slashls binary you can add them in there you can also remove them you can move them around you can also drag and drop and when you have all this thing ready you can just debug and you get the terminal ready for debugging the rub gadget that you have written so you start inside the red so you start stepping and we are doing all this rub gadgets and to finish I would like to show you one of the reasons that you can use R2 for extending or using a specific exploit I'm using dirty code exploit for writing a Neo plugin so this is a vulnerable linux it's one month from one month ago I don't know if you know the dirty code vulnerability but it's pretty funny and you can read more in these URLs what this one allows you to do is to modify files that are not owned by you so you can modify for example the ATC services I don't have permission for this but if I'm using the dirty code plugin I can see this but if I write some string you see that it's not changing the reason for this is because this exploit the service condition so I have to write several times in there I'd rather write this comment 10 times and it will pass the binary so you can just cut and get the file that has been changed so you can use a Neo plugins or any other plugins for exploiting any of the bugs that are in the systems so I'm afraid we run out of time for questions but as Seji said he's going to try to arrange a meeting for you guys, someone here during the Congress still where can they find you? Yeah you know any place that I can meet with our people there I was just linking up with him on Twitter let's thank him again for this great series of demos and give him a round of applause again


There are many constants in the storage business, relentlessly declining costs per bit, innovations that perpetually battle the laws of physics, a seemingly endless flow of venture capital, despite the intense competition. And there's one other constant in the storage business, Eric Herzog, and he joins us today in this CUBE video exclusive to talk about IBM's recent storage announcements. Eric, welcome back to theCUBE. Great Dave, thanks very much. We love being on theCUBE and you guys do a great job of informing the industry about what's going on in storage and IT in general. Well, thank you for that. We're going to cover a lot of ground today. IBM Storage, you made a number of announcements the past month around data resilience, a new as a service model, which a lot of folks are doing in the industry. You've made performance enhancements. Can you give us the top line summary of the hard news, Eric? Sure, the top line summary is of course, cybersecurity is on top of mind for everybody in the recent Fortune 500 list that came out. You probably saw there was a survey of CEOs of Fortune 500 companies, they named cybersecurity as their number one concern, not war, not pandemic, but cybersecurity. Okay, so we've got an announcement around data resilience and cyber resiliency built on our flash system family with our new offering, safeguarded copy. And the second thing is the move to a new method of storage consumption. Storage is a service, a pay as you go model, cloud like the way people buy cloud storage, that's what you can do now from IBM Storage with our storage as a service. Those are the key to takeaways, Dave. Yeah, and I want to stay on the trends that we're seeing in cyber for a moment. The work from home pivot in the hybrid work approach has really created new exposures. People aren't as secure outside of the walled garden of the offices. And we've seen a dramatic escalation in the adversary's capabilities and techniques, not the least of which is island hopping, in other words, putting code fragments in the digital supply chain. They reform once they're inside the company and it's almost like this organic creepy thing that occurs. They're also living, as you know, stealthily for many, many months, sometimes years, exfiltrating data, and then just waiting. And then when companies respond, the incidents response trigger a ransomware incident. So they escalate the cyber crime and it's just a really, really bad situation for victims. What are you seeing in that regard and the trends? Well, one of the key things we see is everyone is very concerned about cybersecurity. The Biden administration has issued victims, not only to the government sector, but to the private sector, cybersecurity is a big issue. Other governments across the world have done the same thing. So at IBM Storage, what we see is taking a comprehensive view. Many people think that cybersecurity is the moat with the alligators, the castle wall, and then of course the sheriff of Nottingham to catch the bad guys. And we know the sheriff of Nottingham doesn't do a good job of catching Robin Hood. So it takes a while, as you just pointed out, sitting there for months or even longer. So one of the key things you need to do in an overall cybersecurity strategy is don't forget storage. Now our announcement around safeguarded copy is very much about rapid recovery after an attack from malware or ransomware. We have a much broader set of cybersecurity technology inside of IBM Storage. For example, with our flash system family, we can encrypt data at rest with no performance penalty. So if someone steals that data, guess what? It's encrypted. We can do anomalous pattern detection with our backup product spectrum protect plus. Why would you care? Well, if the cubes backup was taking two hours on particular data sets and all of a sudden it was taken four hours, maybe someone is encrypting those backup data sets. And so we notify. So what we believe at IBM is that an overarching cybersecurity strategy has to keep the bad guys out, okay? Correct detection, anomalous pattern behavior on the network, on the servers, on the storage and all of that, chasing the bad guy down once they've breached the wall, because that does happen. But if you don't have cyber and data resilience built into your storage technology, you are leaving a gap that the bad guys can exploit. Whether that be the malware ransomware guys, or by the way, Dave, there still is internal IT theft. Back there was a case about 10 years ago now where 10 IT guys stole $175 million.  And that was an internal IT theft. So between the internal IT issues that could approach you, malware and ransomware, comprehensive cybersecurity strategy must include storage. So I want to ask you about, come back to Safeguard copy. And you mentioned some features, some capabilities in encrypting data at rest, your anomalous pattern recognition. Inferring, you're taking a holistic approach, but of course you've got a storage centricity. What's different about your cyber solution? What's your unique value prop relative to your company? Well, when you look at Safeguarded copy, what it does is it creates immutable copies that are logically air gapped, but logically air gapped locally. So what that means is if you have a malware, ransomware attack and you need to do recovery, whether it be a surgical recovery or a full on recovery because they attacked everything, then we can do recovery in a couple hours versus a couple of days or a couple of weeks. Now, in addition to the logical local air gapping with Safeguarded copy, you also could do remote logical air gapping by snapping out to the cloud, which we also have on our flash system products. And you also of course could take our flash system products and back up to take, giving you a physical air gap. In short, we give our customers three different ways to help with malware and ransomware. Air gapping locally, right? Yeah, please continue, I'm sorry. So our air gapping locally for rapid recovery, air gapping remotely, which again then puts it on the cloud provider network. So hopefully they can't breach that. And then clearly a physical air gap going out to tape. All three and on the main frame, we have Safeguarded copy already Dave and several of our main frame customers actually do two of those things. They'll do Safeguarded copy for rapid recovery locally, but they'll also take that Safeguarded copy and either put it out to tape or put it out to a cloud provider with a remote logical air gap using a snapshot. I want to ask you a question about management because when you ask CISOs, what's your number one challenge? They'll say lack of talent. We've got all these tools, we lack of skills to really do all this stuff, can't hire people fast enough and they don't have the skills. So when you think about, and so what you do is you bring a lot of automation into the orchestration and management. My question is this, when you set up air gaps, do you recommend or what do you see in terms of logically and physically, not only physically separating the data, but also the management and orchestration and automation? Does that have to be logically air gapped as well or can you use the same management system? What's the best practice there? Ah, so what we do is we work with our copy management software, which will manage regular copies as well, but Safeguarded copies are immutable. You can't write to them, okay? You can't get rid of them and they're logically air gap from the local hosts. So the hosts, for the Safeguarded copies, that immutable copy you just made, the hosts don't even know that it's there. So you manage that with our copy management software, which by the way, will manage regular snapshots and replicas as well. But what that allows you to do is allows you to automate. For example, you could automate recovery across multiple flash system arrays. The copy services manager will allow you to set different parameters for different Safeguarded copies. So a certain Safeguarded copy, you could say, make me copy every four hours. And then on another volume, on a different data set, you could say, make me a copy every 12 hours. Once you set all that stuff up Dave, it's completely automated, completely automated. So I want to come back to something you mentioned about anomalous pattern recognition and how you help with threat detection. So a couple of multi-part question here. First of all, the backup corpus is an obvious target. So that's an area that you have to protect. And so, and you're saying, okay, you use the example if your backup's taking too long. But so how do you do that? What's the technology behind that? And then can you go beyond, should you go beyond just the backup corpus, with primary data or copies on prime, et cetera? Two-part question there. So when we look at it, the anomalous pattern detection as part of our backup software, say Spectrum Protect, and what it does, it uses AI-based technology. It recognizes the pattern. So it knows that the backup data set for the cube takes two hours and it recognizes that. And it sees that as the normal state of events. So if all of a sudden that backup that the cube was doing used to take two hours and starts taking four, what it does is that's an anomalous pattern. It's not a normal pattern. It'll send a note to the backup admin, the storage admin, whoever you designate it to and say, the backup data set for the cube that used to take two hours has taken four hours. You probably ought to check that, okay? So when we view cyber resiliency from a storage perspective, it's broad. We just talked about anomalous pattern detection in Spectrum Protect. We were talking most of the conversation about our safeguarded copy, which is available on the main frame for several years and is now available on Flash Systems, making immutable, local, air-gapped copies that can be rapidly recovered and are immutable and can help you recover for a malware, ransomware attack. Our data at rest encryption happens to be with no performance penalty. So when you look at it, you need to create an overarching strategy for cybersecurity. And then when you look at your storage estate, you need to look at your secondary storage, backup, replicas, snaps, archive and have a strategy there to protect that. And then you need a strategy to protect your primary storage, which would be things like safeguarded copy and encryption. So then you put it all together. And in fact, Dave, one of the things we offer is a free cyber resilience assessment. It's not only for IBM storage, but it happens to be a cyber resilience assessment that conforms to the NIST framework and it's heterogeneous.  It's all about the data sets, not about the storage. So we said, you said these 10 data sets are critical. Why are you not encrypting them? Okay, these data sets are XYZ, why are you not air gapping them? So we come up based on the NIST framework, a set of recommendations that are not IBM specific, but they are storage specific. Here's how you make your storage more resilient, both your secondary storage and your primary storage. That's how we see the big thing in safeguarded copy, of course, fits in on the primary storage side, A, on the main frame, which we've had for several years now and B, in the Linux world, the Unix world and the Windows server world on our flash system portfolio with the announcement we did on July 20th. Great, thank you for painting that picture. Eric, are you seeing any use case patterns emerge in this space? Well, we see a couple of things. First of all is A, most resellers and most end users don't see storage as an overarching part of the cybersecurity strategy and that's starting to change. Second thing we're seeing is more and more storage companies are trying to get into this bailiwick of offering cyber and data resilience. The value IBM brings, of course, is much longer experience to that and we even integrate with other products. So for example, IBM offers a product called Q-Radar from the security division, it's not a storage product, it's a security product and it helps you with early data breach recognition. So it looks at servers, network access, right? It looks at the storage and it actually integrates now with our safeguarded copy. So part of the value that we bring is this overarching strategy of A, comprehensive data and cyber resilience across our whole portfolio, including safeguarded copy on July 20th announcement, but also integration beyond storage now with our Q-Radar product from IBM's security division and there will be future announcements coming in both Q4 and Q1 of additional integration with other security technologies. So you can see how storage can be a vital cog in the corporate cybersecurity strategy. Got it, thank you. Let's pivot to the as-of-service, it's a cloud obviously is brought in as-of-service. It seems like everybody has one now. You guys have announced obviously HPE, Dell, Lenovo, Cisco, Pure, everybody's got now their as-of-service model. What do we need to know about your as-of-service solution and why is it different from the others? Sure, well one of the big differences is we actually go on actual storage, not effective. So when you look at effective storage, which most of them do, that includes creating the RAID data sets and other things, so you're basically paying for that. Second thing we do is we have a bigger margin. So for example, if theCUBE says we want SLA three, and we sell it by the SLA, Dave, SLA one, two and three. So let's say theCUBE needs SLA three, and the minimum capacity is 100 terabytes. But let's say you think you need 300 terabytes, no problem. You also have a variable. One of the key differences is unlike many of our competitors, the rate for the base and the rate for the variable are identical. Several of our competitors, when you're in the base, you pay a certain amount. When you go into the variable, they charge you a premium. The other key differentiator is around data reduction. Some of our competitors and all storage companies have data reduction technology, right? Block level D-dup, thin provisioning, compression, right? We all offer those features. The difference is with IBM's pay as you grow, storage is a service model. If you have certain data sets that are not very dedupable, not very compressible, we absorb that. With our competitors, most of them, if the data set is not easily dedupable, compressible, and they don't see the value, they actually charge you a premium for that. So that is a huge difference. And then last big difference is our 100% availability guarantee. We have that on our flash system product line. We're the only one offering 100% availability guarantee. We also, against many of the competitors, offer a better base. Nines, as you know, availability characteristics. We offer six nines of availability, which is five minutes and 26 seconds of downtime, and the 100% availability offering. Some of our competitors only offer four nines of availability. And if you want five or six, they charge you extra. We give you six nines base in, which is only five minutes and change of downtime in a year. So those are the key differences between us and the other as a service models out there.  I mean, that's the basic philosophy of these things, right?  I mean, now in, maybe I sign up for a multi-year term, I commit this much, whatever, 100 terabytes, whatever the minimum is. And then I can say, hey, you know what? This is working for me. We CFO likes it, and the IT guys can provision more seamlessly. We got our charge back or show back, I want to now make a bigger commitment. And I can, and I want to sort of, can I break my three-year term and come back and then renegotiate, you know, kind of like reserved instances, maybe bigger and pay less? How do you approach that? Well, what you do is we do a couple of things. First of all, you could always add additional capacity. Okay, you just call up. We assign a technical account manager to every account. So in addition to what you get from the regular sales team and what you get from our valued business partners, by the way, we did factor in the business partners day into this. So business partners will have a great pay-as-you-go storage as a service solution that includes partners and their ability to leverage. In fact, several of our partners that do have both MSP and MHP businesses are working right now to leverage our storage as a service and then add on their own value with their own MSP and MHP capabilities. And they can white label that? Is that right or? Well, you'd still have storage as a service from IBM. They would resell that to the cube and then they'd add in their own MHP or MSP. Got it. That said, partners interested in doing a white label, we would certainly entertain that capability. I interrupt you. Carry on, please. You can go ahead and add more capacity, not a problem. You also could change the SLA. So the cube, one of the leading industry analyst firms, you bought every analyst firm in the world and you're using IBM Storage as a service, the pay-as-you-go cloud-like model. So what you do is you call up the technical account manager and say, Eric, we bought all these other companies, they're using on-prem storage. We'd like to move to storage as a service for all the companies we acquired. We can do that, so that would up your capacity. And then you can say, now we've been at SLA too, but because we're adding all these new applications or workloads from our acquired companies, we want some of it to be at SLA one.  All right, I got the hard question is what if all those other companies, the cube acquired, are also IBM's storage as a service customers? Can I, what's that discussion like? Hey, can I consolidate those and get a better deal? Yeah, if they're all storage as a service customers and Dave, I love that thought, we would just figure out a way to consolidate the agreement. The agreements are one through five years. One other thing also that's very unique is let's say for whatever reason, and we all love finance people. Let's say the IT guys have called up financing, we did a one year contract, we now like to do a three year contract, the one year's coming up and guess what? Finance is delayed for whatever reason, the PO doesn't go through. So the ITI calls up the technical account managers, we love your service, it's delayed in finance. We will let them stay on their storage as a service, even though they don't have a contract. Now, of course they've told us they wanna do one, but if they exceed the contract by a quarter or two because they can't get the finance guys who are messing with the IT guys, that's fine. One of the key differentiators, exactly the same price. Some of our competitors will also extend without a contract, but until you do a contract, they charge you a premium. We do not. Whatever, if you're at SLA 3, you're at SLA 3 will extend you and no big deal, right?  So that is something we can do and we'll do on a continual basis. Last question, let's go way out. So, we're not doing any kind of near-term forecast. I'm trying to understand how popular you think as a service is gonna be. I mean, if you think about the end of the decade, let's think industry total, you know, one of the IBM specific. How popular do you think as a service models will be? Do you think it will be the majority of the transacted business or it's kind of more of a, just one of many? So I think there will be many, okay? Some people will still have bare metal on-premises. Some people will still do virtualization on-premises or in a hybrid cloud configuration. What I do think though is storage as a service will be over 50% by the end. Remember, we're sitting at 2021. So we're talking now 2029. So I think storage as a service will be over 50%. I think most that storage as a service will be in a hybrid cloud model. I think the days of 100% cloud, which is the way it started, I think a lot of people realize that 100% cloud actually is more expensive than a hybrid cloud or fully on-prem. I was at a major university in New York. They are in the healthcare space. And I know their CIO from one of my past lives was talking to him. They did a full-on analysis of all the cloud providers going 100% cloud. And their analysis showed that 100% cloud, particularly for highly transactional workloads was 50% more expensive than buying it, paying the maintenance and paying their employees. So he did an all-in view. So what I think it's going to be is, storage as a service will be over 50%. I think most that storage as a service will be in a hybrid cloud configuration with storage on-prem or in a colo, like what our IBM pay-as-you-go service will do. And then it'll be accessed and available through a hybrid cloud configuration with IBM Cloud, Google, Amazon, Azure, whoever the cloud provider is. So I do think that you're looking at over 50% of the storage being as a service, but I do think the bulk of that as a service will be as a service through someone like IBM or our competitors. And then part of it will be from the cloud providers. But I do think you're going to see a mix because right now the expense of going 100% cloud storage is dramatically understated. And when someone does analysis like that major university in New York did, they had a guy from finance help them do the analysis. And it was 50% more expensive than doing on-prem as a, either on-prem or on-prem as a service. Both were way cheaper. Okay, but you own the asset, right? And as a service model, okay. And I would bet, I would bet that over the lifetime value, the spend and it as a service model, just like the cloud, if you do this with IBM or any other competitors, I would bet that overall you're going to spend more just like you saying in the cloud, but the benefit is the flexibility that you get. Yeah, yeah. If you compare it to the, so obviously the number one model would be to buy. That's probably going to be the least expensive, right? But it's also the least flexible. Then you also have leasing, right? More flexibility, but leasing usually is more expensive.  Same thing with any IT asset, now storage network servers all are available on leasing. The net net is at the bottom line. That's more than buying it up front. And then storage as a service will also be more expensive than buying in my friend, but ultimate capability, altering SLAs, adding new capacity, being able to handle it up very quickly. We can provision the storage. As you mentioned, the IT guys can easily provision. We can version the storage in 10 minutes. If you bought from IBM storage or any competitor and you bought and you need more storage, A, you got to put a PO through your system. And if you're not the cube, but you're a giant global fortune 500, sometimes that takes weeks to get the PO done, then the PO has to go to the business partner. The business partner's got to give a PO to the distributor and a PO to IBM. So it can take you weeks to actually get the additional storage that you need. Okay. With storage as a service from IBM with our pay-as-you-go cloud-like model, all you have to do is provision and you're done. And by the way, we provide a 50% overage for free. So if they end up needing more storage, that 50% is actually sitting on-prem already. And if they get to 75% utilization of the total amount of storage, we then call them up, the technical account manager would call them up and their business partner and say, Dave, do you know that you guys are 75% full? We'd like to come add some additional storage to get you back down to a 50% margin. And by the way, most of our competitors only do a 25% margin. So again, another differentiator for IBM storage as a service. What about, I said the last question, but I have another question. What about day one? Like how long does it take if I want to start fresh as a service? How long does it take to get up and running? Basically you put the PO through, right? Whatever it takes on your side or through your business partner. We then will sign the technical account manager. We'll call you up because you need to tell us, do you want it in a colo facility that you're working with or do you want to put it on-prem? And then once we do that, we just schedule a time through your RT guys do the install. So, you know, probably two weeks. Yeah, okay.  Or on-prem, okay, we're going to have guys in the office a good day when it's not going to be too busy. Could you come two weeks from Thursday? Which now would be three weeks for sake of argument. But that would be, you know, we interface with a customer with a technical account manager to do it on your schedule, on your time, whether you do it in your own facility or use a colo provider. Yeah, but once I tell you, once we get through all that stuff, it's two weeks from when that's all agreed. It's like the Xerox copy or salesman. Where are you going to put it? Once you decide where you're going to put it, then it's a couple of weeks. It's not a month or two months. No, it's not. And when you do additional capacity, remember there's a 50% margin sitting there. So if you need to go into the variable and use it, and when we hit 75%, we actually track it with our storage insights pro. So we'll call you up and say, Dave, you're at 76%. We'd like to add more storage to give you better margin of extra storage. And he would say, great, when can we do it? So we're proactive about that to make sure that you stay at that 50% margin. Again, our competitors all do only a 25% margin. So we're giving you that better margin, a larger margin, in case you really have a high capacity demand for that quarter.  Great, Eric gotta go. Thank you so much for taking us through that. Great detail, I really appreciate it. Always good to see you. Great, thanks, Dave, really appreciate it. All right, thank you for watching this Cube Conversation. This is Dave Vellante and we'll see you next time.