All right, so this is dispelling the dark magic inside a ruby debugger. My name is Daniel, and I watched the eclipse this summer. Who here traveled to see the total eclipse? Anyone? Okay, fair amount of hands. I really felt like it was worth it. It was a great experience. I packed up my family. We drove a few hours down south from where we live in Seattle, down to Central Oregon, converged on a small town in Oregon called Madras, Oregon, along with about pretty much the entire West Coast, I think. It was an epic traffic jam getting out of there. It was amazing. But the eclipse was really awesome. This was a photo that I took right at the beginning of totality. Really surreal, amazing, but surreal experience. I remember looking up at this and it's weird feeling. It's like your body is telling you that it's not supposed to look like that. Like there's something wrong. You're seeing a special effect or something like that. It's just this eerie experience. And I remember thinking that it's, I'm starting to understand maybe the reaction of some of our pre-scientific pre-industrial ancestors that they might have had seeing something like this and concluding that, oh man, the world's about to end, or there's some sort of dark magic involved here. Because looking at this, I realized that the only thing that was really keeping me from having that same reaction was the knowledge of what was going on, right? The understanding of a little bit of the inner workings of what it was I was seeing. And I realized that as a developer, I often have a similar response to some of my tools. In particular, debuggers. I've been developing professionally since the late 90s or so. And throughout that career, I've been very hesitant to use debuggers. Pretty much that entire time. And it's because they felt kind of eerie, kind of spooky. These tools are doing something bizarre to my program. They're casting weird spells on the virtual machine. They're doing something that's stopping my program from running and causing the machine to do something else. It just felt eerie. It was almost intimidating. In general, it just didn't feel comfortable. I wasn't comfortable with these tools. And so I spent my entire, pretty much my entire career as a debugging troubleshooting using PUDES, or printf, or whatever the call happened to be in the languages I was using. I was a PUDES debuggerer. And I know I'm not alone in that. This was an article that some of you may recognize. One of our Ruby luminaries, member of the core team, wrote this early last year. PUDES debuggerer, we call ourselves. Well, it turns out that debuggers really aren't as spooky as they may seem at first. And to prove that, we're gonna spend this hour looking under the hood at a debugger. We'll see kind of how it works, what it's doing. We'll see that it's not just dark magic. It's just Ruby under the hood. And it's actually not that complicated either. In fact, we're gonna go ahead and just implement a debugger right here. We'll feature a debugger. It's actually not that hard. Just take a few minutes. And then afterwards, if we have time, we'll spend some time looking at a real-life production debugger. Some of the techniques that we use to implement it. We'll look at some of the debugging facilities of the Ruby VM. It'll be fun. It'll be fun. So, let's see. Before I get started, a little bit about myself. As I said, my name is Daniel. I've been developing since the late 90s or so. Started with Ruby around 2005. Right as Rails was starting to kind of enter the picture. Joins kind of a series of Rails startups. Then, about four and a half years ago, I decided to switch gears a bit. So, I joined a kind of a small company. I joined this company. I'm part of the cloud platform team. And I work on stuff to help Ruby developers use the cloud platform. So, it's actually been a really good gig. I really enjoyed it. But, because I work at a small company, there's a little bit of small company bureaucracy that I have to get through. So, all the code that you see here, sample code, live coding, is copyright the small company and Apache 2 licensed. Okay, so that out of the way, let's write a debugger. What we'll do is we'll just create something very straightforward. We'll create a library that will let you set breakpoints. It will let you open the command shell, debugger shell, let you inspect the program state, and also step through execution of your program. So, kind of the basic functionality that you'll expect in the debugger. So, let me go ahead and switch over to my editor here. And we'll start by taking a really quick look at a kind of a sample toy program that we'll use to demo our debugger features. So, this is just a quick program that prints out a little hello message. So, the sender and the recipients of the message will be passed on the command line. Sender will be set in an instance variable. Recipient gets set or passed to a method. We have a method that prints out a hello message. Another method that prints it out twice, because why not? And that's it. So, here's how that actually works. Read.read.rb. So, hello world from Ubikoff, okay. That was fun. Let's go write a debugger. All right, so, we're gonna start by managing breakpoints. So, we'll be able to add breakpoints to a program and we'll store breakpoints in the struct. And each breakpoint will have a name and then it'll be a file in line that the breakpoint is in. All of the methods in this class will just make class methods just for simplicity's sake. So, we'll have a method to add a breakpoint. Name, file line. And we'll just store all the breakpoints in an array. We'll need an initialization methods to initialize that array. So, breakpoints. And we'll make sure that that initialization gets called when we require our file. Okay, so, we have a simple library that lets you add breakpoints to an array. We'll go back to our toy program and we'll require that file. And we'll set a breakpoint. Let's see, so, we'll give it a name, my breakpoint. We'll set the breakpoint in this file. And we'll set it over here on line 11. So, all right. So, we've added a breakpoint to our program. Let's see what happens. And nothing happens because we added a breakpoint to an array but we didn't actually do anything with it. We didn't actually have any code that breaks at that breakpoint. So, how do you do that? How do you interrupt a running Ruby program to implement a breakpoint? Well, one thing that you can do is you can use a powerful class in the Ruby Core library called TracePoint. TracePoint, it's part of Ruby Core. It's been there since Ruby 2.0, I believe. It's a Ruby class that lets you listen for events that happen at the Ruby virtual machine level. It lets you register callbacks that will get called whenever those events take place. So, for example, this method listens for the method call event. So, this example listens for the method call event. Whenever a method is called, it executes this block which prints out a little message. TracePoint also passes an object to the block that includes some information about the events that just happens. For example, the name of the method. A number of events are supported. Methods called, return from methods. Exceptions being raised, threads starting and ending. There's even an event for moving to the next line in your Ruby application. And it's this event that we're gonna use for now to implement breakpoints. So, each time we move to a new line in the program, we'll register this callback that searches through our breakpoints and looks for, do we have a breakpoint here at this line? So, let's go back to our code. And implement that. So, we'll create a tracepoint. Make sure that we call this method from my initialization. I often forget that when I practice this. So, tracepoint. Okay, we're gonna trace the line event. And each time we move to a new line, we'll search our breakpoints. We'll look for a breakpoint with the same file as where we were at. And also the same line number. If we find a breakpoint, then for now, let's just print out a little message. It says, hey, we found a breakpoint. Breakpoint, we can provide the, print out the name of the breakpoint.  For example, the name of the class that we're in. There is also the method, as we saw. Let's also, just for kicks, print out the file name we're at and the line that we're in. All right, so we set up a line tracepoint that when we hit a breakpoint should print this out. So, let's see if that works. And there we go. Remember, we set our breakpoints here on line 11. And looks like we hit that breakpoint. My breakpoint in the method hello twice, on line 11. So, there we go. We can detect breakpoints. Now, once we know we've hit a breakpoint, what do we do? A lot of debuggers provide a little command line tool, kind of a command line interface that lets you start interacting with the program at that point, so you can query the program, see what its state is, see what's going on, and just kind of get information about how the program is operating. So, if we wanted to add a band line to our debugger, an easy way to do that is to use IRB. Now, many of you are probably familiar with IRB as the Ruby REPL, you can invoke it from your shell, run Ruby code on it, and see what happens. But it's also a part of the Ruby standard library, and you can call it from within your Ruby program. So, what we're gonna do is we're gonna use another method on the trace point, binding. This returns a binding object that provides a bunch of context information about where you're at in your program and what the state is. When you've required IRB, it adds a method, an IRB method, to the binding object that opens an IRB shell using that binding as the context. So, that's what we'll do. When we hit a break point, we'll open an IRB shell. So, let's try that. We'll look back at our program, and we hit the break point, and we've got an IRB shell. All right, now here's where things get interesting. We've got an IRB shell open with the current context here on line 11. What's going on here? Well, we have a local variable, recipient, that's in scope. So, we can look at its value, and we can see that it's world, right? We're in the context of an object. We have access to the member variables, such as sender. So, we can see its value. Since we're in the context of this object, we can call methods. Let's try calling hello, and we can see what it does. We can even change the values of things. You can change the state of the program. So, we have the recipient local variable. Let's change its value to, say, New Orleans. Let's change the sender, also, to Unity, say. So, we can change the state of the program. And now, when we continue execution of the program by just exiting out of our IRB shell, the program will continue with that changed state. So, now we've got a new output for the program. So, that's pretty much it. You know, that's the basics of a debugger. Real production debuggers will often include additional features on top of this. Since we have some time, let's try implementing a few of those. So, we notice that we continued program execution using IRB's exit commands to just exit to the IRB shell. Normally, a Ruby or a debugger shell will use a command like continue, or since continue is a reserved word, maybe caught to continue execution of the program. So, if we wanted to offer that as a command, that is add a command to IRB, the easiest way to do that is to add methods to a particular module, IRB extends command bundle. Any methods present in this module get added to IRB as a command. So, let's create a command called cont. We'll defer the implementation of this back to our mini debug module, and we'll pass in an object called IRB context. This is an object that's available to IRB commands and it lets you interact with the IRB session. So, if we go back to mini debug and we implement this, cont pass in IRB context, and again, to continue the program, we just have to exit the IRB shell. So, we'll do that, rbcontext.exit. Now, when we go back into our debugger, we should be able to use the cont commands to continue the program, and there we go. Okay, what else can we do? Let's implement stepping. So, stepping means we, so in our example, we break at line 11. Stepping means that you can execute one line of your program, and then go back into your shell so you can see what effect that line had, and you can do this repeatedly, so you can kind of step by step, line by line, go through your program, and you'll see how its state evolves over time. So, to implement kind of a line by line process, well, we already have something that lets us do something line by line, that's our trace point here. So, let's modify this to implement stepping. Instead of opening our IRB shell when, just when we hit a break point, we'll also open that shell if we're kind of in a stepping mode. So, let's create a kind of a stepping mode, and we'll initialize it here in our initialization, stepping to false. When we hit a break point, we can set it to true, and then as long as we're stepping, that's when we will open our debugger shell. Now, continue means you wanna continue your program and stop stepping, so let's turn it off here. Okay, so we've got a stepping mode here. How do we actually implement step? Well, first we need to create a IRB command for it, so we'll do that, step, and then we'll implement it. Like Kant, like continuing the program, stepping just involves exiting out of IRB, but unlike continue, stepping doesn't turn stepping mode off, so we leave it on so that the next time we hit our line trace points, we'll go back and open the debugger shell again. So that's stepping, let's try that out. We'll go back and look at our program. Again, we are breaking on line 11, so there it is. Now if we step, and we've executed a line, and now we're on line seven, which is here. So see what happened here. We were on line 11, we executed one line, and that was a call to the hello method, so the next line that gets executed is here, line seven. Again, if we step again, now we're on line 12, so we were here, we exited the hello method, and we're back here on line 12. Okay, and again, if we Kant, then the rest of the program will run. Notice what happened here, when we stepped that first time, we went into the implementation of this hello method. That's called a step in. You're stepping into the implementation of something that you're calling. So sometimes that's a useful thing to have. Other times you'll want to use a variation on this called step over. Step over means that if you make a method call, you'll step over the call to that method. So you won't break in the implementation of that, but you'll break on the next line of the method that you're currently in. So for example, from line 11, step in, put you on line seven, step over, we'll step over this hello call, and put you on line 12. So how would we implement this? We need to keep track of methods. We need to keep track of method calls and method returns so we know when we're back on our original method. Well, we have a way to keep track of things like that, method calls and method returns. Again, trace point. So we'll go back here and try implementing step over. And the way we'll do that is we'll just keep track of our current stack depth and every time we hit a method call or a method return, we'll modify that. So we'll initialize it at zero and then we'll create another set of trace points. That's trace points. Once again, we'll make sure that this gets called when we initialize. So this time we're gonna trace the call event. And we'll also trace be call. Be call is a call event for blocks, which for our purposes are kind of like methods. So each time we get one of those events, we will increment our depth. And then similarly, when we return from a method or from a block, we'll decrement our depth. So that will keep track of our stack depth. Now we know at any point in our program what, how deep we are in method calls. So to implement step over, let's go ahead and rename this so that we know what we're doing. Step over. So to implement step over, we just have to keep kind of record where we are, what our stack depth currently is, so that when we go and look at, should we open the shell again? Are we at that same method? So what we'll do is we'll kind of record this target, say target depth. We'll make sure that we initialize that also when we set up, when we first hit our break point. And then it's only when we're back at that target depth that we open our debugger shell. All right, so let's try that out. I hope I did that right. So again, we broke on line 11. Step over and there we go, we're on line 12. So that's step over. We saw step in, there's also step out, meaning that you don't break out into your debugger shell again until you've left the current method, so your one level, one stack depth above where you started. So as you can imagine, you can use that same mechanism to implement step out. I won't do that here for time, but it should be pretty straightforward once we have a way to kind of track our stack depth. So that's pretty much it. We have a debugger. We can set break points. We have a command shell that we can use to look at current program states, change things, see what effect that has. We can step through the execution of our program. This is a debugger that's usable. Let's see what it looks like. It's just about 60 lines of code or so. So it's not that hard. Not that hard. So let's see, since we have a little bit of time, let's go a little bit deeper. My team at my small company, among other things, we implemented the Ruby version of a product that we have called Stackdriver Debugger. This is a debugger that operates a little bit differently from the one that we just implemented here. It's designed for web applications, live web applications, where you actually might have a user on the other end that's waiting for a response. So you can't set a break point and then just an interactive shell because that kind of stops your program and you have a user waiting for a response. You also have to be careful about what you do because if you have a real user, you don't want to break things for that user. But it would still be really cool if you can still set break points, if you can still interrogate the program states and just kind of see what's going on. See, look at the behavior of your program all without redeploying. So that's what Stackdriver Debugger is designed for. Google actually has been running an internal version of this tool for some time. We just fairly recently released a version of it for the cloud platform, so cloud customers can use it now. But as a production caliber debugger, it employs, and it's a little bit different. So it employs a few different techniques and some more advanced techniques. I'll go through some of those so that you can see some of the other things that debuggers can do. So first, threads. Typically in a Ruby web app, you might have multiple threads running. Each thread might be running a separate request. And so when you're in a debugging session, you often want to limit your analysis to one thread, one particular request that you're interested in. You don't want other requests to interfere with that. You don't want to interfere with other requests. And so in particular, if you set a trace point, you want that trace point to be scoped to just one thread. Now, unfortunately, the Ruby TracePoint API applies globally. It applies to all threads at once. And so we kind of have to drop down to C. There is a C API for trace points that does happen to support thread scoped trace points. So there's no way yet to access it from pure Ruby. So you have to write a C extension for this. So that's what Stackdriver's Ruby Debugger does. Pretty much all or most of our trace points are thread scoped using this technique. In general, this is one of a number of kind of advanced Ruby features, Ruby VM features, that are exposed in C, but not yet in Ruby. Given how useful this particular one is for debuggers, it might be something that we should consider maybe exposing this one at the Ruby API level. Second, speed. In the mini debugger that we built, we used a line trace to detect breakpoints. Now, this means that we're executing extra Ruby code for every line of code in our program, right? Of course, this can be slow. This can be a major performance hit, and I'm sure some of you are probably thinking that as we implemented this. So we don't wanna hurt performance of a user application in a live debugger like this. So we put in a lot of optimizations into Stackdriver Debugger just to make sure that the impact on performance is minimal. One example, since line tracing is very slow, it just fires for every line, we actually selectively turn it on and off based on how close we are to a breakpoint. So for example, we can do this by using other trace points. This is kind of a simplified example of what this kind of technique could look like. You can listen to certain events that could indicate that you're moving from one file to another or from one method to another, and then trigger off that to check, okay, am I close to a breakpoint now? Is there a breakpoint in this file or in this method? If so, then you turn on your line tracer. So this is a bit of a hack. In an ideal world, we'd really love to have native breakpoint support in the Ruby VM. And it turns out that we actually kind of do, sort of, but it's a little trickier than it sounds. Different debuggers have different needs, and Stackdriver Debugger is a little bit unique. So for example, Ruby actually has an experimental C API called line trace specify, and it's meant for this kind of debugger that has line-based breakpoints, but Stackdriver, the way that we want to manage our breakpoints didn't quite work for us, and so we actually had to fall back to line trace points for our implementation. Overall, how to design a good API for breakpoints and for debugging in general is difficult because you have a number of different use cases, a number of different kinds of things that debuggers might want to do. We at Google have started thinking about this, how to do this for Ruby, and I'm sure the Ruby core team has been working on this for quite some time as well, so I hope that working together, we can start coming up with how this should really look. Side effects. Stackdriver Debugger, again, it's designed to run on live web applications. In such a case, it's very important so that you avoid changing things. You avoid changing the behavior or the state of the program. So the developer operating the debugger should be able to observe things that are going on in their program, but not modify anything, not change things that could break a user. So to help developers protect against this, we actually wrote a side effect detector and that's part of the debugger. It's not perfect, it's conservative, but you might find it interesting how it works. Side effects in Ruby tend to fall into two categories. First, there is object changes, so you have things like the changes to the value of an instance variable, for example.  We can detect these side effects by actually looking for these kinds of operations at the bytecode level. So we built a bytecode analyzer. Whenever we run Ruby code at a breakpoint, we compile and we analyze the bytecode implementation and then if we encounter any state changing bytecodes such as set instance variable, we throw an exception rather than actually running that code. C methods are another common source of side effects. It's really difficult to analyze the behavior of a compiled C method, so what we do is we whitelist C methods that we know to be safe and then we just prevent all others. So again, it's conservative. Now, preventing side effects, this is kind of, seems kind of esoteric, a little bit of a bizarre use case. Actually, it's conceptually really simple what we're doing. What we're really talking about here is immutability. Immutability for Ruby. Now, some of the other, I've been to some of the other talks and several people have been talking about immutability, often in the context of functional languages. It's very common part of the functional paradigm, so languages like Haskell, Elixir use it. And it provides things like performance gains, safer code, code that's easier to reason about. Immutability makes a little bit less sense for an object-oriented language like Ruby, but for our debugger, we do have a use case for it. Either something straight out immutability or maybe something else similar, like maybe we could have some kind of code jail that prevents one piece of code from modifying the state or resources owned by another piece of code, which, you know, come to think about it, sounds suspiciously like guilds, maybe? Kind of sort of. So anyway, just some additional use cases to think about as we think about the future evolution of our language. So we've gotten pretty deep here. Let's come back and recap a little bit of what we've seen. Again, at the basic level, debuggers are quite straightforward, actually. You set a breakpoint, and then you can observe what's going on, and for some debuggers, you can change things. You can step through the execution and see what's going on, how things evolve over time. We implemented a simple but full-featured Ruby debugger right here. We did it from scratch, it's not that hard. Then we talked about a real debugger, a Stackdriver debugger. We talked about some of the techniques that we used to implement it, some of the challenges that we faced. We saw that Ruby itself has support for some debugging capabilities, some useful APIs that can be used by debuggers, but there are some things that maybe could be added or changed to make things better, faster or safer. Now, if you'd like to experiment with the debugger yourself, here are a couple of suggestions. Bybug may be familiar to a lot of you already. Rails brings in by default. It's a traditional shell-based debugger, so it's good for Ruby command line tools, applications like that. For the web application case, you can try a tool like Stackdriver Debugger. These are both open source on GitHub, so you can kind of look at the source, see what's going on, how things work. In both cases, the core of these debuggers are exactly the same as what we just implemented. They're based on trace points. So now that we've seen a bit about what's going on under the hood, I hope that interacting with the debugger maybe just feels a little less intimidating. It's not rocket science, it's not dark magic. It's just Ruby. So, thank you for coming. We'll give you something. I don't know if we have time. We probably have just a couple minutes for questions if there are any. Okay, so the question is, in fact, let's go back and look at that. In our example program, when we stepped into the hello method, and then when we stepped, again, we kind of stepped over this puts method and exit hello method. So why didn't it step into puts? Is that right? Puts is actually, as I understand it, a C method. So you actually need to use a different trace point. There's a C call trace point that will detect that rather than line trace points. And then, of course, since it's a C method, it's hard to debug that using a Ruby debugger, so we didn't do that. Anything else? It's really hard for me to see, so shouts if you're, okay, I think that's it then. Thank you.


So good afternoon. Thank you all for coming. My name is Francisco Brasileiro. I'm with the Federal University of Campina Grande in Brazil. Together with Andre, we're going to present a work that we've done at the university to ease the task of people that have a bag of task application and want to run them on the cloud. So a bit of the outline of the presentation, we started with a bit of a motivation for the work. We show how, what's involved in managing the execution of a bag of task application. Then we show how the AuroBall system is designed. Then we show how we can connect good cloud using this middleware called Fogball. In between these two parts, we're going to show small demonstrations of the systems in action. Then we open for questions. So many science applications. I don't know how many of you were in the previous presentation where people from the University of Melbourne have said that 9% of the jobs that they received in their HPC system was for one core. So these are very popular applications where the computation, the size of the computation comes from the fact that you have to run many small computations. And this can be very easily parallelized. The other thing is that they come in sporadic demand. So for instance, in the university, two months before the deadline of a conference, there are researchers using the computing facilities to get the results of their experiments. But after the paper is submitted, there is a time where they sleep, they eat, they do other things. And so you don't have that much demand from that specific group of users. So this kind of, the other thing is that the faster you can run this scientific application, the better because you can speed up the cycle of research development since you can run faster your experiments. You can analyze the data. The data will be ready for analysis earlier. So you can plan a new set of experiments and run a new set of experiments. So the time to, here's not the time to market, but the time to paper is reduced if you have a lot of resources available. The cloud is a nice setting to run this application because theoretically it's the same cost. If you use the same amount of computing power, it doesn't matter if you do that in one hour or in one year, you are paying the same bill. So if you have lots of resources available, you may be able to run your application much faster. Okay, so what a typical user has to do to run a bag-of-desk application in the cloud. So the user is sitting there on his or her workstation and has access to a cloud service. So it has to create a VM. Then it has to stage input files in this VM. Then it has to SSH to the VM to start the remote execution. Then it needs to monitor this execution from time to time. And finally, when the output is ready, you have to retrieve the output data. And if in a public cloud you want to destroy the VM, in a private cloud it would be nice to do that as well, but some people don't. So this is quite time consuming and you have to do that for thousands of, of course, you have to automate this procedure. And this is essentially what the AHAbo service does. So we have a service that runs in the same administrative domain of the user and the cloud service. So what the user does is simply to submit a job through a command line interface to this AHAbo service. So what is a job? A job is a file that describes the tasks that need to, that belong to that bag of tasks application. Normally, this file will have hundreds or thousands of blocks like this one, where the only thing that changes is the input file and the output file. So normally you will have some, some script that will automatically generate this job file. It has also some other clauses that can be used. You have the label to name your file, the requirements for the resources that you will need from the cloud. These are comments that should be executed before any of these tasks. So when you create the, when you access the, the resource in the cloud, the first thing you do is to, to run what's in the neat part of this job.  So it's pretty straightforward what the user has to do. After submitting the job to the AHAbo service, then the user needs just to wait for the results to appear on its, on the desktop. So what the AHAbo service does is it creates the infrastructure. It stage in the input data. It starts the execution of the task, monitors the execution of this task. If a task fails for whatever reason, then it can be, it will be automatically restarted. And then when the outputs are ready, the AHAbo fetches the, the outputs. And most importantly, it destroys the VMs. So we'll shift gears here and I passed over to Andrei that will show how this works in action. Okay. So here in this first demo, I'm going to show you how AHAbo is used to automate the task of submitting a lot of smaller tasks to the cloud. And I hope you can at least have an idea of what is being showed here. So what happens here is that we have a job description file, just like Fubika said. And in this job description file, we have five simple tasks.  So it's the simplest task you could have. And after you get the task, the, the tasks described in the job file, what we would have you is that you would submit this job file to the AHAbo service. And here we're using the command line client. And before it completes, let me show you that the, this virtual machines here were already there. And in a few seconds, a few other virtual machines will be created from the AHAbo service. So let's change to the AHAbo dashboard, log in. Then we were able to see the job, click on the job. And then the, here it has already interpreted the job file, detected the tasks that will need to be executed. And now what happens is that it will start creating the VMs that will execute those tasks. And then if we wait for a few seconds, so this video is a bit faster than regular speed, the tasks are now ready. And the instances will be created in the horizon interface. So you see five instance have been created. Then if we go back to the AHAbo dashboard, we'll be able to see that the tasks will start running soon. Okay. So now they're running. They're running those five VMs that were created. And next what's going to happen is that when the execution finishes, then the VMs will be deleted. So one thing that the user may want is that the VMs be kept after the task is completed. Because then if you submit a new set of tasks, what happens is that you can reuse the machines that are already on. And that would reduce the overhead of the initial provision. So now the tests are completed and the machines will be slowly deleted. Okay. So Fubika, maybe you want to detail a little bit more about Fubika. Thank you. Well, D, so all is nice. But the problem is the user has a quota in the cloud. So I want to run my 200 simulation, but I can only spawn five VMs. I'll rather spawn 200 VMs and run all the simulations at the same time. So we need to find a way to increase the capacity that I am able to reach. So this is true even in the public cloud scenario, because out of the counter, the number of virtual machines that you can create even in a public cloud is limited. And this is because it facilitates the long-term capacity planning of these providers. So capacity can be extended in several ways. One way is using a multi-cloud. So you have access to several public clouds and you create VMs in all those public clouds. Another one is cloud bursting. So I have my private cloud. And when I exhaust the limit of the number of resources that I can use in my quota, I could jump to the public cloud. I could federate private cloud. So the idea is remember that this kind of workload is very ephemeral. So I will submit my 1,000 tasks job, but then I will stay quiet for a couple of weeks, digesting the data of that experiment. Meanwhile, my quota in the cloud would be available for the people to use. So we could do a kind of exchange of quotas in a federated cloud in order to be able to access more resources at the same time. And of course you can do combinations of all that stuff. So in the WebCG, we have also developed a middleware called Fogball. In fact, it's a suite of open source software that does different things. And among them, it provides support for multi-clouds, for cloud bursting and for federating private clouds. In the demo that we are going to show in a minute, we are going to focus on the federation of private clouds. Fogball also gives support to the deployment of opportunistic clouds using desktops that is particularly suitable to this kind of bag-of-test application, but we are not going to talk much about this today. So I think we can move to the demos so that you have an idea of how the system works, and then I will explain the internals of the system. Not today. So again, what we have here is the terminal that we use to submit a job composed of several tasks. And because this time the process is a little bit different because the high ball service will contact the Fogball service that federates two private clouds. It's useful for us to take a look also at the Fogball dashboard. And the Fogball dashboard here is showing that there is a user logged in, which is the high ball service, and we logged in as a user so that we could see in the UI the things that the service is doing in the background. In a summary, this user has access to 56 vCPUs, 36 gigabytes of RAM, and 67 instances. And these resources are spread into two different private clouds. So we have one here that is, it's a bit difficult to see, but it ends with rmp.br and the other ends with ufcg.do.br. And if you look at these columns here at this table, this table is showing the quota and the usage of the current Fogball service user. And here we can see that this user has zero instances in the rmp.br cloud and it also has zero instances in the ufcg.do.br cloud. And you can also see that there is additional instances here. So there are three instances in this ufcg cloud. These are the same instances that were already running in the previous demo. So let's take a look now at the job. So here I'm showing you again, so there are no instances being created in Fogball at this moment. So there are the same three instances in Horizon, which are the same three instances that were there at the beginning of the previous experiment. And let me show you the job file. So the job file this time has 15 tasks and these 15 tasks have a larger sleep time. So now I put here 75 seconds because we needed a bit more time to be able to access VMs in the different clouds, which take a bit more. Otherwise, all the tasks would run in the local cloud and then we wouldn't show what we want to show. Could not be a good demo. So again, using the CLI client to submit the job and okay, then we can change back to the eyeball dashboard, login, see that there is a job pending. Okay. So here we see that the job has around 15 tasks. The 15 tasks are ready to run. So it means that in a few seconds, the instances will start being created. So let's go back to the Fogball dashboard. And here we see something that it's interesting. So the way that the Fogball service works is through asynchronous requests. And what you see here is that the eyeball service requested 15 instances. And we know already from the previous demo that there would be not enough space to create the 15 instances in the local cloud, so the UFCG cloud, because the three existing instances have already used a large amount of memory. And as the VMs are being created, the orders will be fulfilled. Okay. So four instances created. Now seven instances have been created. And one other interesting thing is that it will prioritize the execution in the local cloud as we would expect. So what you can, what you could see here, if it was a little bit better to read, would be that all this instance ID, they have the name of the cloud embedded in it. So this one is maybe a bit easier to read. The instance ID ends with usg.edu.br. And these eight instances were created in the local cloud. But eyeball requested 15 instances, as you saw on the previous order page. So then the Fogball will help creating the additional instances. But let's take a look here at the local cloud. So horizon, the instance have been created in the local cloud. So eight instances. And if we update the page, then we would be able to see that the additional instances, so the seven additional instances have been created in a different cloud. So if you cannot read the name, at least you can see that these names are longer than these other ones, because they are in the, they use the other cloud name, which ends with rmp.br. So I have seven additional VMs here. And now that the VMs are created, we would be able to see here in the usage page that the, the high ball service user is using its quota. So here we can see the UFCG cloud. So it has 11 instances in total, but eight for the current user, which is the high ball service user. And we see that with these eight instances, we have already reached the RAM quota. So that's why it could not create the 15 instances. And the other seven instances were created in this cloud here, which is the remote cloud. So now we have our 15 instances to execute the 15 tasks. The tasks will be running soon. Okay, so we have one running. Let's update. And then we have a few more running. So they're running in the VMs that were created both in the local cloud and the remote cloud. And now some tasks are already being completed, because it was not a very, a very long sleep. So if we take a look here at the instance, we'll see that some of the instances are already being terminated. Okay, so one configuration that the user could play with is that the, the instance could be held. But here for this demo, it's better not to use that. So when there are no ready tasks, the instance will be finished as soon as the tasks are completed. Should be very soon. Yes. So now we have all the tasks completed. And then it's just a matter of seconds to have all the instances deleted. So here you see in the dashboard from Fortbow that all the instances that have been created through Fortbow have been finished. And we could also go to horizon and refresh the page and see that all the instances created in the local cloud have also been deleted. Just the three original instances have been kept. So that's, that's it for the second demo. Right. Well, what I'm going to do now in the next five minutes before we move to questions is to speak a little bit what's going on behind the, under the his sabotage. So how does Fortbow work? So the idea is that we are building a layer on top of the, of the cloud orchestrator to deal with all the, only with the federation aspects of the, of the problem. So essentially we have two services, one that implements a discovery service. So it's this membership manager here and an allocation manager that interfaces the, the other, the users with the, with the federation. One thing that we didn't mention in the, in the demo is that those two clouds they use different cloud middleware. So the UFCG one uses open stack, while the RNP one uses cloud stack. So through Fortbow, Fortbow provides an OCCI API. So the ARBO services uses these OCCI API to interact with any cloud orchestrator in a transparent way. In fact, we can, we can also use settings where we have cloud bursting in, in, in private cloud. We have adapters that will work with Azure and AWS as well. So the first thing that one of the main services that the Fortbow layer provides is the, is a service for the federation of identities. The, the service works in, in, in layers. So at the federation layer, a local request will come with the federation credentials. These credentials are authenticated and authorized at the appropriate service. And this service can be configured. For instance, that cloud that we showed uses a federation of LDAP providers. But we have implementation with VOMS, which is people, anybody from the grid, European grid initiative there? Well, never mind. And, and another one that works with SHIBOL. So the, the deployment that we are doing with the Brazilian N-Wren uses the federation of identities that they, they run that's based on, on SHIBOL implementation. So these are authenticated and authorized at, at the federation level. And then you have a mapping to a credential on your local cloud. And this credential is the one that is used to access the cloud. So what you make available to the federation, to the different users will be defined by this mapping. As we've seen, that is a possibility to send requests that cannot be fulfilled locally to a remote cloud. And at the remote cloud, you go through the same authorization and authentication process with the addition that you also have the credentials of the football manager that is sending the request. So that is sending the request over. So you can use this information as well to define how you map or even if you are going to allow that user to use your resources. An important thing is that mapping is, is defined in an autonomous way by each cloud administrator. So that's why we call a federation. It's that there are some rules that everybody needs to follow but there are rules that each member of the federation may have the flexibility to define. The mapping is one of those. The middleware that you use to operate your cloud is another one. Football was conceived to be easily extensible. So we, the architecture is based on plugins and we have two kinds of plugins. One family of plugins is the interoperability plugins and it's in charge of allowing football to communicate with the underlying cloud. So if you want to provide football support to a new middleware, what you have to do is to develop the interoperability plugins for that middleware. And the behavior plugins change the way your node in the federation behaves. For instance, you can change the way you prioritize the allocation of remote resources. So one site might follow one policy while another site follows a different one. The other thing that we took care was to deal with the fact that in some of the private clouds that we needed to put in the federation that drove our development is that they don't offer public IPs. So they are private clouds, so they don't need to offer public IPs to every VM that was created. So we created a tunneling service that automatically provides a public IP to every VM that is created in the private cloud. Another thing that we use is a messaging service to allow all these components to exchange messages without having to open strange ports in your firewall. So this is used in our current implementation, XMPP. Next one. Some success stories. So football was developed in the context of a project that was jointly funded by the Brazilian government through the CNPQ agency and the European Commission. And in that context, the idea was to join clouds in Brazil with clouds in Europe. The clouds in Brazil were running OpenStack while the clouds in Europe were running OpenNabla. These clouds also belong to the EGI federation, so there are some services that the European partners wanted to use, for instance, this VO management system that they use. So we developed the behavioral plugins in football to allow the federation to use this federated service. Another case that we are currently deploying in Brazil is with the Brazilian N-Ren. The idea is to federate clouds in several institutions across the country. And these clouds use at least three different flavors of orchestrators, CloudStack and OpenStack being a tool of them. We are using, as I said before, a federation called CAFE for the identity provision at the federation level. And this was the cloud that we used in the demos that we showed before. And I think I conclude the talk with that, and I'm open to questions if you have any. I was told that they are recording, so if you want to make a question, if you can come closer to the microphone, so that the question will also be recorded, it would be great. Thank you. Thank you. Thanks very much, guys. What do you do if your token expires? I mean, the Arabal Service User Act is on your behalf, yes, but does it actually create new tokens? Does it cash your credentials as well, or does it just use the token and then, when that expires, are you unable to create new bag of task VMs? You mean the federated token that, as a user, you authenticate to... Well, it depends. Are you talking about VOMPs or any token? It was the federated acts authorization and authentication token. I think it was a few slides back. So the Fogbo authenticates to the cloud. So Fogbo has credentials to the cloud, and you may have credentials to Fogbo. So there could be different levels of authentications and different tokens in use. That's what he was asking. So any case that you're using Fogbo, Fogbo has its own means to authenticate together with the cloud. So it can continue working on it, whether your authentication with Fogbo has expired or not. I think the question was, if I create a VM and during the lifetime of my VM, someone changes my credentials in such a way that I wouldn't be able to create that VM. Yeah. Or if the token that you used to create that VM then expired because it was so old. But I don't need that token any longer. Once the VM is given to the user, it has a VM and an SSH to the VM, and the token is no longer needed. Okay. So all of the VMs that are created are created at the first moment. And... No, no, no. Right, yes. Anyway, the VMs you still need to be deleted. So you still need the access to do this. And that's done in the context of Fogbo, if you're using Fogbo. I mean, when you need to create the VM, you need to have the authorization to do so. After that, we don't... It's like opening a file. If you open a file and you have the rights to read the file and you keep the file open and someone goes there and changed that permission so that you can't read, you will still be able to read the file as long as you have the handle to the file. So it's a similar kind of situation. So it's a breach, but... Okay. What you can do is you could have a service at your cloud that would periodically access all the... But we don't do that. Any other questions? So I'm Samuel. I have a question about how the schedule works in the case. I have... Like in the demo, we have two clouds and we had other 15 VMs and some of the VMs needed to be created on the other cloud. What if I have more than one remote cloud? How can I define the behavior that let's say I want to balance that like half go to the second cloud and the other half goes to the third cloud? Currently, we have a very naive scheduler in our ball that simply passes your job, assesses the amount of resources that are needed to run that job and sends the orders to for-ball. And as the resources appear, it uses whatever resources is there as making no differentiation between them. Our plan is for the our ball service to... When submitting your job, you might also submit a scheduler together. Yeah, exactly because you may have different contracts with different remote clouds. Yeah. Okay. Thank you. And even different applications might have different ways to schedule the task. This is just a proof of concept to show that... In fact, what we wanted with our ball was we developed this Federation middleware and we showcased it. Everybody was very happy with that. So we wanted to have users into this Federation to try to use it. And bag of tasks seemed to be the easiest way to get people on board. So the our ball is just a facilitator for people to join in the Federation without much work with a slow, I mean not steep, learning course. Congratulations. Thanks. Can I... I came in late so I didn't miss the first part, but can I assume that football is federating instances, right? It federates create instant creation, VM creation from, you know, different clouds. The Federation, it can be different levels, right? It can federate identity, it can federate your cloud so that you can share resources like this one which is basic, this resource is basically just pure instances, VMs and nothing else. We can also federate, we can also... Well, what we do is with football allows you to have access to a number of clouds in a Federation to create basic resources. What are these basic resources? You can create VMs, you can create volumes and you can create networks within a single cloud. For instance, I cannot yet create a VM at one provider, a volume in another provider and attach this... We have done that. Yeah, I've seen the massages. Yeah, I would love to, I would be really interested in, you know, getting gap, but in any case, so one thing that, so you, what kind of Federation, like when it came in, you have already have, you know, multiple clouds up and then you have, you know, resource assigned to it. Is there any special schedule that does that or this is all manual, like this cloud will have only this kind of resource? You mean because there are already instances in that project? No, yes. For example, this cloud can only have, like, you know, 10 limit number of instances that can be starting up and the second cloud would have 20. Do you have some sort of schedule for that or how do you... Yeah, that's it. Yeah. Well, I'm going live now, so I'm not responsible for what to... So this is the Federation, the Brazilian N-Ren Federation. So we have those two clouds and a catch-all cloud for people that don't... So we have one of the models, the business models that we use in this cloud is that you have the notion of your local cloud and you have priorities over the resources on the local cloud and then you can cloud burst to other clouds. And you get priority on the other clouds based on how much resources you offered along the time to the other clouds. Do you teach how Paul Paul knows that this pool would have this kind of resource? Do you manually type in it? No, no, no. It... Perhaps you... We may be running out of time. Yeah. I'm just... So this component here is the one that does all the interactions with the cloud, but there is this component here that implements this directory. So this is the one that... And that information that appears in the dashboard is user dependent. Like I logged in as a certain user and then the resources that are made available for me, depending on my credentials. So you might log in in the same system and you will have different... And this is all based on the mapping that is done at each local cloud. So the administrator of this cloud defines how much resources I can access, how many resources you can access, and this service puts all together. Okay? I think we are running later. We'll be around. So if you have questions, we'd love to talk. Thank you.


So I will talk about the breaking code today and getting cheating in contract breed. About us, Rezo is leading some sort of company for doing trading on the stock exchange and Amir is an accountant who works for let's say one of the big four companies in accounting in the world and I'm a researcher in the field of trusted computing and virtualization in the College of Management in Israel and in the University of Iwasco in Finland. I lead truly protect startup in the trusted computing field and I'm also a breed life master. Amir by the way was the European champion in breed 2003. So everything we describe is something that interested us. To study we are not really crypto geeks, this is all basically hobby work. Now about today's talk we will talk about the scientific aspect of breaking codes and some real life example. This is really about some top pairs like best players in the world being found cheating in breed in 2015. We focused about two Israelis, Mr. Ron Schwarz and Mr. Lutan Fischer. We focused about Israelis because we were Israelis. We were actually sitting in front of the computer rooting for the local team to win and we actually tried to protect them even when the acquisition arrived and then we found out we tried to protect them and we found out they were really cheating but not in the way people thought they were cheating. So that's what we describe and we represent our findings and also other people's findings and whatever was not found by us we will basically say if we seen further than others that's because we stood on giant shoulders and we were able to actually see further. We basically comment mostly about the Israeli Breeds Federation talks which are available online. We are not invited to the American contract bridge hearings or the European bridge league hearings and these are not available online but we know more or less what happened from rumors. So first of all why would somebody sit in bridge? I mean that sounds like a game that like four old ladies play in the kitchen and doesn't sound like something that you'll make a lot of money. So apparently first of all bridge is not a spectator sport. Nobody will pay to watch bridge like people are paying to watch football or basketball or boxing. People are not paying to watch bridge. However people actually will pay money to professionals to play with them or to play in a team with them and as a result there are professional bridge players making significant amounts of money. Let's say between 200,000 to half a million dollar a year which perhaps not compete with Cristiano Ronaldo but it's enough money that somebody would want to compete for to play for and on the plus side the career of a bridge player begins let's say semi-professionally at the age of 14. Let's say some play professionally at the age of 18 and it can go on for 35 to 45 years on average. Some play professionally even when they are 90 years old, 89 years old. So it's pretty long career and it's a pretty high paycheck. There is motivation to actually sit. It's not just something that you would see for fun. Okay, so I'm trying to talk as much as I can about bridge without actually teaching bridge. So bridge is a game that played between two pairs north south and east west. There is nothing special about the direction. It's just the names like I can play north south in one board and on the next round I'll be east west. There is nothing special about it and you can actually rotate the board. In a team's game four players plays against four players. One team sends a north south pair to play against the east west pair of the other team. The other team sends north south pair east west pair to play against sorry north south pair against east west pair of the first team. Now in a normal game of bridge you play with identical cards to minimize the element of luck. So our north south plays the same card as their north south. Our east west plays the same card as their east west and the cards are held on something like that called the board. Each round you pull the cards from the board and you see your cards after you end the round you return the cards to the board and they are playing identical same cards on the other table. Now a board usually takes about seven or eight minutes to play. A short competition takes about 24 boards or about three hours and let's say World Championship takes about, takes hundreds of boards and several days. If you play every single board in a World Championship and you reach the final then you play something like 800 or 900 boards. Now the last thing is that in bridge everything that you play, every card that you play, everything that you do is going to encode some meaning describing your hand to your partner. For example a high card followed by a low card may indicate that you have an even or odd number of cards in that suit that you favor an higher ranking suit or a lower ranking suit that you like the lead or you don't like the lead. Everything basically is encoded there. And this encoding is not cryptographic which means the other party if I'm north-south east-west is entitled to know all my encoding methods or everything that I, everything systematically that we do, the other party is entitled to know. There is full disclosure. And in the last, the last thing is that in bridge like in many other sports most of the times where you actually lose points you screw yourself. Most of the acts that actually cost your point or win point to the other teams are self-inflicted. You try to guess what cards happen to be dealt right now to partner based on the beating, based on the play of the card until that point. Sometimes you'll get it right, sometimes you'll get it wrong. Usually good players will get it right more often than bad players. So on a single round there is let's say 50 plus some epsilon cents for the better team to win. If you play significant amount of rounds then eventually the, the better team is vast favor to win. I mean, let's think about the casino here. Sometimes the casino pays the gambler but the casino in the long run is usually vastly favorable to win. So that's all about bridge. I'm not going to talk about bridge anymore. And sorry. There are basically two types of cheating. The first type of cheating which is also the most common and that's the one I'm going to focus on is conveying information that illegally not true game means. Game means means that, for example, you cannot convey information by making faces, making sounds, I don't know, pulling your tongue, scratching your head. That's basically illegal to convey information in that matter. You can only convey information using the card that you play or the beats that you make. The second type of information of cheating which is rare and I'm not going to deal with but it also exists is reporting results that on a method never happened. Let's say I pay you to say that I won or something. Falling the dealing of the card, falling the replication of the card, falling the board, illegally looking at your opponent card, trying to play a card that was already played, using illegal substances. I don't know. All that stuff is considered illegal but it's done much less often than the first one which is by the way not very often at all. I mean in the world class level, bridge is being played for about eight years and there were let's say less than 50 cases of cheating, less than 20 cases or 30 cases of actually conveying illegal information. So first of all, to prevent conveying information even accidentally, you don't really speak when you play read. You may see it in the newspaper like North opened one spade or one club and sorry, what it means is that North pulls like something like that from a reading box and places it on the table. So there is no like forceful one club, loud one club, hesitating double, forceful double. That's basically all gone from the real game of reads. Information is not supposed to transfer and oral bidding is really rare unless it's really like an amateur game. Now in professional games, they used to play with those boards. Then in 1965, two British players were found to convey information by the way they held the cards. Like if you held it in a certain way to convey some information about the distribution of hearts that you have in your hands. And as a result, in tournament playing professional games, there is like a screen that goes diagonally over the table and east doesn't see west, north doesn't see south. You don't see your partner so you can't convey illegal information. Now as we will see in a second, screens did not prevent illegal information passage but they actually made it more difficult to transfer information. So as I said, there is a screen that goes from north west to south east. Basically south and west see each other, north and east see each other but every player doesn't see his partner and as a result, you cannot convey information and you cannot transfer information to your partner illegally. Now that did not prevent seating. As it occurred in 1975, two Italian players were caught tapping each other's foot under the screen which led to first small t-tables to be placed under the tables and then that led to screens to go actually under the tables. And in 2013, Coing and other sounds were actually found to pass the screens which I will soon demonstrate and all that happened before the main events actually occurred. Okay, so that basically demonstrates, sorry, how breed is, oh wait, sorry, for some reason. Okay, fine, let's do it like that. That basically demonstrates how breed is played in a tournament play. Basically, there is the screen, there is the beads and there is like a tray with the player's passes and they transfer the beads. When the play is basically over, people collect their beads, remove the tray, put the board on the table, open a small shutter in the screen so they can see, they can see each other play and basically play. Now, leads us to the event that happened in 2013. These are basically two German players that were actually caught hitting and here is one thing that they did. You see, already seen his card, he's now placing the tray and he makes like a short hand movement that actually says I don't have anything in my hand and that's illegal the passing of information to his partner and that was actually found long before we approached the field. Another thing that they did, if you see, is actually coin and that indicates that he has short page in his hand. He has only one page in his hand and here again he's going to call just once to show that he has only one cloud in his hand. So that was actually, that's actually demonstrate how bridge professionally is being played. By the way, those two Germans that were caught hitting were also the world champions for team years, that is for players above 60 or something and they actually, they actually been caught red-handed and removed from play for basically life. Now we go back to our story. That's Boye Bruglund, he's the hero of our story. He's a bridge professional, he hold the World Championship for 2007 and after the event that I will describe, he won the 2015 Personality of the Hero World. Those are the two cheaters that we basically help to uncover. Lutan Fischer and Ron Svartz and in 2014 and 2015 they basically won everything that you can win. They won the World Championship, the Transnational Teams, they won the European Championship with Israel, they won three different North American Championship. When we caught them, it was like catching Lance Armstrong cheating. At the time they were one of very few pairs that were considered like really in the World Top and a very short list of pairs and that guy is not a professional bridge player or anything, but for him it was important for us to have those videos. He actually started taping videos and allowed cheaters to be uncovered and he put all the videos online of tournament plays in Europe and if it weren't for him, catching of cheaters would have been impossible and as we will soon discover, most of European top pairs were sitting at a time. So what happened or sorry several top European pairs were sitting at a time. So what happened in 2015 North American Bridge Championship? So Boja and his team were playing against their ex-teammates Lutan Fischer and Ron Svartz. They were actually teammates before the play and Ron and Lutan took a series of let's say ethical or bridge questionable decisions that led Boja to suspect that they are well sitting. Boja created a website bridgeheaters.com, by the way it's still online if you want to come check and got sued for one million dollars in like for slur and Boja called for basically crowdsourcing people to watch the bridge games between Fischer, Svartz and their opponents and try to find the code they were using to cheat. A Swedish player, Perula Kulin, breaks we will see what breaks means but he supposedly broken the code. The code as he described it was when they are sitting north-south and they are pushing the board. They will put the board on a certain position depending on which shoot they want lead. For example if I want a spade to be lead I will push it towards partner. If I want club to be lead I will push the board towards me. If I want diamonds to be lead I put in the middle. If I want heart to be lead I put it on the side and if there is no preference then I just put the board somewhere and not remove it until partner leads. That's basically the sign for no preference. So that's bridgeheaters.com that's the website and that's basically the accusation against them. Basically in all those cases they put the board in the center and that's where they are actually declaring the hand. Now when they are defending the hand they are taking the board and put it somewhere according to a signal. Sorry that's basically all the normal position. Now you see actually pushes the board to a position that's not normal. So you push putting it on the side and that's actually a case for an art signal, again a case for the heart signal, again a case for the heart signal. That's basically the irregular signaling here against for the spade signal. That's basically the irregularity and here is the code. So when he pushes it all the way through that's the signal for spade. When he places it on the side that's the signal for heart and when he places it in the middle that's the signal for diamond. When he keeps it in his side of the board that's the signal for club. That's how it looks like when I'm supposed to lead. So he pushes it all the way for a spade and spades the spade. He makes the unnatural move of placing that in the side after it was actually in the middle and that's called for a heart and he places it exactly in the middle and starts lead diamonds. Yeah, we looked at something like 40 or 50 hours of videos trying to write everything that happened and those Perola Collin and the Swedish guys they did the same. We looked, we scattered counting like times it touches the shirt, times that he called, times that that's like 40 or 50 hours of probably the most boring show in the world but in making lots of observation. So Kit will see another world champion, oh sorry, Kit will see another world champion has actually wrote an article in Bridgewinners.com that's the most popular bridge forum about the events. He basically convinces the most of the world that they are sitting and what happens now? So first of all Ron and Lutan are bent from playing by the Israeli Bridge Federation, by the European Bridge League and by the American Contracted Bridge League. So they cannot play. The three organizations independently charges them of ethical misconduct and Israel resigns from the World Bridge Championships, Sweden takes its place and the world moves on and finds three other cheaters that we never discussed. The world number one and number two players Flavio Fantoni and Claudio Nunes are found cheating and delivering information, by the way they play the card if it's played diagonally or horizontally on the board and they are bent from play and Monaco resigns from the World Bridge Championship. Smirnov and another German players, not the ones from the videos at first but another pair is also found cheating, they admit and they are judged by the European Bridge League and by the German Bridge Federation and others Budzinski and Cesar Balicki to Polish players are let's say bent from play suspected of cheating and Poland does not resign from the World Championship and also wins the World Championship with two other players, not with Balicki Smirnov and we are not happy. Now, why are we unhappy? So we are unhappy with the scientific method, we think that it might have been affected by prejudice against the pair that people didn't like the pair. There is no fresh data, people looked at boards as they were already played and developed a method based on the board that were already played. The sample size it was actually limited to some international event to the European Bridge Championship which we found too small and we had, there were only 11 suspected boards of cheating and out of them we had two significant outliers that we thought okay that's impossible, a pair in that level World Champions, they might get one wrong but two out of 11 is basically two mats and when all this started at least two of us were defending the pair, getting a lot of abuse in bridge forums, trying to prove that the scientific method that declared them cheaters was insufficient. Rezzo actually defended them online and actually gave a speech and testimony in the Israeli Bridge Federation. I questioned the research method online. Amir was the only one that was convinced that they were cheating but still was unhappy with the result and we all had to explain like why Svart's movement looks so unnatural. I mean let's say they are not cheating, why do they make all those weird movements and unnatural movements to cheat? We respect to Pritzadis, I would say that Pritzadis and Svart were involved in cheating as juniors and they are far from being the nicest players in the world. They are obnoxious, they are show-offs and it's very possible that players wanted to put them out of the way. I mean they are proud and annoying. French boards, so oh and by the way several board that were inconclusive mysteriously changed their marking over like the internet to feed accusation against them better which was again weird. French boards, so the same boards were used to raise the hypothesis and to actually judge they are guilty and that doesn't work that way because Brits played by human beings, human beings score, human beings need to go to the bathroom, human beings make some noises, human beings place the boards not so very accurately at the table and if there was no separation and there was no prediction you can just say that okay you did a Texas Arpshooter and you draw the mark around where you actually hit which is not something that is scientifically possible. How do I'm supposed to deal with no fresh data? So first of all there were more data, there was data from the Polish Premier League for Brits which Fischer & Schwarz played as like a sponsored players, there were data from a other tournament that they played and somebody actually suggested to use that as well. Also there was data from boards that they were playing east west. Now when they are playing east west they are not start with removing the tray but they can somehow touch the board and we thought okay maybe that's signaling in some way and there was a group and I think that that was what's done in the European Bridge League I'm not sure because I was never present in their like discussions that's right to use the code but with the east west hand as well as the north south and nobody argued that the east west and are basically new data fresh data and there is one more method that I used and that's something that using archaeology how do you determine if something is historically significant when you find something I mean you cannot model anything in advance because you don't know what you're going to find and then after you found it how do you say it's significant or not and we found a paper from 2007 analyzing a case where they found the tomb of Jesus in Israel they found the tomb of Jesus this they actually say like a several important names from the New Testament like Jesus, Matias, Mary Magdalene and they all found it in the same burial tomb and they tried to figure out what are the odds to get so many important names so many important historical figures in one burial tomb and they said okay that's the grave of Jesus and basically what they did was to multiply all possible names as were found in other historical information and then they said okay so we don't have fresh data but we've demonstrated the odds to find something as significant is very low and if you start multiplying then we figured out that the results are not that significant in bridge anymore so they didn't like using that method now the kid will see you actually posted the original accusation have sent me a private message claiming that the true test for hypothesis is the power of prediction he basically agrees and then he said okay they used just one set of boards I predicted using the three other sets except that wasn't what happened I mean there was a crowd sourcing call and all the hands were actually available they actually reviewed all hands and then they raised the hypothesis so you cannot check the hypothesis that you raised using four sets of and on three sets of and simply because somebody put just one set of and on the website so we disagreed the european bleed bleed league had a hearing they argued that okay we can use the the four east west end the five east west and from the same tournament where they actually touch the board and use that as fresh data and I would say okay that's fresh data but then the north soft code included five options including where you don't touch the tray and the board remains for no preference and that doesn't work well for east west because east west doesn't touch the board most of the time and north soft are charged with removing the tray so they developed a four a four-way signaling and they just removed the no no preference and they said if we don't touch the board that means nothing that's like random noise but that makes the east west hands and not fresh anymore in our opinion because you only develop that method after you watch the east west end you show that there is no sign for no preference and you decided what it means I mean you could say that putting the the tray on smart side or on wrong side is no preference or anything like that and you picked specific meaning so we said okay the east west hands are not fresh anymore then we said the sample size is too small all in all there were only 37 boards were not where we're fissure and smarts defended there were only 64 boards in question they played 27 hands they were on defense the 37 hands out of them a signal was actually given and a signal was actually given where there was no weak lead there was no a board that remains for no preference and there was no case where it's seamless for diamonds and in that case the board actually ends in the proper position and the signal was actually given in only 11 times and out of which nine were correct and if you consider the importance of archeological finding nine out of nine out of 11 is not really sufficient and fortunately more boards were available so we looked at other torment but first how was nine out of 11 calculated so experts were given a and by the boards and said if you had an illegal signaling method what would you signal with such a hand some experts would pick let's say spades some would pick signal for heart some would say signal for no preference and then if any of the experts chose the signal actually on the table that was considered true positive however experts are not always having the same opinion on certain boards experts can pick three or four different opinions so the result of what would you signal is basically close to a to a coin toss and not really to to one out of five or one out of four cases and there were just two significant outliers that couldn't be explained I had the world-class player tell me I'm sure they cheat but this is not a way that they cheat it's impossible they see this way I want them to be caught for real the first outlier was a dead ball where Svartz was supposedly signaling for clubs and if you look at Svartz hand he first he beat spades he beat three spades then he has the king queen fourth of hearts he has the queen jack six times of of spades I could find a reason to to play for a spade signal or to play to play for a heart signal or even to play for no preference signal but one thing that I'm not going to choose is signaling for clubs holding a no clubs that's basically impossible another outlier which is less obvious is that hand where Svartz holds king jack fourth of clubs but he actually doubles five clubs so he says that he has clubs and once more no expert would actually choose a club signal and if you look Fischer actually signals plays spades and Fischer is not likely to hold clubs in that auction anyway so those two are basically two unexplained balls that couldn't be explained or adjusted or anything using the old method and as I said are we alone so the vast majority let's say 90 percent of the world is more or less convinced that Fischer and Svartz are sitting and that the method is actually what was published in Bridgewinners and there is a minority that rejected the claim including Mr. Inon Lyon who is world champion in mind games Olympics in 2014 and he is like actually giving testimony that that's impossible that that's the method of sitting that two world class players would use and they get support from professors from Weitzman Institute of Science and CERN claiming that the statistic was actually wrong Mr. Ilam Gross both Inon and Ilam have actually gave their testimony in the Israeli Bridge Federation and it's available online. We believe we don't know that sponsors paid for their defense and Ron claimed that the defense costs one million seconds which is between one quarter of a million of euros or one quarter of a million of dollars so something we think somebody else paid for their defense and we we try to look at the stuff and what did we find first of all we find that if you switch the spade signal and the club signal and look at the same data you get eight out of eleven that are correct so nine out of eleven doesn't sound like too much if you can switch the signals and still get the same results by the way the signal only includes three shoots spades clubs and hearts and we switched two of them and got the same result so it doesn't sound like too much and then when we try to look at other stuff which requires some bridge knowledge but when we look at odd even ends or when we look at high car points we can find other signaling methods that are just as good getting the same nine of out of eleven but then when we look at other ends we find that the other codes collapse and the code proposed on the internet is usually working okay but introduces one extra outlier from the police bridge league so perhaps it's not a correct code after all but again we are not sure why the movement of the board is so weird so we look at the hands some more and we find out that fissile signals like three or four times more than svarts fissile signaling whenever is on defense svarts signaling whenever he actually has something to say and we start with the hypothesis that let's not use near far as pure calling suggested but let's use near svarts near fissile signaling method which is basically our innovation so if you look at fissile signaling that remains the same because near is basically near fissile but if you look at svart signaling then his signaling is actually the opposite of calling method and all of a sudden basically it all makes sense we have 11 out of 11 everything is correct we don't have any outliers so we wanted to test it further and we looked at additional board that they played we watched at some more hours of basically four fat men playing bridge we found seven more hands from the polish bridge league we found five more hands from the european bridge championship where they played east west we found one more bridge and from the north american bridge championship and we found we found one more hand from the european open team championship in tromso and basically it all makes sense it all fits and then we have to figure out what to do and i'm taking a legal detour now i'm saying once more i'm not a lawyer i'm not so sure that everything that i described is correct but fissile and svarts were facing three different trials on three different jurisdiction the american control bridge league the european bridge league and the israeli bridge federation and we are not sure i mean they already have submitted the accusations can you change the accusation while the trial still goes on and if you can't what does it mean if we publish that the code is wrong does double zeppel the applies i mean if they are found innocent of using one code can you accuse them of using the other code and if they are found not guilty are they off the hook and do they need to be proven guilty beyond reasonable doubt or is there a more electrically criteria we don't know but i can tell you what we decided and we decided to wait until the hearing in the european bridge league and not publish anything by them because we thought that okay the code is not like airtight it's not correct but we think that it's enough to actually convict them we don't want to to mess with the the legal case that actually happens and we had everything unpublished until european bridge league actually delivered the verdict but we did talk about the results and it's possible that the judges in the european bridge league knew that the code is not correct but they knew that there is a correct code that they cannot modify and they cannot actually use to solve the pair after the ebl generated the guilty verdict we actually released our results and the israeli bridge federation for sure and we think the american contract bridge league we held a trial in august we think that they used our code to accuse the pair so the aftermath for fissure and waltz the european bridge league convicted fissure and waltz of sitting and bent them for playing for five years and they also issued a massive fine which we think that fissure and waltz will not pay and they will be bent for life the european bridge league actually is noted that they are not using proven beyond all reasonable doubt but only proven to sufficient probability they are sitting perhaps it's because they heard of our code the american contract bridge league expelled fissure and waltz for good they removed them for to play for life the israeli bridge federation which we think had the most Torah examination and the longest hearings have concluded its hearing and is supposed to release a verdict but after the two previous verdicts were actually published without anything but a guilty verdict will be released as for the other theaters that we didn't deal with the italian bridge federation and the european bridge league convicted from tony nunes the american contract bridge league expelled them for life the german bridge federation and the european bridge federation convicted piekrak and smirnov and the ebl is still reviewing the balicki-zmuzinski case we think that they actually i don't want to give predictions fissure open up a pub smart fissure this degree in chemistry and they are hopefully outside of bridge world for good we wish them all the best but don't play bridge ever again and what did we achieve so fissure and waltz are outside the world of bridge hopefully for life the sponsors deserted the fissure and waltz fissure and waltz reported that they fired their lawyers that's when we heard that it cost one million shekels to actually pay them and there is no longer a thread of doubt in anybody's mind that they are worth sitting all the players that supported them initially deserted and i still argue about scientific method with some professor from chicago and perhaps the argument will never end some advice for cheaters if you do cheat try not to be obnoxious one of the way the reasons that from tony nunes managed to cheat for about 20 years and fissure and waltz were caught after one year after cameras were available was because fissure and waltz were obnoxious and basically raised the entire world against them and we figured that it seems pretty easy to develop some unbreakable code for example generate some pseudo random string which you cannot break and just move the code rotate the code based on the pseudo random string and that's basically unbreakable and if you are caught sitting and somebody is accusing you of using incorrect code set up one of the things that fissure and waltz did which was very bad for them was trying to basically accuse boye of slur swinging for one million dollars trying to raise too much noise and as a result we got involved and we broke their code so perhaps if if only they were setting up this would have gone away and advice for researchers, authorities, vigilantes so in reads there is this show us the code we will not convict the pair unless somebody shows the code and prove that they were actually cheating and that doesn't work as we said I mean I can define unbreakable code I can just generate some random string of numbers and reads players are very good at memorizing numbers so we think that basically all future codes and all future tutors will not be detectable by breaking the code and another thing is when trying to break the code try to look at the code from the recipient perspective because for example one of the things that we found out in one of the cases east west helped fissure and waltz and east removed the boards from the table and placed it on the table but that was actually sending a signal to the other end so don't ignore those things don't ignore those outliers try to look at camera angles we actually sat near a screen and try to figure out how things look from the player's perspective not only from the camera perspective and things sometimes look different so that's actually important there is now talk in reads to stop using physical cards start playing on ipad or stuff and start sending electronic signals because that will remove cheating well i guess i can tell you in this room that any electronic means is not going to stop cheating people will break into the iphone or ipad people will find some ways to broadcast information i mean that the fact that you are using cards or not using cards is not going to stop or prevent cheating what would prevent cheating is when somebody does something irregular then questioning try to figure to receive the logic and if somebody makes too many questionable moves and is world champion is probably cheating one more thing don't expect any phrases i'm still involved in credits wars about it that's why i was very precise at what we contributed and what others contributed but people are more happy to to fight over credit than to actually think what you did is good and the last thing is something very very unique we actually had two cheating pairs the gentlemen smirnoff and peacock playing against fissure and waltz on the european bridge team championship and one and a half hours into the match it appears that fissure was caught cheating peacock the catazim tells him that he's moving the world and he's signaling by moving the world and fissure more or less laughs it off and then two hours into the fissure tells peacock okay i know you are you are cheating you are cheating in a certain way you are placing the cards in a certain way to tell your partner if you are minimum or maximum for your beads and he's writing down like peacock makes some beads that says six to ten points he writes down you have 10 to 11 points and basically catazim online and of course neither smirnoff nor peacock nor fissure nor waltz actually reported each other they basically signaled okay we know and let's continue and get on with our cheating so that was actually funny okay so okay fissure actually moves the board because that's how he's cheating and he's actually telling him i know that you are cheating then 30 minutes later he makes an opening bead of two diamonds which he places near him which is six or ten points fissure is actually moving the the bead oh shit fissure is actually moving the bead so now it's a it's near his partner now it's actually so six points instead of ten and he's writing down on his note that you have you have shown ten to eleven points which is actually read the bell and smirnoff just says like okay and proceed so basically two cheaters catching each other live breaking each other code during play and of course not reporting and again credit so we use the work of boy brookland perula calling a train tira and the kids will see the videos that i presented especially with the audio and like just the stuff is by michael clark and lots of thanks to in only one and that you know sir woody fridtlander gilad of fear nicola summand and professor like lauler for helping with the stuff and generating more data on the on the case thank you