Thanks, Pat, especially for putting this on. It's hard to believe that it's been a year since the last one. I was here last year with a panel with Charles Nutter, and Tim, and John Lam, and giving, basically on an implementation panel talking, kind of doing a Q&A about things. So it's great to be back. So I guess I'm going to talk about a few things today. I'm going to talk about sort of where Rubinius is right now. I know that that's a pretty common question, so I thought I'd get that out of the way. But really what I want to focus on for the majority of the talk is not so much that, but the kind of almost the things that I've learned, kind of heading up that project and how the community around the project has really made it possible. And kind of go into that sort of from a, this is how I run my project, kind of point of view. So first and foremost, I wouldn't be here, and I wouldn't have all this time to do any of this work if it weren't for engineering. So big ups to them for basically providing me with the resources and the time and the immense vision to believe that this is something that they wanted that can be funded full time. So on to status. So let me take you back about a year to March 16, 2007. 2 PM, there was me, up on this stage wearing a soccer jersey with the presenter panel. This is one of the lovely conference videos from that talk. And a lot has happened in that year since I gave, since I was up talking about what was going on then. Then I was talking about how I didn't have a full time job doing this. It was still a hobby. And I was talking about how won't be great when we can run any code at that point. It was very much still an experiment. And I'm really happy to say that we've really come a long way since then. Well, let's cover what else has happened in the Ruby implementation sphere that encompassed that stage that morning or that afternoon. So we have JRuby 1.1 or so or whatever they're on at the moment. Ruby CLR has largely been replaced with Iron Ruby. The status of cardinal remains largely unknown. And so obviously, we want to move on to what has happened in the Rubinius world since then. Well, I'm super happy to say that we've had millions of small victories in the last year. There are really too many to account for in a presentation to talk about how, oh, we managed to get this running or we did this. We got these tests written. The project itself has managed to shore itself up an enormous degree through very small contributions and has really managed to kind of elevate the completeness and the overall strategy of the project. And of course, as with any large project, we've had thousands of missteps in the last year. Most of those have probably been my mistakes or just making the wrong assumption and backing out. But that's part of having a big project. We've had hundreds of questions about how the thing is supposed to work, re-evaluating, having lots of debates about what we're actually trying to accomplish, have been to tens or dozens of conferences, whatever. But really, it's all been in pursuant of one goal. And that is really to build a VM where writing in Ruby is not an afterthought, where every day you come in to build this VM and you're writing in Ruby. And that's OK with me. So in that last year, what have we output? Well, really one of the biggest things that's come out of it is wisdom. And I don't mean like, I've learned a lot of things. But the whole community, the whole project, the whole community has learned an enormous amount about how to not just how to build this project, but also how to function as a very large, very transparent community. We have almost never hear about the Rubinius Core team. Because I kind of frown on that sort of phraseology. Because it puts the people that work on Rubinius in two separate camps. It puts them in the I have the keys, I have the power kind of camp, and the sort of the proletariat camp. And that has a, I'm going to kind of go on to show you later in the presentation, how that has really, I think that that is a really dangerous model and mindset to get into with the project as you go on. And this is one of the hundreds of questions is, is it done? Mike asked me that this morning. Like, how's it going? Are we going to get a big announcement here? And we all know this, but I'm learning it again for the first time that time estimates are hard. But when you're highly transparent, everybody knows why things are going slow. And I get originally, probably I said it on this stage last year that, oh, well, I want to have a 1.0 by RubyConf in November. We were not even half done. I mean, if I know what half done is at this point by RubyConf. And mainly because one of the things that we've learned over time is what the scope of the project is. We kind of went into this thinking, well, we'll figure it out as we go along. And we had a feeling for what it might be. But we really have learned over time what that scope is. And we've gotten better about time estimates. So it's still hard. One of the biggest things to come out of the project so far is this giant spec suite. And that's really, in the last year, been a big mover and shaker in all of the whole Ruby community in terms of implementers. You see people like JRuby really runs the test suite, or the spec suite that we've been working on, to make sure that JRuby is working fine. And we're planning on actually spinning it off as a separate project so that JRuby has, and other implementers, have better access to it. At the RubyConf presentation in November, John Lam got up to talk about Iron Ruby and basically showed it running the Rubinius specs, just right up there. We didn't care. We didn't hear about it. It just happened. It showed up in a presentation, that kind of thing. And being highly transparent and really sort of giving this spec suite away has actually proved pretty great. It has really given a lot of people things to do and an entryway into the project. Another thing that we really gained is this idea of continual progress. We make mistakes. We have to go back on what was going on and rewind, start over on certain things. But for all in all, the project has continued to move on in an upward trajectory as the long-term trend, which is really the whole key of it. And then we have the goals that people in this room can actually really kind of, if you've never seen Rubinius and you can't really look at the code, goals that you can sink your teeth into, like it runs IRB and it runs RubyGems. Those are two huge goals that we have managed to accomplish in the last year that are a presentation by John Miami was saying that it's hard giving a presentation about a Ruby VM because all of the things, all of the successes that you have look very paltry from the external world. When I say that, oh, it runs IRB, that took a year and a half worth of work just to run IRB. And but to the external world, that's sort of like, OK, well, why is that a big deal? So it's kind of interesting how to take my word for it. How about that? And then RubyGems is a fairly new accomplishment. Ingeniard hired Eric Hodel just this, I guess, the end of last year. And he is one of the main RubyGems contributors now and has actually, he and I worked a lot on getting RubyGems integrated nicely into Rubinius. So you can do stuff like Rubinius, Gem, install, that kind of thing. And I'm really happy to say that that is working very well right now. It's still got bugs here and there, but we've been able to do things like install Gems and run stuff directly to Gems and all that kind of stuff. I'm going, I don't have a ton of slides for this talk. And looking at my time, what I wanted to say real fast was if you have questions, typically what I like to do is go ahead and just raise your hand and we'll kind of go off on a tangent right there. And I'll kind of have to rein it back in if I feel like the tangent is going too long. But hopefully, I'm not boring you up here. So moving a little bit on to community next. I've learned that the one thing that I've really learned in terms of Ruby, the Ruby community and sort of the Ruby ecosystem is that this, which is now considered, I think it's officially an adage that the Ruby community people are very nice. I think it's gone around long enough. The student will be myth, actually. So that's the next level after adage, I think. So we're currently at adage. So it's really proved to be very true that the community is fairly small but very excitable, which for the most part in terms of a community is a very good thing. And if you look at other communities, you'll see that either because of market forces or internal community interactions that they become very cynical about new ideas and new things. And one of the greatest things about the Ruby community in general is that it's been very open and very embracing of new ideas. A good example is that in the last year, we've seen this huge upturn in Ruby projects using things like Git or Mercurial. That's those sorts of things. Things that in other communities tend to take a long time for the vast majority of people to take off of them. They're very slow to adopt in the wider community sense. But in the Ruby community, you see that like, it happens at a very quick pace and people are very eager to jump on and eager to move forward. I think that in terms of the community is something that if you have a project and you're looking to get people in, you really wanna capitalize on. You really wanna use the excitement of the community to your benefit. And sort of to that end, it's important to realize that just like you have a project and I have a big project for Rubinius, but I am not, I am not Rubinius anymore. I mean, a few years ago, I could probably make that claim, but it's actually not, I mean, it's not mine anymore. At the moment, I'm steering the ship, but it is moving under its own power because of the community that has sort of risen up around it and risen up in terms of the whole Ruby community in general. Even just not in terms of people who are actually contributing code every day and really watching the project closely, but in terms of this idea that the Ruby community is really embracing of these new ideas. And so I can float this out pretty easily and get people excited and really they, the whole community tends to kind of embrace that and is able to run with it. And so sort of, that was a really big goal in the last year that I realized that even if that I wasn't really the momentum for the project anymore, that the momentum has actually taken out of my hands and now I'm actually riding with the wave of the project now. And that's really because the whole Ruby community is now the Rubinius community. They for the most part merged for the most part. They have merged. They were fairly segmented. There was people who were working on Rubinius that were in the Ruby community. But now I get enough interaction from sort of people who have just a really passing knowledge of the project. They say, hey look, it looks great. I can't wait until it gets to 1.0. And that really tells me that it's grown now to this point where now it's the same community. They're no longer separate. They're no longer one is the super set of the other. It's finally kind of encompassed both. And that's a huge thing. Which kind of moved me on to my big next section here of talking about community. That famous Apple guy said that if I have seen further it is by standing on the shoulders of giants. And I wanna revise that a little bit today. Now I'm gonna make up a new one here. If we have seen further, it's because we built an awesome tree house. Sort of like this. And it's really because we have made that decision as a community to push, to get together and to build this thing up from sticks and mud and lacquer, it looks like. But the big thing is that you really can't buy a community. I mean, if you have a big project out there and you're trying to rally people behind it, it doesn't, it's very hard to do that in an inorganic way. It's hard, you can't really go out and other, it's happened in the past where companies have tried to grow these communities to start a website, they get some street people, handing out swag, that kind of thing. And they try and really buy a community through kind of not really excitement. And you find that almost universally those communities fail. So what I'm gonna talk about next is sort of a set of bullet points that I've put together that talk about what I feel are the real big points that have made this Rebinius community really quite amazing. And that first one is that these are sort of from a project management's point of view. That first one is that there really is no task is too small, you know. Early on when we were writing specs and we were starting to take contributions from people, which is about a year ago. There was this feeling that, oh well, yeah, we'll get contributions from a few people, but there'll be one line specs here and there.  Because we have seen an enormous amount of people who they write the one line spec that they come in, they realize they're like, oh well, it's, you know, Array has got such and such behavior that is really trivial, really, I'll just write a one line spec to test that edge case, no problem, they write that one. And we've put the barrier so low in terms of contribution that we'll take, we take one line documentation changes as valid patches to get a commit right, to get commit rights to the project. So there's no bottom in terms of the size of that task that the person can actually perform. And the reason for that is that will never be, if you make, if you open that door for them, even though it's this tiny little mouse door, if it's very easy for them, they'll go through and they'll do something else. They'll go to the next level, they'll say, oh, okay, well that was so simple, I did it in an afternoon, and it was this one line, I've got a few more hours here and there, let's see, oh, okay, the next one, now I've got a 10 line one.  And the ramp up for that is just, it's amazing to see what opening that door for anybody who wants to walk through it is able to do. And really that's because, like I was saying, every contributor has worth. There is, even if you, how do I put this? There are personalities that will grade on you. And there are personalities of people in the community that maybe you're not so fond of. But what you'll find is that if you go ahead and you kind of put your personal feelings aside and you say like, even the annoying 15 year old who wants to come in and he's super excited and he produces a lot of noise and a one line patch, that's still a contribution. Because it has grown the community, even if it's not in terms of that, even if that one line patch is all they do and they disappear, they might tell their friends or they might create some buzz. You have no idea where those, that next sort of star contributor will come from. And it's, you need to foster every, everybody who comes to that door is someone that you should talk to. And that same way I've kind of gone over this, that every contribution has its worth. One line documentation changes are great and lots of, we have had a significant number of contributions, some of them mine, that have really proved what not to do. We have in that same vein a policy for the most part of asking for forgiveness rather than permission.  You know, that's, honestly, that's not that big a deal. We can always go back and we can always revert that commit and say like, hey, you know, that was a great try. What were you trying to accomplish? And really, you can go forward from there and proving that, okay, well, we tried that approach. It didn't work. What can, how can we move forward from here? And so a big part of that is this idea that debate is healthy and every project will have friction no matter what. This idea that you will, you're able to get a very small close knit team of people who all have the same mindset and who all are striving towards the same goal and that those are the people who you want to be the, to say the quote unquote core contributors is pretty much a blatant fallacy because even amongst those core people, there will be dissent. And if the, sort of as the project leader, if you don't foster that debate as a healthy part of the development process, you will easily begin to grow animosity amongst either the core team or the external contributors. But if you say, you know what, I can be wrong here and you can accept that an open discussion about what should be done and that nothing is sacred, that there's no sacred cows in the room, that you can really put everything on the table and have a really honest discussion, that that does amazing things for the sort of the lifeline of the project and the quality of conversation that occurs in the project because as a project manager, there's this big tendency to really see, especially in the beginning, where in the same way that say you have a startup company and you are working towards, you are working towards some goal with your project, you tend to, you tend to start to mix the two, you and the project. And it's very easy to do and I'm more than guilty of doing this as well. But you have to, at the end of the day, when you sit down and you're not working on the project anymore, you have to remember that you are not your project. No matter what you may feel about the project, no matter how excited you are about it, no matter how good it makes you feel, you are not it. It has to have a life of its own that is actually separate from you in order to grow properly. And really, I'm talking to you because it's very easy to fall into this ego trap that you start a project and it starts to take off and you see like, wow, this is great. This is all me, right? And maybe you're not thinking that. Maybe you're not that.  But it's so easy to fall into that ego trap of when someone breaks the project, you go, they're hurting me. They stab me in the chest, you know? Or when bad press comes out about your project or when someone dogs it, you tend to see it as a personal attack rather than an opening for how to move the project forward in a different direction. Because if you separate your own personal ego from the project, amazing things tend to happen because when someone attacks you, you're able to say, oh, okay, well that's a good point. Why do you think that? And how should we change it going forward? If the person has, I mean, there's plenty of people out there who have good points which brings me to my next bullet point here which is that you're not always right. And the corollary here is that it's actually very important to be wrong in public. When you have that healthy debate to a project, you, and you've distanced yourself a little bit ego-wise from the project and you've started to let that project live, it's very important, and this sometimes doesn't come naturally, sometimes it does, but it's important to realize that your decisions up front are, you may think that they're done with the best intentions and that you have the best kernel of information to make a change or to make an architectural decision, but by and large, you're not gonna always be right. And I've been wrong tons of times. But when you're wrong, don't go behind closed doors and talk to the person who called you out and don't tell them, oh, you know what, I don't kind of weasel your way out of it. Don't say, oh, well, I've seen your point of view and or I tried that before and I guess I missed something. Don't turn it on you, don't make it all about you still, and don't do it behind closed doors. Do it out in the open, do it in on the mailing list, do it on the IRC channel, do it wherever you know that the other people live and be ready to admit to say like, you know what, I fucked up and we're gonna go in this direction and then this is my idea for why I think that we should move forward doing this other thing. Because if you're ready to go out and you're ready to put it all out there to say that you as someone, typically as a project leader, people look up to that you have made a mistake and that you wanna move forward, that does an amazing thing for fostering the rest of the community for saying like, oh, okay, great. So he is open to new things and that the next time someone comes around and they say, you have a healthy debate and let's say the idea is totally stupid, right? Because that happens a lot. You'll start into a debate with someone about an idea where they're like, hey, how about this, this, and this and you realize fairly early in this debate that really there's no way getting around it. This is a ridiculous idea. But that if you have fostered this idea that you can be wrong when you shoot them down, they're like, oh, okay, well, thanks for listening. I guess maybe we'll talk about it later. You go, okay, great, we'll talk about it later. Because you've told, you've showed them that you're taking their idea to heart and that you're actually giving it some credence. And that is a really, you know, I guess I can't stress it enough that that does just, it puts this sense in the mind of the other people in the community that you're there listening to them and that as you make changes and that as you make decisions for your project, it's not you making them by yourself. And so as that community grows, and I talked about this earlier about this idea of core contributors, the idea that there are keys to the castle, I'm gonna put forth as a myth. If you have an open source project, there's no such thing as keys to your open source project because you by default have left the front door open. So yeah, there's keys, but who's using the keys at this point? They're all walking through that open front door. And so there's really no reason to bolt down and to really carve out this tiny niche of people that you feel or that you have sort of elected to be your main contributors. Because what's gonna happen is if you make it very difficult to be in that inner echelon, there's gonna be external work that's done that you're not able to get access to. Because as an open source project, you're already giving it away and people externally are gonna be, they're gonna be doing work on it whether or not you like it or not. And if you're not willing to show people that you're open to taking contributions from anybody and anywhere, it tends to foster us and them mentality over time. That can prove to be the death of many projects. And there's numerous projects that go through their lifetime as a tiny commit team of five people where they're very selective about what they decide to take in and they're very closed. And over time, that community withers. And it could be just because those five people decide, you know what I got better things to do. They get jobs, they get whatever. Because typically people aren't working on open source as their only source of revenue. And if you've already closed it down too much, then the minute those five people go away and the minute you've established it very much as this closed thing, it becomes the project very, very quickly begins to rot. But if you have left the door open and you have invited as many people in as you can, and you've said everybody has value and every contribution has value, then over time as those people tend to check out because they have normal everyday lives. They're not sitting at their computer 24 seven. As they leave, you've left it open so wide that you get more people come in.   And that's fine because they still contributed in some way.  They are the nasty external virus-ridden world to the community's immune system that they have come and they have said, you know, I think you guys are really doing a ridiculous thing. And that fosters debate inside the community to say like, okay, what do we think about this? Do we really think we're doing something ridiculous? How should we go forward? Should we make a response? And that the ideas and the philosophies of the project tend to get built up over time better if you're more open to those sorts of things. So the next point is to foster experimentation. With a large group of contributors, this is fairly easy. This happens without you asking for it because people have a lot of different interests as you get a large group. But by and large, you should very rarely say, no, don't do that now. It now is not the time or I don't think we're prepared for X, Y, and Z. You can certainly say, especially if you have any kind of control over the person's time, you can say, well, that's a good idea, but your time is better spent doing something else.  You know, we've had one guy in particular who showed up and said, I'm really interested in LLVM, which is a kind of new sort of like a, it's a new very low level VM that generates machine code directly. That's actually not important for this example, but we came in and he said, I'm really interested in LLVM and Rubinius seems like fun. It seems like you guys are having a lot of fun in this community. I wanna do something with Rubinius and LLVM. Why not say sure? Why not say, have a good time, knock yourself out, and provide that person with as many pointers as you can.  Those are the ones that a lot of times, the person doesn't know what they're getting into. You know, maybe a week later they show up and say like, wow, that shit's hard. I don't know what I was thinking, you know? That's okay, you know?  So if I approach that, I'm gonna approach that with that knowledge now. But you also get this idea of say, we've got people who, you know, another kind of good example of this is Rubinius, the kernel of Rubinius being all written in Ruby is actually really great for experimentation because we get people who show up and say like, you know what, I really think that the way that string operates at the very lowest level is not real great. And I wanna write an entirely new string class. And you go, yeah, sure, absolutely, you know, go for it. What point, what can we provide you? What information do you need to begin? And really to, you know, to let every form, every expression of a person's creativity be a valid expression. Never, the idea that you would say, that someone would show up and say, oh, well, you know, like, I think Rubinius would be really awesome if it had this feature. And that the last thing, the worst thing that you can do to that person is to say, well, well, no one wants that feature. That's a ridiculous idea. That's the last thing, and the worst thing that you can say to that person because there are so many things that come out of experimentation other than the intended result. The idea that this person could start to experiment, they could get halfway through and they could get to the shit's hard stage, but they could realize along the way, wow, look at all these other things. Look at this, look at this, look at this, look at this. I'd love to work on this, and maybe those other points of experimentation have very real end results. And so you get an enormous amount of work from people by just letting them kind of play, letting their mind wander. It's almost like free association, but with code, right? You let them go down these really long, really long paths, and they may never get there, but along the way they could discover amazing things that they wanna work on, or amazing things that the project can benefit from. And by fostering that experimentation, it leads to sort of a second, the sort of the next stage of this, well, they're related, which is that excitement really is contagious. And that you, as a project leader, you have to lead by example in being excited about things. And that what I mean here is that if you get people excited about your project, it's almost impossible to use a cliche to contain someone's excitement. It's almost impossible to contain that excitement about a project for that person, because invariably they tell their friends.  Look at this code that I wrote. And that builds out your community. And then in that same way, by fostering experimentation, you foster this idea of being creative. And I am a terrible math student. I had to take linear algebra like three times in college just to get a C. But, and so I kind of consider myself that more, that kind of more creative, like abstract minded programmer. Not so concerned with the algorithms and the math of what goes on, but in the overall construction and the overall beauty of the thing that gets accomplished. And it's very hard to express and to sort of, to put a finger on what is creativity and why people get excited about it. But you can easily see that, if you kind of look at the, like take painters today. Experimentation in painting is the number one thing that is encouraged. And that this idea that people should learn from the masters and then begin to experiment with what they've learned and go off and do their own thing, fosters this excitement in their art. And they become very creative and then they begin to give back to their internal communities. But they get, they're excited because they have all these avenues and all these openings to continue forward. And so if you can make everything, if you can make your project excitable, if you can make your, if you can, if you can make it fun. I mean, if you can never say to someone, no, if you can say to someone, maybe not now, occasionally, or that didn't work, you know, we try again, that's okay. But you're really able to, if you can get that ball rolling in terms of excitement, then things will really, really start to spiral. So that is actually where my slides end. So in terms of time, I'm, I guess we could do a number of different things here. I can open it up for questions if there are, it's pretty early for questions. So do we have questions? Otherwise we can kind of go from there. Okay. All right. We, we can talk about, we can do questions for hours. So, okay. So let's do it right here. I was just wondering, I've never heard of Rubinius before right now. I was wondering if you could just kind of give a brief overview of what it is. Sure. So, and he asked if I could give a real brief overview of what Rubinius is. So the idea behind Rubinius was to build a new Ruby virtual machine where as much of it was written in Ruby as possible. And for the most part, that means that kernel of code that is available when the whole machine, when the VM first boots up. So what I mean by that is so say array and all the methods on array and string, all the methods on string, all those sorts of things. In Rubinius, all of those are all written directly in Ruby. And the idea is to really drive forward this idea that a Ruby VM doesn't need to be, it doesn't need to be written in some of the language. When we see like, the current Ruby is written 100% in C. There's actually no Ruby, ironically, there's no Ruby in Ruby. It's all written in C. In the case of JRuby, it's all written in Java. In the case of Iron Ruby, it's all written in C sharp. So this idea that those are great, but I don't really like writing in those languages. I really like writing in Ruby. So how can we foster that idea of writing as much in Ruby as you can? So that's the goal is to build that VM. And in the course of building that VM, to build it with unique features and performance and all that kind of things that other VMs don't have. So that's the brief overview. So, yes. And I agree with that. I'm wondering how you handle the situation where somebody does have an idea that doesn't really fit with the project. And if you were to take a project in the core direction and that person's going to take it all away, you see what I'm saying? It's obvious when they can have it through and they're like, oh, I have a chain now why this doesn't work with the project because it doesn't fit in the direction. But, you know, somebody turns it into a convenient situation and decided to want to have a sort of strange method that didn't do any simple plus. And the rest of the community wouldn't consider that as a good direction. How do you handle it? Well, I think the first thing is, did you say a poor direction? Did you use the word poor? I did, but I didn't. So the first thing is to not put a value judgment on it, right? Because it can certainly not be the direction that the you or the community you have has decided is a valid one. There can certainly still be a direction.  Because if you went down this way and you thought, you know, this is really, this is the way, this is the thing right here. And then after a while you're like, you know what?  And you went, go down some of the direction. You've basically done what this other person is doing, just now, it's now extra body instead of intra body, right? So it's, in terms of what you would do, okay, so say that adding a new method to symbol is actually fairly benign. Let's take a bigger one. Like let's say that someone wants to take Ruby and they want to add innumerable features to the VM and they actually want to customize the whole kernel to be something completely different. To be no longer in any way, shape, or form, like say that all the classes are named different, all the methods are named different, it has completely different functionality. So the idea of what do you do when that happens? Well, I would say you slap the guy on the back and you say, have a good time. And there's really, what? What if I'm in the end and you don't put that in the core? Well, that's okay, I mean if you, the, all of these points that I've given can't be taken individually. They have to be taken as a larger strategy for how to manage a project, right? So that if you've got the other things, if you've got open debate, then it's easy, or it's, well, it's easier to over time to say to this person who's, say, forked your project and has taken it in a completely different direction to say like, you know what, we could talk about this all day, but it seems like you want to take it in this direction and I want to take it in this direction. And there are certain points where we meet up, right? So how can we move forward? And if, you know, maybe you wish him luck in his new project. You say, maybe you suggest a new name for the project for him.  And you tell people, hey, there's this new project out there. Someone forked my thing and they're working on this. If you're interested, go check it out. If you quell those things, if you stamp them out as soon as they show up, then what will happen is that those people will say, wow, they're not very open to this debate idea, right? But if you, I mean, there are so many different ways it can go in the end. If you let this person run with their idea, right? I think we're all naturally, humans are naturally risk averse. And this idea that someone has taken your thing and has taken it in a different direction than maybe you see is a big risk. But the number of question marks that are down that road are really too, I mean, there's just, you can't hypothesize what could happen. Any number of, like, they could go down that road and it could turn into nothing. They could go down that road and it could actually turn into something much bigger than your own project. But if you, at the beginning, if you foster that experimentation and if you tell that person, absolutely, let's figure out how to contribute. If you wanna do this thing, that doesn't really meet up with us, but let's figure out how we can remain a community in the larger sense. Then if their thing takes off and maybe yours doesn't, so what? You are still one community then.  He was right. You maybe it takes you a year to realize that, like, wow, you know, Nokia bought his company. Obviously he did something right, you know? So, you know, if you have been friendly to him and if he is your confidant and is in your community, then that year later, you're gonna be really happy that you fostered that experimentation.   So he has provided some body of work that was previously unavailable. And is it your place actually to even tell him that no, you shouldn't be doing that?  And you certainly have control over his view of what his work might be. But there's really no, there's no downside to letting him run as fast and as far with a project as he wants to. You, I mean, this idea that the community will fork and splinter over time because there are forks, I think, you know, and that's certainly happened with open source projects over time that, you know, they, the, but if you go back and you look at the seed of that splintering in the community, you'll find, I think by and large, that where the fork took place, there was a personal attack or someone took something very personal on them and not as something larger to the project and that now they have, it's not a technical problem anymore. They have a personal problem with some other person and that that becomes the fracturing of the community and the fracturing overall. Whereas, you know, it's just software, man.  It's just, there's no reason that those things can't continue to exist and grow organically and that you can't, on a human, personal level, let that occur. You know, who knows what might come out of it. So I guess that's my long-winded answer. Sure. Tom. I was gonna say that without the student version control, I think the risk there to split in the community is a much more significant issue.  Every other form. So as they continue to develop and, you know, do some crazy new versions of Vinias that isn't even Ruby, they can still find bugs in the VM and you can take those patches around it, right? Sure. You know, they don't have to be competitive at all. Yeah, I don't know. Okay, I was gonna be more like a super, than the final set or, if they're going forward, it's fine. But if they're also doing, you know, pretty bad, if they're the main cause or anything like, keep forward. But if it is, we're gonna get it. So that's right. It's, I mean, the most beautiful example of that, I think is, you know, the Antwerp and the Branch and Lens.    Come on.  Your presentation is horrible. Yeah. Thank you. Thank you, Josh. So, I guess the, the, the idea, this is a good point. This is certainly, thank you for bringing it up because I didn't really talk about it much because I was talking about the more technical. So, whenever you've worked in the company, there's always the cheap technical fix, right? There's always that like ridiculous way of solving some personal problem, right? And this tends to be the other way around, which is that the idea that like your, your project might fork and splinter over time, tends to be, tends to be viewed through the lens of, this is gonna be very hard technically. And thankfully, software has now in software we've developed like new techniques for dealing with massively distributed, massively forked architectures, right? But even in terms of subversion, you could easily do this.  I think you're doing an amazing thing. I love this fork. I want you, can you move your thing into my subversion repository? I wanna host it too. And that saying like, you know, why do we have to be separate? You wanna do your own thing? We share a lot. Why not, why not, I'll give you a hosting, you know? I'll, I'll, I'll pony up and I'll let you do it inside. And then over time, you know, maybe you realize, you know, maybe you get to that point where they're like, okay, well, I need this functionality that's tied up in here. And so then you begin to work as a community to refactor over time. And who knows what may come out of that? You may, you may turn into the project may begin to grow these nice modular architectural organic pieces where now, because you have the opportunity to splinter a little bit, to fork a little bit, now you've been able to focus and say like, oh, okay, well, great. You know, we actually have this really awesome little sub thing here that we could both share. Let's get that done. Okay, sweet, you know, we've got that. And by, you know, the, this idea that you, you know, that it's not really a technical problem from, from that's, at least from my point of view. So, yes. How do you decide and handle what goes into the main code and contribute to this? How do I, or how do, how do, how do you? Sure. You, you have to have a debate about it. You have to, I mean, your project has to be a republic, right? And that, you know, there are certainly, there are ideas for laws that never get passed and all kinds of things that happen all the time. That, but by having a debate about whether or not those things are, are useful for this stated goal and this stated direction, you're able to judge as in a debate setting whether or not those contributions fall within that realm.  And so they, in a debate setting, have that. They're very closed, obviously, you know, it's a very hard process to become a Supreme Court judge. Very easy to become a Rubinius Committer, right?  No, no, Supreme Court humor, okay. So I guess you have to just take it peace at a time, you know, some of them come in, someone has a contribution and you look at it and you say like, you know, this is good, but I'm not really sure, like where are you taking this?  And maybe at the first, maybe at the outset, it feels like it doesn't fit in with the goal. You talk to the guy, you have a little just one-on-one discussion or you have a round table discussion with the person and you realize actually no, it does fit in with the goal and you just, maybe you didn't see it at first, right? Or maybe you decide that no, it really does not fit in with the goal and you basically tell the person, you know, this is a really, I see where you're going with this and it's an interesting idea, but it just isn't, it doesn't fall underneath the realm of where we are right now. That person can do any number of things. They can decide to go off and do their own thing to take those patches that they're doing and kind of take them in a different direction. But you should still keep that person in your community. You should say like, you know, there's no reason that they can't have a separate goal and be going on a separate direction and still remain part of the larger, the larger pie. Yes, behind you. You give us some great insights on what it's like to run an open source project and stuff and experiences. I'm wondering if you could also focus a little bit on, you know, dealing with the larger Ruby community and how public outward facing stuff. You know, there's a lot of great about how Ruby this will be in 1.9 and Ruby 2.0 or something. How's that going to be? Okay, so the question was how does the, how is the larger like sort of Ruby community seen Ruby isn't by and large? Well, I guess let me, I'll preface that this with a question that I get fairly often, which is, oh okay, is Ruby is going to be Ruby 2.0 or is it going to be the next Ruby that we're all using day to day? And my answer is unequivocally, that's not up to me. I mean, I'm working on this project and I want to make it the best project that I think I can make it. But by and large, this is a, a very, I'm sure there is another government term here, but it's a very loose knit group of people who for the most part, you know, decide to use technology based on their own feelings, right?  It's not going to be in Rubinius swag to try and get people to use it, you know? It's not, it's not a car. It's not a, you know, it's not a toaster, right? It's not an air conditioner. It's a piece of thing, it's a piece of code that I'm giving away for free anyway, right? So the point of, my point of view is that the way that I can make it the best and the way that I can fulfill, I mean, certainly I have a personal goal to see that Rubinius succeeds and that people really love it and they use it day to day and it really grows and it becomes this awesome thing as a personal goal. Sure, you know, we all have personal goals that don't necessarily, that aren't the project goals, right? And so from my point of view, the better I can make the community, the Rubinius community, the better I can make this idea that people will want to use Rubinius and that, like I said earlier, that this idea that the Rubinius community has now basically expanded to encompass all Ruby programmers, even if they are hearing about Rubinius for the first time, it's grown now to this point where it's not really a separate idea anymore and that's really the, that's the way that I've always wanted it to be, that now it becomes this tool, this place, that people are thinking about it and they decide, hey, you know, when we get to 1.0, people are like, okay, great, I'll download it, I'll check it out, you know, I have to, it being free, I almost have to give people reasons not to want to use it, right?  In terms of how it's seen in a larger community sense, like how the larger Ruby community looks in on the work that's being done on Rubinius, it's, for the most part, one of anticipation, the, what I get a lot is people trying to, like, kind of stir things up a little bit, like, oh, what do you, what do you think about JRuby? Do you think it sucks, you know? Or, yeah, well, 1.0 is a piece of shit, you know, like, I can't wait for Rubinius, you know, like, they make those, you know, those larger state, whether or not, I'm not gonna qualify those in any way, shape, or form, but the point for the Ruby community, that they, I suggest, whenever they bring that up, that they realize is that the competition is the healthiest thing that the larger community can have right now. By having multiple VMs, by having multiple projects, by moving things forward in multiple directions, in the same way that, you know, you may fork a project and move it forward, and it will move forward in ways you never envisioned, by having all of those different pieces, all moving simultaneously, and there being enough Ruby programmers for everybody. I mean, again, even if we all succeed, if JRuby is awesome, and Iron Ruby is awesome, and Rubinius is awesome, and YARV is awesome, there's really no, as long as we have some agreed upon conventions, is that's what Rubinius is trying to do with the spec suite. There's really no reason we can't all play in the same pool, you know, there's no limited resource of Ruby programmers, and by having all of those options, that only grows the number of Ruby programmers more, because now they can see JRuby is bringing Java people in, Iron Ruby is bringing C-sharp people in. Rubinius has brought some small talk people in, and that it starts to glom on, and by having all of these healthy projects within the community, that makes the whole Ruby community healthier, so, let's see. Think, like, what, two more questions, time-wise? Okay, Tim. I'm gonna probably be, I totally buy everything you're saying, but I do kind of have a deficit, I have a quick question. I do think the decentralized approach is the bizarre approach and sort of the singularity of open source as well, but, and contributing to Merb, for example, much easier. I still got my patches into Merb back channel-wise. I'm curious about what you're saying, obviously Linux is a huge project that you can get, but maybe there's a little poll and that is that you have such a small, self-selected, like, clean edge group, that people at a small Ruby comp haven't heard of a project, and does that maybe give a different view on how this grows. I think you actually have thoughts on how this does grow, but I kind of want to hear your thoughts on, oh, hey, this doesn't scale when your project gets so big that you're provided with health vampires, and you can't bump a package. Sure.  That would be awesome. If every day I had 500 people sending in patches because they were like, oh, this is awesome, and they send in their most ridiculous patches, that would be the coolest thing ever. Because this is not, I mean, absolutely, the current, technical, a little bit technical, but mostly human architecture that we have for the project where, you know, we have, for the most part, a certain number of people, typically, that review those patches, and they review initial patches for people and submit them, and we get commit bits, and the amount of, like, daily eyes and review that goes on when, you know, when someone, like, when a new contributor commits code, there is about, I don't know, there's probably 10 or so people that read the commit logs. And when they see, like, especially, and I do this, whenever I see a commit by a brand new person, I read their, I read the diff for their first few commits to check it out, to see what they were up to, right? That doesn't scale if we've got 500 people and they're doing, you know, 500 people a day and they all do a commit that day. I'm just, I mean, that's sort of game over in terms of my free time. So we have to grow the architecture at that point, but what that is, I can't say. I mean, that has to grow as an organic byproduct of the demand.  You forgot that, oh, well, there's all these people in France and they're eight hours off of you. And what are you gonna do now? You know, that kind of thing. So, what, you know, I guess, I mean, I'm not. You mean maybe they're growing your community so hopefully after saying it, it will continue with the scales to hope that we can select itself and moderate itself? Yeah, I guess so, I don't see why it wouldn't. I mean, I guess if you have, so if you have these, you know, 12 or whatever points I gave here, well, nine and some corollaries. I don't have a reason to think that if you have excitement, if you have open debate, if you have all these things in the project that people naturally fall, as you bring in new people, they naturally fall into different buckets. And you have people who are very meticulous. Like you have the, you know, you have the more math-oriented computer programmer, right? Who's like, no, things need to be done in this exact sequence and, you know, I've looked at this thing and it doesn't really work. Those people are really great for reviewing patches because they kick things out. They say like, you know what, this is a great idea, but it's, you know, we need to open it up for debate because I'm not sure that it works right now. And that over time, you get more of those people if you foster them coming into your community. So, I mean, this is a great example of a problem I hope I have, right? I'd love to have this problem of like, there's so many contributors, what am I gonna do?  You know, that's like a, that's a problem you love to have because so few people get to have that problem of saying like, how do we do it?  So basically that's a, you've, you introduced a very normal tiered approach for organization that we've had since like 1780, right? I mean, this is none of this in terms of organizational, especially human-wise is new. Yeah, 700 BC, whatever it may be. So, yeah, so that's my answer. So I am going to give it over to Ezra. Thank you, everybody.


Okay, so we're going to start. So the first speaker, I'm very excited because basically every paper that I read has some mentions to it. So it's pretty good. So we have Miltus for Microsoft Research, a round of applause for him, please. Thanks, Rene. Thanks for waking up so early on a Sunday. So my name is Miltus. I'm a researcher in Microsoft Research in Cambridge, in the UK, Cambridge. And overall, I've been working in research and machine learning and software engineering program language research around this area for the last maybe seven years. So what I wanted to discuss today is more, give you a brief overview of some of the research things that are coming up. There are not necessarily things that will be applicable, things that you will go home and use them right away. But I think it's kind of give us an overview of where we see things going on. And now, of course, the first question when it comes to mind is why can we even use machine learning for source code? What is the thing that allows us to understand, to use machine learning to get aspects of source code? And the answer here is that we have this property that we call as research community, some of us in research community call it a bimodality. So it's the notion that, yes, of course, you write your code to tell to your computer what to do to your GPU, to your CPU, what is the exact instructions. But within the code, because you write it for other people to read, to understand, to extend, to maintain, to debug, you have all these kinds of hints that you give from better variable names so that people can understand what a variable is trying to do for good method names like function names, all these kind of things. So essentially, source code has these two forms of audiences, the machine, of course, and us humans. And because of those things, because it's so costly for us humans to read, to understand the code, we add a lot of, let's say, human level information. So, yeah, it's, I think I muted, yeah, yeah. So, a lot of human information. This is, you can imagine, patterns both in the way we think, the way we write code, these are patterns that we add in our source code. And because we have patterns, we can do machine learning. So maybe the poster kind of story of machine learning for source code is code of the completion. Both Eclipse Visual Studio and Visual Studio Code. Now, when you type code, when you do text dot something, path dot something, instead of just offering you a list of suggestions that are valid for this location, they will learn about your context or about the context that you are currently typing in. And they will say that, well, given the current context of the code, let's say here, you just replaced something with a path, now you are inside an if. Well, probably you're trying to see whether the path starts with something or ends with something. So, essentially they go, we go on GitHub, we scrape all of GitHub, let's say, and we find what are the common patterns of how people write code. And you can get this in a form of auto-completion where instead of getting this long list of all possible options of what are we going to do here, you get them sorted by some probability, probability based on how people have used code before. So this is currently deployed in many places and you probably can use it already. But there have been other things that have been going on in the last maybe five, six years. So one thing I wanted to show here is this idea of predicting type. So JavaScript is very common nowadays, as you probably know. And there are a few pieces of work that essentially what they try to do is they take an untyped snippet of code and they try to predict, let's say, the most probable types for your code. And you can imagine you have a for loop, maybe that's too small to see in the back, and you use for i equals zero. Well, you probably know that i is an integer. So you get these patterns that you can learn from data and you can start predicting types so that you can start migrating, let's say, from JavaScript to TypeScript to Flow or to other typed equivalents. And this is an interesting part because it means that machine learning is starting to understand the nuances of how we use the code and what things mean about this. And also adding types is an interesting paper that came out by some colleagues from UCL and Microsoft Research in the US where they found essentially that if you take a JavaScript program and you manually add type annotations, about 15% of the bugs that were eventually caught in somehow would have become if you had type annotations to start with. So you see that we're getting this idea that machine learning kind is moving us, moving the needle in cases where we have some form of ambiguity. We don't necessarily know what the code is or what the functions are doing in JavaScript, but we learn those kinds of hints statistically about their types, which we can use later to find bugs. We also had recently some other work again, mostly researchy things, not something that you can apply necessarily in your everyday life, where as humans, we write code, but we often use a lot of our primitive type. So you can have a string which represents a password and you define it of type string, password, or adjacent string that you still define as another string or something else. So now what this means is that we start getting these things that we have a latent notion. We don't write it down explicitly. It's hidden behind our mind that we shouldn't probably assign to a variable that is called password and it's a string adjacent thing. That would sound odd. So one thing we looked at is, well, how can we essentially use the concepts, the names of the variables, the names of the methods, the structure of the program to go towards splitting those things and eventually catching bugs? We're not there yet, but I think this is kind of, again, it points out that these natural language information, the names of the variables, for example, is very useful for doing program analysis that we care about and understanding the code in a way that we can help the developers in a semi-automatic way. There's also this interesting tool from Google. As far as I understand, they use it internally. And what they're doing is that they want to say that we want to detect argument swappings. You have a function that, like, take Java's or C-Sharp's substring and you say that it's, you want a substring and you say string.substring, parenthesis, and then you say first the offset and then the length, or maybe it's the other way around. So what they've tried to do is that the, when you define a function, like substring, you define it the formal parameter, the formal arguments are length and offset. And then you pass in a variable that's called off and size. So you've just swapped them. And what they're doing, and these are real Google code, as far as they say in their paper, that the official function, the declaration, says that the first argument is response, the second one is frequency, the third one is this list. Let's not bother. But then when the developer invokes this function, they don't give the name of this function, the developer used frequency first and response second. And you see the types are, again, the same. So the type system says, yeah, that seems fine. But this, there is an inconsistency between the formal and the actual parameters. And essentially this were swapped. So again, more and more kind of information from natural language, from these soft aspects of source code. And this is where, again, machine learning comes in. And you can look at this paper in this location. So overall, I think the broad perspective here is that where do we see that machine learning for source code will start appearing? There are many cases where we want to infer the latent intent of the user. That's the autocompletion case. We don't want the user to tell us, well, I'm typing here, and what I actually want to do is that. So now help me. But the intent, their intent is latent. It's hidden behind the keyboard. And we want to use a machine learning model that essentially infers this latent intent. What does the user want to do? Or what did the user want to do in case of finding bugs? There are other cases where we have ambiguous information. This is most commonly natural language, which is by definition ambiguous. And when we want to get the information in an ambiguous form and understand it in some way, this is where we want to use machine learning. And then, of course, heuristics and our code that a lot of us use because we have to. And in many cases, we want to replace those things with a machine learning component that is adaptive, that is learning as we go through. So in an academic kind of sense, I've written a survey of the research things that have been happening. SourceTotal has a great, awesome website on GitHub with its links. But this is a very broad perspective of, let's say, where research stands. So I want to spend the next maybe 15 to 20 minutes going deeper into one specific kind of work we had in Microsoft Research, which is called detecting variable misuse bugs. So the high-level idea is the following. Let's take this lever. This is from an open source database system in C sharp. And what I'm going to do, and this is the same game I'm going to play with at my machine learning algorithm, is I'm going to blank out this variable usage here. Now you can read the code, maybe not in the back, but the idea here is that you define this class variable and somehow that you get the values, then you assert it's not null, then you define the first variable, you get it again somehow, and you assert what is not null. And you continue your unit test. Now, obviously you would probably say that first is what should go in here, because that's what probably the developer intended to do. But probably because they got here, they took the first two lines, pasted over those two lines, changed a few things on the first line, and then they forgot to change the second line. So now this is something that is very odd, and this is something, a real bug that our system actually caught in source code. You would say that yes, I can use a linter to do this, and yes, of course, you can create a linter rule that says, well, I have assert not null, and well, I haven't changed class again, so I shouldn't be checking this. But the problem here is that we have so many rules that one would need to maintain if you want to do this in such a fine grained way that this would be very hard. And maybe the specific case happens in another, after another million lines of code. That's not really good. You can say you can write formal verifications, that's something that the programming language research has been doing a lot. So mathematical formulas that verify things, or you can run unit tests, but no one tests the tests. So in that sense, that's, we need machine learning to catch these odd mistakes that maybe, you know, you can see it. It's not, once you say it, it's obvious. But in many cases, when you're writing code, you're stuck into this mode and cannot see some things that once you find them, yes, I spend my whole day, obviously instead of I should have been J, but you know, these things take time. So can we use machine learning to improve this? So the idea is the following. We start by blanking out this variable and we're going to say, we're in C sharp land. You can imagine doing this in other languages too, of course. We're going to say, well, given this location from all the variables that are in scope at this location and type correct, so the type system would not complain, which one should we place here? So in this case, this is where only these two options, so our system needs to pick among those two options. And as I said earlier, this is not something that is easy to catch with a static analysis tool. So here's another example that when we dog-footed things within Microsoft, within the IntelliCode program and you see here again another similar example, you create a rectangular from some coordinates. You say X1, Y1, and then the height and width. Well, it's X2 minus X1, Y2 minus X1 again, that seems off. So again, you can get this kind of analysis and trying to get this into developers. Yes, this is, once you see it, it's obvious, but in some cases it's not. So now the question and I'm going to go slightly deeper into the machine learning aspects of things is the question, how do we attack this program? And if you look at this simple snippet of code, the first approach that research do is, let's treat this as a natural language. Let's take that this is a series of tokens, just a big long sequence. And from that, let's try to apply to apply some standard machine learning methods. Now the problem with that is that you lose a lot of context. You lose, for example, that this if statement is within this for statement or that there is this i variable which is used to iterate. So you lose a lot of the structure. You can also, may also want to do this. You can parse the code and compare to natural language and NLP and natural language processing. Machine learning has been used quite a lot there. Parsing code is an ambiguous, most of the cases. So we can parse the code, create a tree and get the abstracts index tree out of here. But again, we're still missing something. In the other day, what we want to do is we want to explore this very rich form of structure that we have.  Data flow is very informative that well we have i which flows from here to here to here to here. And then maybe once we do another loop then we go back and so on. So the idea here is to encode a program by a graph and then use a relatively new machine learning component called graph neural networks that essentially can process and understand graphs. So let's discuss how we get to those graphs. And again, the many things in machine learning, especially in deep learning are design choices. So the way that I'm going to describe the graph is design choices we made. This don't mean that they are the unique and the best choices to make. This means that in this design space, we pick this point. So let's start constructing a graph for this very, very simple snippet of code here that came up from the previous slide. Well, first of all, these are my tokens. I can connect them into a boring chain of, well, assert.now, like here, a boring chain and a special type of edge called next token. That's a bit boring. We can also parse things and yes, that's unambiguous. So we can create the tree and we can connect all the nodes here through this extra type of edge called AST child, for example. So now we have encoded the syntactic parts of the graph but still we are missing many, many things about the semantics of the code over here. So forget about this edge. It will be in the graph but I will stop showing them in the next slide and we'll go to a slightly simpler example. That doesn't do anything real but it has a loop and it will help us to describe how to construct these semantic features within our graphs. So the first thing is we can add an extra type of edge. The previous ones are still included here. That is called last write. Given a specific position within our program, like this Y, when was the last time that Y got written? So in this case, Y is just here. It was written just once but if you are on X, for example, well, the first time that X might have been written is, well, it depends on really where you are. If you are here, the last time you wrote X could be, if you just entered the loop, it could be just here but if you're still looping, it was the previous time you're here. So you encode this information within the graph. Again, just one way to encode this information. The same thing with last use. When was X, for example, last used in my program? Again, the same thing. If we're just entering like Y, let's take again, that's the simplest one. If we just enter the loop, well, it was this instance here but if we are looping, well, it's itself. So we can construct this complex and more complex graph and add more and more of semantic features like computed from, you can imagine adding more of those things. And what happens is that in the end of the day, we have encoded as much information as we think we care at least in our graph about our source code. So if we look like a program, like a simple graph of this, for this example, the graph would actually look something like this. This is not meant for you to read but you see this already becomes quite complicated. So there are a lot of things to parse and what we hope to do is to use machine learning to answer the variable misuse problem I was discussing earlier. I'll get to the point how we do it in the next slide but on average, and this is a very simple example, our graphs, each graph is a single example, a single piece of code, a single variable misuse instance and you get about 900 nodes for graph and about 8,000 edges. So the graphs are not huge. They're not like the Facebook graph of the billion people but they are not small either. So this is the kind of the problem setting we are at and this is how we try to encode programs within graphs and of course the goal here is to take these graphs, push them into a machine learning component that I will discuss in next and then get magically our answer. So how do we encode specifically the variable misuse problem and other problems, other issues can be encoded in different ways. So we had this problem here. We blanked out this variable so we want to create one graph for this case here. What we're gonna do is that, well, we need to predict class but that's not necessarily the point here. We want to replace this syntactically, whatever token was here, we replace it with a slot node that is placed in this location. So now magically we removed any information about what this variable contained here. We don't know what was originally here. We don't know what it will be here. This is going to be our task. Let's try to predict that. And we're going to create extra edges. We're going to cut these candid symbols as we call it. So for everything that is good be in scope and type correct within the slot location, we're going to connect it back to the rest of the graph. We're going to create one node. So in this case we have two things, first and class and we're going to do a speculative data flow analysis. We're going to say if first was in here in the slot, how good data flow around it? If class was in the slot, how good data flow around it? And we connect everything here and there. So now we are, so this gives us essentially a form of an objective. What we want to do is we want to learn something called a representation in machine learning and I'll distribute the vector representation. I'll get to that in the next five minutes. So such that the representation of the correct variable first is as close as possible to the representation of the slot variable and as far away as possible from the class variable. So this is what we're going, we're trying to do with machine learning. So this is the problem setting, our data is graphs. Now the question is what's in the machine learning toolkit that can help us with this problem? So in the next few minutes I'll give an overview of this machine learning components. I won't go into great details because there is not sufficient time. So in the beginning of machine learning people had this idea of local representations. You have a huge vector where everything is zeros but one element, one component is one. And now this helps us discriminate that if the first element is one then our item will be a banana or the other one will be a mango and so on and so forth. But this is not necessarily the most efficient way of learning this. So with machine learning, with deep learning specifically we've moved the two towards these distributed representations, representations that are learned across our data. And the idea here is that we have a much smaller dimension here but these vectors are real vectors in a d-dimensional space. And what they do is that they encode in each of their components some of their attributes. So the meaning is distributed across the components whereas here it's localized. So we can get these distributed representations and you can get this to be anything maybe you have heard of Wartovac. Wartovac is one of the distributed learning representation methods. So now we're going to graph neural networks. And this is essentially the core component that allows us to understand graph, to use graphs with machine learning. So at a very high level what the graph neural network is is that you have a graph representation of your problem. That's up to you what it is. Also you have an initial set of information about each node, local information about node A, B or C and these are distributed vector representations I showed you earlier. And by the end of whatever the graph neural network will do what I want to do is to get representations for those nodes that say not to give them information not just about a node but how it belongs within the broader graph. So how this is done? This is done through something called neural message passing. The idea is let's take this part of the graph here that we have F, D and E. You have the representations over here. And the idea is that F originally has a current representation. It's current state if you wish. It gets as input, it gets messages from its neighbors. We can combine them somehow. Doesn't matter at this point. And you can update your current representation. So at this point as a node if you're a single node you're the node. You receive information from your direct neighbors. You update your own state. That's a graph neural networks. So this is maybe with a bit more concrete equations. I don't think we have a lot of time to go in there. The main component is this GRU. This is a type of an recurrent neural network that is about updating states and giving more information. So this is a graph. And at the first time step my node here have just received information from its distance 1 neighborhood. The same thing for every other node. Because all nodes send and receive messages synchronously in this version of graph neural networks. And the next step, this thing broadens. Now my node 2 received another message from its neighbor. But its neighbor has already received a message from its own neighbors. So now it has information about its distance 2 neighborhood. So in that sense we are getting more and more contextual information as we keep repeating this neural message passing algorithm. So another way of viewing it is that you have the graph. You have it through time steps. And what happens is D and E pass messages to F. They also may receive messages. And this keeps repeating again and again in time. And what the idea is that at the end of the day what's happening is that you're stuck here. The output of the graph neural network is each node has a representation, a distributed vector representation. And the goal here is that you can do anything with this. People have used it for many things like saying which of those nodes should they pick. For example, that's similar to our problem. And like node classification, should this have a label, should it be blue or red, things like that. Or managing like averaging everything here or summarizing it somehow and saying does this graph have property A or X? This has been used a lot in chemistry about classifying molecules. And molecules can't be described within graphs. Going back to variable misuse, the idea here is that we go on GitHub, scrape C sharp code, blank out each variable, ask whether any of the in scope type correct options which one should we use. Assuming there is more than one. And what happens here is that we do get an accuracy of about 85% on projects where we've trained on a portion of them and tested another portion of them. But we also generalize quite well compared to some baselines. So maybe this is a snippet of code, a real snippet of code. And this is the task that our model needs to get. So we have blanked out this variable. And now the question is from the variables that are in scope and type correct, which one should we use? So here there are string variables, three of them, base directory full path and path. Now I won't ask you to, it's because it's fairly early morning to usually ask people, can you think about this? If you think about it, it's not too hard to think about this. You need to reason a bit about the data flow, how things are used, and then you come up with full path. And indeed that's what our logarithm also says that full path should be here. It does that in a few milliseconds though. So in that sense, that's something that helps, gets us closer to understanding source code. So this is essentially how our model kind of works in many cases. And of course it makes mistakes in some cases. So we dogfooted this internally within Microsoft, within this IntelliCode program. And this is a real bug that we caught. Someone was logging something, but then here they decided to use, well, added existing document, really? No, you wanted to use added new document here. So you get this kind of errors that people make. And in many cases, this is a good thing. Overall, I think we found, we learned a lot of lessons, and that's my next slide, but we've decided to discontinue the dog footing that has been happening for the past year. So the main thing is that first of all, we haven't solved the user experience. We haven't solved how to communicate to developers that maybe don't have any experience with machine learning, that our decisions, first of all, are probabilistic. As you saw here, we had a 92% confidence, but it's never 100%. It will never be with machine learning. And also at the same time, it's really hard to tell people you're wrong. So in that sense, we need to be better as a community in finding ways to say this. Of course, there are the questions of false positives. How do you explain false positives? How do you let the users believe that in some cases you have false positives, that's fine. It's up to you to judge. And of course, the developers don't want many false positives. The other thing is machine learning capabilities. Yes, in this local setting of reasoning about code, we get a lot of things. But it's not something that you can do at a much, much larger scale. So you cannot take your full program, your a few billion lines of code, use machine learning and magically understand all the billions of lines of code, how they contribute to a specific point in your program. So I don't think as machine learning researchers, we still have a way to understand and distill the special form that the source code data has, the structure, the size of all these things. So these are open questions that are still bothering us. Then there is metrics and I'm going to go this into the next slide a bit. But in machine learning, you have a loss function, you have something you're trying to optimize. You want to be as accurate at predicting X. You want to have the minimal loss over something. In software engineering, we don't always have that. What is how do you measure quality of a project? Yes, there are some metrics, but these are very high level things. How do you use machine learning? How do you measure things so that you can optimize them with machine learning? Again, that's another big and open problem. In the end, we leave in a low resource world. Yes, there are billions of lines of code, but you have maybe the source code for 10 operating systems. You don't have like ImageNet where you have a 100,000 images and you can use that to generalize to the next one image. We have just 10 operating systems, let's say code of database systems, and we need to generalize on the next one. That's already a problem. This number 10 is actually small because as software engineers, we're trying to be smart, we will use things, so that's a problem. Going back to the learning signals, which I think is the main question here, is we mostly are used in the supervised learning. We have some input data, we have our model, the spherical cow of our problem, and we try to make a target prediction. We minimize using input output examples X and Y, this should be zero, and we try to minimize some loss. Essentially, something that says, well, I want to be able to classify this as closely as to my real data. It doesn't seem that we always have this with software engineering. This is an interesting question, interesting challenge for research, for practitioners. How can we tweak our systems? What do we need to change? How do we measure things? Instead of a conclusion, I think is just to tell you that I think the promise here is that machine learning can help us create those tools that help developers by removing some small, maybe boring, slow tasks that allow the developers to focus on the actual software products that they are trying to build. I think the idea here is that we need to think of this as adding another virtual memory in our team, that initially with baby steps, we'll start pushing forward towards building greater software. So there is a lot of work to be done, but I think it's a very exciting area to be in, and again, thank you for all for being here this early in the morning. Questions? So the question was whether first of all, the graph neural network, how do we initialize the information within the nodes, and whether it's, let's say, robust to removing bits and pieces of the graph neural network. In our case, the way we initialize the graph neural network is that we have natural language information like the variable name, or the name of a type, or the fact that a node is a dot or something like that. So we essentially learn these distributed vector representations for all possible names, all possible bits and pieces of whatever this node represents. In the C sharp land, we add type information because we have that, if you imagine a Python or JavaScript land that we wouldn't add that. So there is a lot of flexibility, this is our design choice. So now with regards to the other question, in practice, if you remove things that are far away from the point that you care about, because in this case, we care about a single point, a single place. Yes, that will be fairly robust. Overall, there's a lot of research on adversarial attacks on graph neural networks. There are cases to trick them like most neural networks. So they get the robustness kind of our eyes depending on the application and the data you have, the training method and so on. So the question was why we discontinued the variable misuse dog footing I was saying. So the first thing was that it's the question of explainability. We cannot necessarily explain to a hundred percent of the developers why we made the suggestion. Even if it's a wrong suggestion, in many cases there is a good reason that there is a good suggestion. But I know it because I developed the algorithm and I understand it's internals. Most software engineers won't understand it or won't be bothered to read everything about my paper to understand why one single code review common is. So I think that's the main problem. So we parse the code with as an abstract syntax tree. We repeat the question first. Yes, sorry. The question is whether do we use just the parse tree or we use the natural language information. So the natural language or within our graph neural network. So the graph itself represents all structural information that is given deterministically. AST, data flow, control flow, all these kind of things. But the nodes have the information like names of variables, names of methods, things like that. This is essentially how we embed natural language information. We don't add things like comments, we don't add anything else, but variable names are very indicative. So that's the only part of the natural language part of the aspects we get. Well, I think we don't have time.