This is, I've been to all 10 Ruby comps and that's not such a huge achievement. Five other people, or four other people have as well. But I've also spoken at every single Ruby comp. And I say this not to brag, but rather to warn you, I am odd, somewhat of a fanboy and very, very sad. But in getting the opportunity to get up in front of the crowd at every single Ruby comp so far, I've had sort of this unique perspective. And the one thing that I'll caveat, everything that I say from here out, is that with any kind of sort of historical record, this is my perspective on history. You may get different perspectives if you go up and talk to different folks who have been in a bunch of different Ruby comps. But this is my take on it. But as I got ready to put together this talk, there was a problem, right? And it was the problem that, it was basically the home video problem, right? So 60 seconds of my very cute kid is very cute. He is very cute. This is my one year old Eton, trying out an ice cream cone for the first time. And so home videos are great and all, but if I made you sit through 45 minutes of me reminiscing with sort of no point, you would want to off yourself at the end. Because my memories are all very meaningful to me, but you probably need something a little bit more to sustain if you're to talk. And so I had to come up with something a little catchier, something a little more interesting. And I think, I hope that I've come up with that. He's really cute, isn't he? Yeah, I think he's definitely gonna be hooked on the fine sugar for the rest of his life, like the rest of us. So to do that, I basically switched this talk from sort of, I was gonna do lightning talks and all my talks prior, which you probably don't care about, but this is gonna be secrets of RubyConf. Basically, throughout the history of coming to RubyConf and being involved with them, I've learned some really interesting insider tips, some things that you would only learn. Some of them you might already know, but some of them you would only learn if you'd spoken at a bunch of these conferences or you'd been at a bunch of them. And I wanna share those with you. I want them to be wider dispersed. I hope that because of them, you're able to get more out of this conference, more out of your involvement with Ruby community in general. And in particular, if you ever decide to create your own community, whether it be around a programming language or more likely a library, even a business, et cetera, that you can take some of these ideas, these secrets and apply them. Now, all my talks always have this idea of being open I want you to tweet about the talk. In particular, again, I'm up against Jim Wyrick. Even if he gets most of the people, I wanna get most of the tweets. So please tweet RubyConf and secrets are the hashtags so that I can find them afterwards. And I look forward to seeing what y'all will say. The one other thing that I'll tell you up front is that I have exactly 44 slides and they all advance after 60 seconds. So this is a little different in that I'm definitely not gonna run over. Might have time for a question or two at the end, but this is sort of a long format Pika, how do you say that, Pika Kucha type talk. So to begin with, we go back to 2001. Very first RubyConf. As they talked about this morning, the pickaxe book has come out here months beforehand. I actually got involved in Ruby first. When the pickaxe book came out, it was at UPSLA, 2001, no, UPSLA 2000, I believe, that it originally came out. I got the pickaxe, got really interested in Ruby. I was working with a small talker at the time and the interesting thing about the genesis of RubyConf is it started off where it sort of tagged along with UPSLA. It would be the two days before UPSLA and then UPSLA and I think this worked out really well because what you found out is that a lot of the people who attended RubyConf were basically, an interest in Ruby were disenfranchised small talkers and that was also the audience for UPSLA. So basically you have all these object people getting interested in Ruby and I say all of these. The first RubyConf was 30 people, approximately. And so we all gathered there in Tampa, Florida. In this hotel, and as I was looking up pictures of the different venues, it was a holiday in when we first were in it. I think early on in RubyConf's history, there was some kind of close correlation between us being at a hotel and it later, like going out of business and getting bought by another chain, so take that for what you will. But RubyConf 2001, the amazing thing to me about it was that there was no corporate entity behind Ruby that was pushing RubyConf forward. There was no sort of organization that was saying, we're gonna make sure that Ruby gets big in the US. Rather, RubyConf happened because a bunch of people on the mailing list, the English language mailing list wasn't even that old itself at the time, said we'd really like to meet this guy max because he posts on the English language mailing list in riddles, these short, pithy sayings, and he's like this mysterious Japanese figure who created this language we love, we'd really like to meet him. So let's put together a conference that basically just has enough money to bring max over and pay for the venue and that's what happened. But sort of my inspiration for this secret of RubyConf is Guy Hurst because, Dave mentioned him in the talk, even though at the last minute some other folks had to step in and make sure that RubyConf happened, it was Guy Hurst who was the genesis for RubyConf. Without him, it would not have happened that first year. And the really amazing thing is that Guy Hurst, as far as I know, isn't involved with Ruby at all now. But yet, at that point in time, he was the right person in the right place and he took initiative to make something happen. And that was awesome because it changed people's lives. RubyConf 2001 changed the trajectory of my professional life, period, flat out. And here's one guy who, because he got involved, made that happen, and not just for me but for a lot of other people. I daresay that the majority of the people in this room would not be here if not for Guy Hurst starting the ball rolling on RubyConf 2001. And the secret here is that leading is being followed. You just start walking in a direction and you start setting the tone for something you're passionate about and you gather people behind you in your wake and something happens. And it's something that anybody can do. There's no need to be a famous person. There's no need to already have some level of notoriety. A lot of times there's no need even to be involved long term. You get something going and then it becomes self-sustaining. And that was the secret that I learned from Guy Hurst that very first year at RubyConf. And it's something that I've seen throughout, woven throughout the Ruby community ever since where regular people get involved and make stuff happen. So 2002, RubyConf moves on to Seattle, Washington. Home of the space needle. That odd piece of architecture that I don't even know what it's there for. And it was originally for the World Trade Fair but it's still there. And in 2002 for me was a particularly memorable RubyConf. And it had nothing to do with Ruby. It had to do with the fact that I'd just gotten married about two weeks earlier and came to RubyConf almost immediately. After the honeymoon and brought my new bride with me. So I was having as good a time at the conference as I was at the hotel as I was at the conference. But so we all gathered together there in RubyConf. And it was, there were a lot of things that were interesting about that year. One was the initial sort of rise of the Seattle Ruby Brigade. The very first .rb group. They coined the .rb sort of post fix for local Ruby meetups. And so a lot of those folks showed up at RubyConf for the first time that year. I also found it interesting when I went and pulled this off of archive. It still says RubyConf.new2001 but then it has the right schedule. So I don't know if that's a fact of archiving or what but it was still a very community driven conference. But the really interesting thing about RubyConf 2002 was that this guy showed up and talked about YAML. And YAML was very new at that point in time, right? It had just come out. And this guy who called himself why the lucky stiff came and talked about YAML. And the weird thing is that looking back why did he was not an alt rocker absurdist personality at that point in time. He was maybe headed in that direction but I remember why as being preppy surfer dude, okay? I couldn't find a picture of him then but he came across as very different than what this persona ended up being. But the great thing about why was that he sort of dug into Ruby early on and he introduced us all to YAML which is pervasive throughout the Ruby community now. And something we're all familiar with. But he worked on all those different projects and constantly invited contributions and he wanted people to make things with the things that he was making. He didn't want them to simply use them but he wanted people to make things. And the secret that I learned from Y was that famous people are able to. So it's really easy to see that person up on stage.  But it has nothing to do with him being famous. It just has to do with the fact that you can't hear you. But in general, anyone who you think of in the Ruby community as being a famous person, they love to talk about the things that they're passionate about. They would love it if you said, oh, I'm really interested in this particular topic. Let's talk about it. And so don't ever hesitate to go up and talk to the famous people in any given community because they're just people like you are, just like why was. Now, nobody's a person like why was. And it's really, anyhow, he made it really hard to say sentences about it. But at the same time, I think that this is yet another lesson that we can learn from Y the Lucky Step. So it's 2003 and RubyConf moves on to Austin Texts. Now you may be starting to notice a pattern here. RubyConf all except for one year has always gone East Coast, West Coast Central. East Coast, West Coast Central. So this is the first Central Conference in Austin, Texas. Memorable for me because I got the opportunity to stay with Hal Fulton. He lived in Austin, Texas and I stayed with him at the conference. Now, I say this and this is just kind of, this is a secret that isn't even on a slide deck. So only you get it and anybody watching the video. And that is that, always stay at the conference hotel. Hal was awesome. I enjoyed spending time with him. But you miss out on stuff if you're not right there where the action is happening. And in particular, in 2003, that action all centered around the project that Eric just talked about, which is Ruby Generals. Now in 2001, at the very first RubyConf, there was a talk by a guy named Brian Levengood on this project called Ruby Generals. And it was this cool package management thing that he was working on. And then it just kind of fizzled that on the line. But then as time went on, right, we're two years later now, and everybody's going, we need a package management solution for Ruby. And so a bunch of folks got together at RubyConf 2003. And I remember coming to the hotel and going into the bar. And there is just this hive of activity. I mean, literally just like people moving all over the place through these round tables like this, like this hive and the super high chairs that go with them. And there's like two or three laptops crammed on each one of these little tiny tables. And people are like crowded around them, hacking away. And what had happened was Rich Kilmer, in particular, I remember sort of saying, and I think Chad made, there were a bunch of people involved in this. But I think of Rich because he's kind of my point's essential, he can hack more code with his pinky than I could do if I could use my toes as well as my fingers. And he basically said, look, everybody's complaining about this, but we should actually do something about it. Let's go and steal the name and any concepts we can from the older regions and let's actually code this up. Let's make something happen. And it did. And now you could say something about the code quality. Typically, a project that starts at a bar is not gonna be the best ever in terms of test coverage and clean code and architecture. But it was a start and it was enough and it got people excited enough that they went back and they continued to refine it and develop it and do more with it. And RubyGems has influenced not just the Ruby community, but it's also influenced a bunch of other programming languages who look at RubyGems and go, wow, we wish we had a package management system like that. And then they build it. And so I am really, really happy to have found that Hackfest going on in the bar in 2003. And the secret is that conferences are for Hackfest and for other such interactive things that you can do only at a conference. It's really easy to come to a conference and basically pick up a bunch of information that you could have just as easily read on the web next week when it got published. Instead, I encourage you to come to a conference and look for those sessions, look for those talks, look for in particular those opportunities to interact with other people in ways that you could not otherwise. So, and the other really interesting thing to me about the RubyConf 2003 Hackfest is that it was so all-inclusive of a conference. I think almost everyone who was at the conference that year touched the RubyGems source code in some way. I remember standing at a table with David Allen Black looking at the version comparison code and doing something with it. And I think that was a common experience for almost everyone that was at the conference that year. And so look for how you can build Hackfest, et cetera, that aren't just the normal characters, but how can you draw in other people, get them excited about what you're doing and hack together with them. So 2004, RubyConf goes to Chance Lee, Virginia, otherwise known as pretty much a suburb of D.C. And this is the year that everything changed, but we didn't know it yet. This was the last year where RubyConf did not, as far as I know, sell out at seats. I don't think there were any other years after this that where RubyConf didn't sell out. But we had no clue, right? We just all went to another RubyConf that was about 60 people, right? So over about over three years at this point, it had pretty much doubled in size, which isn't really a very great growth rate, if you're wondering. But it was a lot of fun. It was usually the same crowd, but this guy with this really hard to understand a proud-nounced name and even harder to spell, he was talking on the mailing list, he kind of popped up all of a sudden and he's like, I'm working on this and sticky thing and I have this framework and you should check it out. It was exciting and it made this blog post or blog engine in 10 minutes video. We're like, okay, this is kind of interesting. So it comes out to RubyConf 2004 and presents on Rails. And it was a good presentation. The main thing I took away with it was he was using TextMate and I was like, I wanna get me some of that. And so he basically gave a pre-release version to most of the people that were there. It was also the beginning of the Mac ascendancy. I actually think Apple should have sponsored that year because it was from sort of that point in time that I remember it became like the only thing, like right now, let's see, I see one non-Mac computer at the moment in the audience. But David came out and he talked about, like I said, the presentation was good, it was solid. It wasn't like, wow, this is the most amazing thing ever. But the interesting thing was he didn't stop there.  But David morphed himself from the geek into the amazing marketer and took Rails to a point where it was sustainable and he was able to work on his passion all the time and have so many other people contributing to this project. And of course it greatly affected the Ruby community as a whole. So what I learned, the secret that I learned from David is that your passion is worth marketing. And this is something that we geeks fail at so utterly, so often. We look at marketing as being a bad word. You know, I don't want to be one of those hype guys with everybody drinking the Kool-Aid, but the truth is if you want to be able to work on your passion long term, if you want a lot of other people using it, you've got to be bold enough to go out there and say, this is awesome. I think this is awesome. I think that you'll find it to be awesome too, and I really think you should get involved. And look at how it's changed this community. It hasn't just improved Ruby around Rails, it has improved Ruby in general. The Rubinius project, as it exists today, would not have happened if not for Rails. You could point it, Rack would not be here if not for Rails. Project after project after project after cool thing after cool thing after cool thing all because one guy, one geek, decided to actually get serious about marketing what he was passionate about. And you can do that too. So 2005, RubyConf goes to San Diego to a sold out crowd of the expectation wasn't that it would sell out. So it wasn't actually that big of a crowd. It was like 150, 170 people. Really interesting venue. It's like, there were like the Bob Hope room and the Lucille Ball room, et cetera. And it's basically like this hotel that in its glory days hosted all these stars. Little did they know that we were coming. And down in the basement, there was the meeting room and it was like, if you've ever, if you ever saw the Rocketeer, sort of that 50s style lounge singer shell on the back kind of a thing. And so we're there in 2005 and it's very obvious that things are changing and they're changing very, very rapidly. And a lot of that was around Rails. Every single break, the Rails guys, and there was now the Rails guys, the Rails guys were in this, in the lobby working on the next release of Rails. There's so many more people. There's a ton of excitement around the whole conference. And I had the opportunity to give what I think was my best talk ever. So if I was gonna personally rank all my talks, 2003 in Austin would have been the worst ever and 2005 would be the best ever. So it was on the long tail of software. You can still find an essay form out on the web. But the thing that I learned from that was that, well, let me tell you first about the person that I met through that. So he's sitting here in the audience.  He's like, hey man, I'm working on some cool stuff. Here's my car. Let me just slate. And I don't remember if he contacted me first or I contacted him first, but we started talking. And because of that initial conversation, we looked at one business idea. He introduced me to his designer friend. And then three years ago, I said, I came up with a business idea and I was looking for people to partner up with. John was one of them. Alex Koldeaufer, the designer he'd introduced me to was another. Daphne Amelia was the fourth one of us. And we basically formed Spreedley at that point in time. Spreedley would not have the founders that it has today and probably wouldn't even exist today if I'd not given that talk. And so the thing that John taught me that day, the secret, was that speakers have the best conversations. They basically get to have a whole set of conversations. Here's why this is, right? So it's this basic human act. I was just reading about speaking. The thing is when I'm up here, I'm like kind of terrified by the fact that I'm facing off against all of you. The book I was reading said that basically there's nothing scarier to our sort of more basic brain than being up in front of a group of other individuals all alone in a situation where we pretty much can't run away.   And so you'll come up and talk to me in a way that you wouldn't have otherwise. And this is why speakers get to have the best conversations, especially technical conferences, is because now we have established rapport and I get to have interesting conversations with all of you that if I was a non-speaker and my normal sort of fearful of public or personal, interpersonal interaction with strangers I wouldn't normally have. So I highly, highly recommend that if you have the opportunity to speak at a conference, take it.  Everybody who does any amount of public speaking has those nervous moments. The difference is that my butterfly is now flying formation, I stole that from somebody else. But so anyhow, take the opportunity to speak because you will have better conversations because of it. So 2006, RubyConf heads to Denver, Colorado, which, if 2005 was the best talk I ever gave, 2006 was bar none of the best RubyConf ever. And it was just a combination of factors. Everything was firing and all the knobs were turned up to 10. The content was amazing, the venue was great, it was our first time ever at an embassy suite with the whole atrium in the middle and people just congregated there and hacked and played Nintendo DS and talked. And there was just this whole, I played my first ever game of Katan, right? So it was just an amazing year for RubyConf. And it was also the very last year that RubyConf was single track. And there's something different about a single track conference. There's something, there's a shared experience that happens at a single track conference that you just, as wonderful as larger conferences are and the things that you can do with them that you can't do at a smaller conference, there's something really cool about a single track conference and RubyConf 2006 was that conference. But there was this one guy who came to RubyConf 2006 and he gave this really funny talk where he was talking about fuzzing web interfaces.   So the rest of the conference, everybody's like, don't be that guy. And so it was really funny. This, of course, was Zed Shaw. And Zed Shaw is a character, as Dave alluded to, in his talk.  But, Oh. Oh. But Zed did teach me something really interesting and it kind of goes back to an underlying principle in the Ruby community and that is men a swan. Maths is nice, so we're nice. And really the thing that I've always appreciated about the Ruby community is that there is no problem with having lots of heated discussion and debate about concepts. But when it descends to the level of personal attack, you stop talking about concepts and you start dealing with all kinds of personal stuff that doesn't really help. And Zed kind of taught me that. But the real secret is that controversy is the best and the worst recruiter for a community. It's the best because there is no better way to attract a whole bunch of attention to your community than have something blow up really big.  But it's also the worst in that it's not, those aren't always the folks that you want to necessarily attract to your community. And also a lot of the people that come in and see that kind of conflict are gonna go, I'm not really interested in this and keep moving. I still like to read Zed, but if I had to be completely honest, I'm glad he hates the Ruby community. So in 2007, RubyConf moves on to Charlotte, North Carolina, three hours from my hometown of Raleigh. And it's nice when it's on the East Coast, I can always try it. And so it met in downtown Charlotte, even though Charlotte's three hours away from Raleigh is the most time I've ever spent in Charlotte before.  Happened in the very impressively shaped comedy hotel. And the thing that I think everyone who's at RubyConf 2007 remembers about that RubyConf is werewolf. It was werewolf this and werewolf that and werewolf the other. And here's the truth, it was a blast. It was so much fun. I remember sitting around and got to know some people about a level that I sort of did before. Now I got to know sort of their dark side. I can tell really good lies side, but still I got to know them at a level that I hadn't previously. The other really good thing about 2007 being about werewolf so much is that I think we burned it completely out of our system. As far as I can tell, there's never been a RubyConf conference since that was dominated by werewolf to that level. It was like, okay, we're done now, that was fun. We moved on, hopefully to hack in pursuits, et cetera. But RubyConf 2007, I gave a talk entitled Why Camping Matters, and I related the importance of hacking to the concept of a bacon, egg, and cheese biscuit. That was highly entertaining if I do say so myself. But the interesting thing was I had somebody come up to me afterwards and say, when you talk at the same conference the same time, you start to get a few groovies. But it's okay. And so one of them came up to me, very positive guy in general, and said, I love the fact that you always have like this theme, and there's always like something different and entertaining and interesting about your talks. And I kind of thought, wow, I'm glad to work here. In 2003. So Joe Bryant basically sort of said this, and then Joe has gone on to be extremely entertaining himself. Has really developed his speaking skills and teaches it a way that I think people really get because it's interactive and interesting and entertaining. He spends a lot of time with Jim Weirich who does the same thing. And so the thing that I really learned from that experience, the secret is that conferences are entertainment. You thought you came here to learn something, but I say that you came here to be entertained and happened to learn something as a side effect. The fun things about conferences are the laughs, the interesting things that you never would have learned unless you'd been sitting across the table from that person, the more intense social interactions, the opportunity to just interact on a level and have fun at a level that you can't have online. And so I would encourage you as you approach your conference attendance as you're here at RubyConf, as you go to conferences in the future, if you ever organize a conference, as you present at conferences, to keep this in mind that you should think as much about entertaining the audience as about, and also seek out those opportunities to be entertained because those are the places where you're going to learn something because it's gonna stick in your mind. I most remember some of the most entertaining talks throughout all of the RubyConf conferences. And so definitely keep that in mind as you attend and as you present. So 2008, RubyConf goes to Orlando, Florida, the first time that it did not cycle. It's stuck on the East Coast, yes, my coast. And stayed there and moved to Orlando, Florida and sort of, it was cool in one way and not so cool in the other, but the centerpiece of RubyConf 2008 was location because this is definitely the only RubyConf where there was a lazy river behind the hotel, okay?  There were good restaurants in the hotel which was good because it was completely isolated from anything else. And so there were there were pluses and minuses there. 2008 was a lot of fun, but the thing that I really noticed in 2008 was some other folks came that I knew like from the Raleigh area, et cetera. And while they enjoyed it, it seemed like I was getting so much more out of the conference. It just seemed like, I was, I just having these much more interesting conversations was that there was some kind of in-crowd, some kind of click going on. I wasn't sure like what exactly was causing this. And so as I, as the week went by and as we sort of kept going through, I started to, I got inspired by a presentation that the Seattle Rivergate did. Basically a bunch of them got up and talked about a whole bunch of different projects that they had going on. And in thinking about that, the sort of the conclusion that I came to with that, they just had Seattle RV to back up a little bit has just always inspired me with their sense of community. The fact that they are just always hacking on something really hardcore and they get so many different people involved in it. And this guy has this project but this other guy works with them on it. And I think they have like weekly hack nights which is really cool. We manage monthly and rally. And so the inspiration for me is sort of the realization that I started coming to came from this guy, Ryan Davis, and what he inspired at Seattle RV. Now this picture was challenging to find. Finding a picture of Ryan that's smiling at the camera and he isn't simultaneously flipping the camera man off. It's very challenging. Is that why you're talking to a person?   And he could definitely take me in a fight. So, but Ryan and the Seattle RV, I just remember at multiple Ruby comps coming along and like sitting down to Ryan, talking some, we kind of knew each other from back in 2002 and he'd always be hacking on something. And man, if you're ready to, he'll just like grab you and be like, I'm working on this and I'm trying to figure this out. He's more than ready to include you at any point in time. And what I came to realize is that within any community, you get clicks that form. They're basically just groupings of people who with common interests and the best clicks are meritocracies. The kinds of groupings that you should seek to form are meritocratists, if that's a word, and in that, you want to invite anyone in that's ready to contribute. If you're ready to dig in, then come on and join us. And the thing that I saw there in 2008 was that, yes, there were some people who weren't as engaged in the conference, but I found that that was mostly because they weren't as involved in the conference. They were working on this thing over here or they were out over here as opposed to really engaging with the conference as a whole. So seek to make and seek to join clicks that are meritocracies. Now, 2009, Rumiqoff moves on to San Francisco, California. And the 2009 was the year that I learned that 5K wasn't simply a measurement of distance. It was actually a race. And a run that you could do. Now I did not run that year, but I am gonna run tomorrow morning as I hope some of you are. John talked me into it. See, there are other things that can happen when you meet people because you give conference talks. And so the really interesting thing to me about 2009 was the contrast with 2001. In 2001, there's 30 people. When the room was asked, that's less, you realize that that is significantly less the number of people in this room as sparsely populated as it is. It's really, it's probably about as many people as there are on this side of the room. And when asked how many of them were getting paid to do Ruby, one of them raised their hand. And yet, last year in San Francisco, we pack out the embassy suites there. And not only that, but the startups in San Francisco set up a bus tour specifically to like bust the Ruby people around because they're all using Rails and they can't find developers. Like this is the contracts, the change that's happened over nine years. And it's absolutely phenomenal, the growth that we've seen. But the downside is that these conferences keep growing bigger and bigger.  So please come up and let's talk about stuff. But seriously, how do you deal with this growth? Well, in 2006, the first Ruby ho-down happened. And this was memorable to me, not because it was the first regional conference, et cetera. It was just the first conference I ever co-organized. And the last one to date as well. And because of it, but the fascinating thing to me about the Ruby community is I've never seen a programming community that has been so focused on spreading out to a bunch of different conferences. Ruby Central sponsors them. Jeremy McAnally is my hero because he co-organized the first ho-down. And when I said, when the second one came up, I said, man, I am too busy, I cannot do this. He just went and did the whole thing himself from then on out. So he's really my hero. And so Jeremy put the ho-down together. But the first regional Ruby conference, I believe, was Yuruko in 2003 in Europe. Some local Rubyists in Europe said, we want to have our conference too. And they got together. I think there were probably less of them than there were at the first Ruby conference in 2001. But they had a blast. They continued to do it. And that has been a constant pattern. And the secret is, and this is really important, if you're going to build a community and you market it like crazy and it grows, recognize that conferences scale best horizontally. So look for ways that you can scale out your community, spread people out, find lots of events that people can get involved with. It's one of the reasons that this conference has such great talks is because there are so many regional conferences that act as proving grounds for new speakers and give the new folks the opportunity to show their stuff and for people to get excited about it. And the thing is, when you do a regional conference, can I just say, make it a single track? I've seen some folks starting up regional conferences and trying to do multi-track. But the cool part about a small regional conference is you can do one track and it gives you shared experience across the whole group. So conferences do scale best horizontally. The big ones are fun. Don't deny it. But the small ones are where I think the real learning and the real growth happen. So here we are, 2010, New Orleans, Louisiana. And who knows?  And I think, what does nine, 10 years from now look like? I have no earthly idea because I sure would not have predicted this in 2001. None of us would have predicted this back in 2001. We never, we, at all programming languages hope for the killer application. Very few of them find it. And it's not something that you can sort of create out of the blue. But it's really fun to be here. My hope is that these secrets of RubyConf give you some ideas for maximizing your experience here. That as you go through the conference, you'll be looking for those sessions that are going to entertain you as well as inform you that in the future as you present that you will be looking towards how you can learn from the audience afterwards by having an ongoing conversation with them. But my real, the thing that I really wonder about this year is when I look back at it in 10 years, what am I going to remember? I don't know yet. It just started, right? And some of the things that I remember from so long ago, I didn't seem to matter at the time. I know one thing that I will remember about this year and that is being put up against Jim Weirich. Bless his heart. And so, what I'm wondering is when I think of the people who influenced me at RubyConf 2010, who will it be? It could be one of you. It probably will be one of you. And I look forward to seeing how I am shaped by yet another conference. And I haven't decided. Next year, do I do another talk or do I skip a year to take the pressure off? I'm not sure. But I want to thank all of you for coming, for listening, for being involved. And I want to go ahead and take time for two or three questions I can answer on history. I will also mention, I have all of RubyConf t-shirts laid out over here. Feel free to check them out if you're interested. So, any questions? Yes? So, I'm going to mark the final five questions by asking you if you'd like to come to our little regional conference in Sweden and talk about removing it soon. It sounds awesome, man. It is awesome. Yeah, I've heard really good things about it. So, I will take that as an invite and go ahead and get it on the calendar. Awesome. I'll talk more later. Yeah. Anybody else? Nathaniel, I just want to say this is a great topic to encourage people to get out of their shell and to speak about their passions. And I really want to just sleep. What was your 2003 talk? What was so bad? I don't remember it being bad. So, it was a clueless Ruby hacker explorer security and the speaker is always much more critical of their talks than anybody else. But to me, it was just kind of flat. So, you can actually find the, I went back and looked at the slides and there's some content there, I guess. Yeah. Yeah, follow up on the Nordic Ruby thing. We were talking last night about trying to resurrect their hello-hunter, so if you want to come up with it. Yeah. Because we might kill it. We made you really comfortable lying next to you. Well, that would be great. Wow. Is that West Coast door even open? Yeah. I don't know if it's going to be open. Does that make it a little bit psychic? One, if I go to Australia. That would be, no. Well, the problem with Hawaii is I have to bring my whole family. If I go to Hawaii by myself, I am in so much trouble. The thing about I was at San Diego, and that long-tailed talk was awesome. Yeah. It was awesome. You know, I don't want to toot my own horn, but at the same time, if I look at all of my talks, some of which have been quite lame, that one was awesome. So, you should go and read it if you haven't. It's still out there. I am sorry to say that the mailing list that will spawn from that is totally defunct. Yeah. 100% spam right now, so. I forgot there was a mailing list. I don't know yet. I don't know. Not sure. I haven't decided on that. Not sure what I think about this in any way. I have a question on exactly this topic. Yes. You've encouraged people to participate deeply with the commerce that they attend, and yet, if you bring your family, isn't that some other distraction? Are there some other obligated distractive time with them if you rub them along? How do you balance those things? I would like to be the same, and I don't see myself holding on. Well, so, Katie and I have an agreement that when I go to conferences, I disappear. And she likes to come along just so that she gets to see me at all, as opposed to not seeing me at all. And so, the kids, even though they don't, no, I did see there was a kids' track this year. I didn't know that that was happening, but as my kids get older, that could be a lot of fun to do. But at the same time, the kids get something out of just being here and just coming in when I'm giving a talk and being in the back and just seeing that this is when I come to a conference is what I do. But it is challenging, but again, it's mostly about setting expectations. All right, well, thank you so much for coming.


Good afternoon, my name is Mesh, and I work as a principal consultant at Fort Worths. And my colleague, Mushtaq, is a Scala geek. So last several months, we have been working on this very interesting project called 30-meter telescope. And we are building core infrastructure services and common frameworks for the telescope software. And one of the first things we build over these months is a service discovery mechanism. And we are going to share some of our learnings of how we could use CRDTs to effectively build a service discovery mechanism. Quickly about thought worlds, I think, because a lot many people are, we are thought workers, we can quickly skip through this slide. But we are around 4,500 people across 15 countries and 42 offices. We are from thought worlds, Pune. We are doing this project from Pune office. So a quick word on 30-meter telescope. This is going to be one of the largest optical telescopes when it's operational. So it's going to be operational in 2027. As of now, that's the plan. And when I say largest, what it means is the size of the primary mirror of the telescope is 30 meters in diameter. The current largest that's operational is 10 meters of the mirror. So building the telescope of this size, having this mirror of this size and operating it is a huge technological challenge. A lot of hardware that is getting built, even that is getting built for the first time, known as built, for example, mirror of this size. So this mirror is going to be made up of 500 mirror segments. It's not going to be single mirror. And you have actuators and controls for controlling each of that mirror segment. And a lot of that is experimenting. People who are building it are doing it for the first time. There are five countries who are involved in building this. So it's a consortium of five countries, US, Canada, China, Japan and India. And software development is primarily happening in India. So let's look at the logical architecture for which we build this service discovery mechanism. So as you see in this building, the telescope is going to be a huge structure. So it's going to be a four or five-story building like this one with aperture on the top, which will have that mirror for the telescope. And obviously to operate a telescope this large, there will be hundreds of hardware controls that you need to have. So if I just talk of logical architecture in terms of layers, at the bottom most layer, you have the hardware controls with embedded software on them and each having its own proprietary protocol to control that hardware. To talk to this hardware, there is something called as hardware control demon layer, which is similar to device drivers. So each hardware control demon will be built specifically for a particular hardware. And on this side it will talk the proprietary protocol for that hardware. And it will expose programmatic interfaces for upper layers to talk and control this hardware. Now about this, this is more like aggregating layer which is called as assembly layer. To give you an example of this, telescope has a subsystem called as laser subsystem which is used for forming an artificial star in the sky or calibrating the telescope. And that subsystem has six laser devices which need to be controlled. Now in terms of the hardware and its CDs, you will have six devices to be controlled at this layer. And you will have six hardware control demon instances running for controlling each of that. Now to control this subsystem as a whole, this laser control subsystem, you will have a laser control assembly to which the upper layer, upper operational software will talk to control this laser. About this, we have something called as a sequencer layer. And sequencer is to control multiple subsystems. So when you schedule a telescope observation, you need to operate on a telescope as a whole. So you need to move a telescope, calibrate, then take pictures. And all of that are steps that you need to program. And that's what happens at a sequencer layer. And then you have application through which you control all of this. So this is how the layers and these are the components. Above this hardware, all the components are built on JVM. So they are Java components. And because the communication between all these components is asynchronous message parsing. So they do maintain state for which is device specific state. And it's not synchronous. So it's asynchronous message parsing. So actors was a very well suited way of building these components. So these components are getting built as ACCA actors because it needs to be on JVM. And one of the key requirements for this was if you need to talk to any one component, you need to talk to other components like ACCA actor. It needs to be discoverable. So the IP address and port, you should not be keeping any configuration. So if a particular actor or device restarts on a different port, it needs to be discoverable at runtime. So there needs to be a zero configuration. And this is a very common requirement if you are familiar with microservices. I mean microservices primarily in the container world, you need to build something like service discovery mechanism. And most container frameworks give you that. So we had to build something very similar but slightly different because we also were using actors and we needed to restore actors. So let's look at some of the common requirements that your service registry, yeah. So we'll cover that. The first half of the talk is giving the context and component. The second half will focus on the CRDT specific solution. So we are just building the context on what the discovery mechanism is and how then the second half we'll see how CRDTs can solve that. So for building service registry and having a service discovery mechanism, you need to fulfill these common requirements. Any service registry or any mechanism needs to have. First is it itself should have minimal configuration. So the registry itself should not have a lot of configuration that you need to have. It should be fault tolerant because your whole system depends on the service registry to be operational. So there shouldn't be any single point of failure. You need to have health checks built in. Because if you are figuring out what other services are available and their endpoints, you also need to know if they are healthy or not. And other important requirement is that of watches. And what it means is when you look for a particular service, if that service is not available, you should be able to register your interest saying when this is available, notify me. Or if there is any change, like a service registers itself, but then it restarts and then registers a different IP address and hope, anyone who is dependent on that service should be notified for that change. It should expose APIs so that services can self-register themselves using that API. And if there are any third-party libraries or tools that you are using, like for example we are using Redis or Kafka, let's say, you should be able to have agents which can register those services as well. And obviously you should support not only actor rates but all kinds of services, so HTTP services or just TCP ones. So let's quickly see, because as I said, this is a common requirement fulfilled by many frameworks in the microservices world. One of the common questions we get is why not to use something like console or zookeeper, which already solved this problem. And we did evaluate, so before moving to CRDT-based solution, we did evaluate all three of these. So let's very quickly go through how service discovery works in zookeeper, HDD and console. So zookeeper, if you know, at the core is 3 to 5 node cluster of servers, which form a cluster using a protocol called as Zab. So that gives you fault tolerance and linearizable guarantees. If anyone goes down, you already have values preserved in others. And then it gives you a file system kind of an API, which the clients can use for various purposes. One of that can be service discovery. So to use it for service discovery, what you can do is, when a service registers itself, you can create a node in zookeeper and you can have a parent node for each service type and for each instance, you can have guy nodes. And then a client can look up zookeeper plan. The dependent service can look up zookeeper plan. Health check is a bit rudimentary in zookeeper. So the node that you create for a service instance is typically ephemeral node. And what it means to be ephemeral node is, till the point that connection from that service is open to zookeeper, that node is present. The time the connection drops, node goes away. And that's kind of an indication that the service is not available. It's very rudimentary support. It does support watches. So all the three frameworks do support watches. But watches are because this is client server. You have a separate set of servers, which you need to talk to. Watches are implemented using long polling. So you need to have a connection open. So there is a watch request that you can make to zookeeper. But you need to always have a connection open. And on that connection, you get notifications. So if, for example, you watch the service one node, anything that happens, new instance getting added or removed, you will get notified. Given you have a connection open for that. The second one is its CD. Very similar to zookeeper. At a core, it has three to five server instances. Forming a cluster. And you do this draft protocol for giving you consistency. Again, it has a GRPC-based client API, which clients can use to register, services can use to register themselves. And before, it's not a file system API, but it's a file system kind of API. It's more like a key value store. And its CD is more popular because Kubernetes internally uses it for exactly this purpose. Service discovery for Docker containers. And what it essentially does is you register using the GRPC API, and it creates a key value pair in its CD. And you can keep timeouts. So you can create a key value store, key and a value with a timeout. And then it's the responsibility of the client to keep it alive. If it goes away, I mean, if it cannot keep it alive, it expires after that one time. And you also have APIs similar to Jupyter for Lookup and Watches. Watches here are slightly more optimal because it's GRPC. It works on HPV2, so you don't need to create multiple connections. On a single connection, you can have multiple watches. The third one is Consul. Now, to start with, it again has at least similar architecture at a core 325 node cluster, which is RAF based. What is interesting in terms of Consul is it also has a concept of agents. And your services deal with agents to register themselves. So either you can use an API to register, or you can have a configuration file and an agent on behalf of the service registers that service. It also has a concept of health check built-in. So you can register health checks, which Consul can use to determine the health of the service. And it has Lookups and Watch interface as well. Watches, again, because this is long pole, because it's a client server, it's based on long pole. Now, interesting part of Consul is the way agents and servers form a cluster. So across servers, the consistency is maintained by RAF. But agents and servers together form a cluster using something called as a Gossip Protocol, which is served. And Gossip Protocol is a very common protocol that is used in clustered applications for maintaining group membership list. So in Consul, you know about all the services, but you also know the state of all the nodes, if the node is alive or not, or which all nodes are part of the cluster. And that information is propagated across the cluster using the Gossip Protocol, which is served. And on the service registration, the service registry is kept on the server. But what is interesting is that, because Consul is used for other purposes than service discovery, like keeping configuration and all, you need to have the central registry. But if you just need to have... Yeah, sure. It is a key value store. I mean, it's a key value store on top of which you build service registry. But the core usage of Consul is not just service discovery. I mean, you can use it for other purposes, like keeping application configuration as well. But if you just need service discovery, keeping information about services, and that needs to be propagated across the cluster, if you have something like Gossip, which Consul already has. Consul, like the node information is propagated using Gossip Protocol. If there is a user API that is exposed and that is available to you at application layer, and your application can form a cluster where you can keep your own information that can be propagated on Gossip, you probably can build service discovery or service registry on top of Gossip. And that data structure that you need for doing these kind of things is CRDD, essentially. Consul, HashiCorp has a library called as MemberList, which just gives you this surf implementation. And it doesn't have exact CRDD support, but it has a broadcast support. So you can keep application-specific data which can be propagated across all the nodes. And because we were using Akka, Akka has a concept of a cluster in build. It's a very good support for clustering. And on top of that cluster, it has CRDD library that is also very well... That's also very good, called distributed data. So we could use distributed data on top of Akka cluster to implement something which is very similar to Consul and which worked very well for us. So Muscat will talk about that specific implementation. Before we go to CRDD, I think in my words, I will just say what is intuition? Intuition is that if whatever mechanism gossip-based cluster of Consul is using to maintain its own member list, available to you as an application developer, then you can keep your own member list, which will be your services. That's the intuition. And then if you just read up, you will know that in any gossip-based cluster, the membership list is kept consistent, eventually consistent, using a data system, which is CRDD. Whether you call it a CRDD or not, does not sign up. That was the case with Akka, but then Akka moved ahead and actually provided the same mechanism that they're using to maintain the cluster list as a library. And we are going to talk about that later. But before we do that, let's revisit CRDD. How many of you heard CRDD in the keynote in November? Okay, so you registered that. It's an interesting RLang community champion, this idea, and it's getting popular in all gossip-based clusters. And we'll see why. But to understand what a CRDD is based on the definition, there are generally a conflict-free replicated data type, which means if your cluster has 10 nodes, the information that you're trying to update and get will be replicated, all of that, across the 10 nodes. That's the meaning of it. The meaning of conflict-free is that there will be a mechanism to come up with the same list across the nodes, which will not involve conflict. And it will not require a coordination. For example, Raft is a coordination-based system. It values consistency. So coordination will give you a consistent set of values. But here you can do it without coordination. And why, how is it possible? Because the operations allowed on these data structures are very, very restricted. And that restriction is defined or enforced by them requiring to have a merge function. I think briefly we talked of merge function. Let's see what it is with an example. Again, to build an intuition set is a, I can say, is a CRDD. And assume that if you have a three-node cluster which allows add operation, add of members, services, whatever you call it. And each such operation is allowed concurrently on different nodes. At some point in time, you will have different data. So in the first node, you will know about AB. And on the last node, you'll know about B and so on. So you are looking at different data. If you query at that point for a set value or check if a value is present, you might get different results or different nodes. But we also know that the reason we chose set as an example is because it has some interesting properties. Like it has a union function. And all of us understand set union intuitively. That set union, if we apply basically n number of times, it gives you the same set. And set union, if you apply in any order to three sets, it will give you the same value. And using that principle, you can be assured that eventually all these nodes are going to see the same set of information. This union function, which is available on a set, is called as a merge function. Why? Because it has certain properties that are essential. And those properties are associative, commutative, and idempotive. I mean, those are the terms from algebra. But just for intuition, it means that if operation is associative and commutative, then order does not matter. Which means if I'm getting in a large cluster, I'm getting basically a union operation coming via two different routes. In one case, I will receive your information before his information and that should be all right. That should give the same result if the sequence was reversed. So it is essential that union function, which is a merge function, is associative and commutative because we don't want to care about the order. We want to be order independent. Same thing for idempotency. So if these operations are performed multiple number of times, then why will that be a case? This is a cluster with not a reliable delivery, which means some messages will be dropped, so there won't be acknowledgment, so they will be tried again, so there will be duplicates. And set union has a property that duplicates does not matter, which is called idempotency. So using these three properties, we can ensure that these sets will eventually converge without coordination will converge. So that's the inclusion of CRD. CRDs have few types, as you heard in the morning. That's not the focus, but I'll just, for the bigger context, I will mention that and then we'll move on to the other example. So the first type of CRDTs are commutative. CMRDTs, they are common. They are operations based, which means that the operations, chain operations, which happen on the daily basis, only the operations are propagated to different operations. They are a little bit less constraining than what we saw, because they do not require the idempotency, which means you can, so they do not require outputters, but on the other hand, they require the reliable delivery, which means that it's essential that you get a message exactly one. But, right, so you can get the message in any order, that is fine, but you need to get exactly one that because you are going to get the message exactly once, you don't have to apply that again and again, which means if your operation is not idempotent, that is fine. But your operation still needs to be commutative because you can get them in any order. So it's slightly less constraining to have a data structure, and this kind of data types have a very important role in building the event-sourced system, where you want a bank account to be available in the network partition across two data centers, but then it should be able to merge. And you still want meaningful operations to have not a very convincing operation, like just an added person. But there are other types of CRDDs. They're called CVRDs or version CRDs. They're state-based. Every time that is updated, the full state of the data is basically transferred. And they're constraining because they require all three properties that we saw earlier, because they do not enforce a reliable delivery. They do not enforce on the protocol a reliable delivery, and that's why they work very well with the gossip protocol. And then we are talking of akka question, which is based on the gossip protocol. So the data types that we are going to talk about are basically CVRDs, which are part of akka library called as distributed data. One small thing, even though you pass the entire state in service registry, your services will be few hundred, so the state is not there. On top of that, there is an optimization called delta CRDDs, which tries to basically pass the minimum information, only the changed information, change state to the other news. And that support is available here in the data. Now this is just a list of CRDs which are available in the library like counters, sets. The first three are the primitives. Set is easy to explain. Let's go to LWW register. LWW last write-up register, which well, there is a decision to be made, which if you are updating the same value, then the decision will be made based on two writes later, and that value will replace that. In case of service discovery, if you want exactly one service to be associated with the name, in some cases, you want multiple instances to be associated with the name, which is a usual case for the doctor. Then you can do the set against the name. But in our case, telescope we want exactly one instance of a hard name to be available, so CRD is the last one. The last one is a composite data type. So CRD is actually composed because of this algebraic property. And what is possible is that you can have a CRDT map where key is a key, a string, or some value, but the value of the map would itself be a CRDT. And if you do that, then you can actually create the whole map and give it the same characteristic like the underlying one. That's a very nice property because of the algebraic property. So we are going to use one of that, which is LWW map. The values are registered. Now the question is, why are we using two and not just one CRDT? We'll come to that. So we have two very distinct requirements. One requirement is that I should be able to watch a particular service against the name and get all the event, not only when it comes up, but when it changes the code and stuff like that. And that is supported currently in Akka only at a CRDT level. So I cannot say that of this whole map, I'm interested only in one key that is not supported. So I have to keep a register separate as a top-level entity. But then there is another requirement that at any given point in time, I should be able to query and find out and list all the services in my system or maybe filter them. Well, if I'm just holding all the top-level values, I have no way to do that. So I need to put them inside another data structure, so that is the constraint that we are working with right now. So we have to leave it to CRD. Obviously, each add-and-remove operation of a service will involve changes to both the CRD. So before we go to the implementation, let's see how it looks in the bigger picture. So these are the three nodes that we are talking about, which are forming different machines. Each one has a JVM inside. JVM inside of a telescope-related software. It's not required that each machine has a single JVM. They can have one. So they are going to have the real deformation. But then each JVM will participate in the implementation. It's unlike consumer. On top of that, we have this location service as our library. We are doing this library. On top of Aka Cluster, it forms a ring basically, there is a way to discover this cluster. We will not go there, but we will talk in the last slide. And then on top of it, you have a CRDT copy of your location directory. So basically, registry is replicated on all this. But interesting thing is that all of this is within the same JVM problem. If you take this simple picture, this is one JVM problem within which the Aka Cluster and also a registry, which is a in-memory copy of the registry that is available to you. And that has very interesting. So now let's see how this registry is accessed. Well, we said that we have to support ActiveRef, PCPN, HDD. So let's talk on ActiveRef first. Assume that the first machine is going to have three different Actors systems having N number of Actors. Each one is the name. They can directly call the CRDT with API that you have provided and give you the demo and do the in-process call. So it's like an application. I'm building an application.  I make a call to register myself or discover something. So that's important to keep in mind. That is also true if you are able to create the JVM-based HTTP server for example. We are using Akka HTTP. So we create the Akka HTTP instance all the main methods from there. But after it is started, we register it to the location. Again, it is a local copy. That will not be true if I'm starting Redis. Because Redis is an external processor. So to bridge that, you have to create an agent, JVM-based agent that will run within the process. And on behalf of Redis, you have to register and keep track of the process. So that is the utility that we are creating. So that's a full picture of how internal it works. And based on the discussion so far we can see or discuss some advantages we can debate. So the first one is that we are required to use Akka. That's the architectural choice. So on top of that if we had to work for anything else we would have to have another setup. For example, we require a setup of a consumer as well as they work around. Plus, what we get is a in-process library. Which is not possible in any of the approaches that we saw so far. They were external. So g-r2, g-r2, g-r5, g-r5 90% of our services are actors. And these actors have a very special way of doing serialization deep simulation. It is as simple as string serialization deep simulation because they have a string besides it. And that is available out of the box if you are using Akka and Akka is good at data. So we don't have to write any custom acoustic simulation for the actor. That was a good benefit. Each of the actors whether it is an R-Lang or Akka can be watched for the death. And if there is a death then I get an event. Well based on that event in the cluster can be unregistered. Which is a death watch and then doing a house duty call. It is possible because each actor has to watch a movie. That is like a edit event. It may be constant, but this is direct. And because the replicated copy of CRDT is present on all the instances the most common operation that we do which is a lookup is basically very, very far. It's like an in-memory call. So when we lookup we have... So CRDT is allowing you to do consistency for example. It doesn't matter if it is like scale data because then my call will fail and that is fine. It is to an extent that each call that we make to remote goes first with a lookup and then call. And that is a no cost to us. So that pattern we could not be doing because it's going to take few milliseconds 200 milliseconds. It will cache it for a while and then it will... And the last is that the watch operation subscription is a silly push bit. There is no long pulling involved. I just register my interest and the whole actually this is not even there. So I keep getting the update if the service goes off without having to keep the connection dedicated to it. But there are challenges and that is first challenge arises because we are using two CRDT. And there is no guarantee of atomicity across the CRDT update. Which means I am successful in updating the first one and then I fail for the second one. It's a partial success. There is no way to have it more than that. So what do we do? Well, the work around was simpler for us because we just error out. We return error response to the client which is a hint that I try again. And when client tries again that is perfectly fine because registering operation is important. So from the same post you can register in number of times. So, which means that they can start for the 10th attempt they will be successful. Well, it's not foolproof if someone wants to analyze user and trying to break the system. But given that it's an internal application I think that's a good enough correct. And in future, if a CRDT allows us to watch events only for a given key in a map then maybe we don't need to wait just to do it. The second thing is the whole thing is tied to the cracker. Which means you have to know how to do trap a cracker which any way is possible.  There is no way to know whether it is unreachable temporarily or it is really down. Well, you can know as an external party but as a member of a cluster you can never be sure because you never know whether it is because of a network partition or this is genuinely down. And if you try to make your own decision in the cluster then it can give right to split brain scenario which is common on all such non-marketing problems. So the usual recommendation is that avoid making mechanisms of calling something down. But we did not listen to that advice and what we do is actually we just if for 10 seconds the member is not accessible we call it down.  But that is not the main reason. The main reason is that if there is a network partition this is a telescope system. There are much bigger problems to solve on a level and it will not be functional just because our software is not working is the least of the problem solution. It is very unusual to go with automatic downing but we have found that. So one of the challenges then I have a small demo. How much time do we have? So I am going to talk of registration registration and then subscription which means that let's create two applications So I am going to start these applications I am going to start these applications So the service what is it doing? It is just subscribing to a name and watching basically events of registration and then service I am basically this is how the API looks you get a handle in memory just like the API calls it so this is all the local they are not doing and then what I do in the other process is that first I register an actor called A after some time I register it and then after some time I register on the same name I register an actor called B so I should see three events so the first one is a listener and the second one is actually doing the job so once they will form the cluster I have disabled the log and now they start seeing the first second so this is a simple demo slightly more interesting demo looking at 100 of them just to give you a feel of the performance for example I am watching instead of one name I am watching 100 and unregistering against one name I am registering and unregistering against 100 to give you a visual impression I am introducing some delay there but that is for the demo so how would this look like you can see that in less than 200 seconds the batches of updates will be transmitted across the cluster so you would not even notice the difference well it is not real time CRGD updates are propagated in batches of but that is completely accurate so to go to the statistics and this is registering right now and after it is done and you can do this comparatively it is CRGD updates so about the team I think some team members are here we are doing main development but the project of is describing the requirement of the language is the Government of India channel and these are the links for it is an open source work so whatever we have done is on GitHub it is also very documented because we are the live view provider and some other device so we are trying to document it and location service is just one of the services we are building we are building a lot of things backed by SVN so DNS service can provide you service discovery via DNS DNS 2782 records so SRV records so it is a standard you can say in many of the service discovery mechanisms trying to provide a standard way if we had one with concern maybe we would have but we could use this as a lab way if you need to for example we extract this as a general purpose service discovery then probably we if we want our library to be used by any project then we have to come up with some standard that is the question console also provides something called network tomography how many instances of your service any given in service do you have exactly so then you probably won't need that so much because network tomography gives you between nodes which is part of the service protocol so I think I very cleverly just talked of ad on the set because it is easy to explain because Vivo is not what the world but I will give you as the artist of the this is ad this is a notion of food stones it is a very technical I won't pull your chance but I will give you an intuition because those counters are for example instead of having just one set if you have two sets one where you keep on the addition one where you keep and then whenever you want to list then you have to do a taper and give you that if we give you some of that one well it will be not usable because then once entry is removed it can never be added so it is not usable but you get the picture of the mechanism to come up with very smart ways of implementing that and still giving you the guarantee it is our register sitting as a value inside a map so any operation that we do on that map is atomic that is fine but it so happens that we have to keep that register outside also just for the tracking need and then we are actually dealing with and the moment we wrap into a new one I won't be able to track just the one value that is the limitation of the implementation right now but yes once we compose them then that will solve the problem Thank you for the talk you are using a last register and map how do you determine what is the last value if there are two concrete updates to the same element how do you determine what is the last It is not ideal if you just think from there it is like very crude right but the time stamps are not reliable but then imagine that in our use case what is going to happen is one SCD by design will be started on so we expect that no other machine is trying to start that and now assume that by some mistake two machines are trying to start the same device jam any which wins we don't care will you because they are studying the same product if they do mischief if they say that against Galeel I am giving you a device driver and then someone is giving me against Galeel giving me something else that is a mischief and that can happen but then we are okay with that but yes it is a good question in general last night that too based on the time stamp is not a good strategy if you want to bank things are going more distributed CRD joins they join officially itself giving this kind of library they are also giving we are using we did not write our CRD service I thought that you combine both your stuff so we use the CRD library which Akka avoids and then we implemented a service discovery API on top of it and our talk is about how cool it is so yeah but Akka is also giving a more service like our opening service because they are also having node failing stuff they are working on it if you want the closest analogies the question is that is Akka cluster using the same mechanism for doing service discovery then that's a very interesting question because Akka type if you are following the type Akka type Akka type doesn't have so far a notion of a remote actor so which means that it has local actor or it has a clustered actor which means to discover a local actor well have a reference to discover a remote or a clustered actor so any mechanism is service discovery and they are providing a service discovery mechanism how inbuilt at part of Akka type but it will work only with actor actor and that mechanism they call receptionist so if you just look at Akka type receptionist you will see CRD is being used for a very similar reason and it came much later after that and there is another so we heard from Lightbridge that Akka type