Hi everyone! Welcome back to the Red Dot RubyConf 2016 Hey So once again, I'm Nat. A lot of people can't remember my name and I'm so sad So for those of you who were wondering yesterday, I actually also work at pivotal as a software engineer Okay, and next Thank you for adhering to our code of conduct Please stay nice. We're very happy that you guys are so awesome Next, Wi-Fi. If some of you have problems with Wi-Fi, please I suggest for getting the network and then rejoining it Next up would be our drop spot So I hope all of you have taken a look at the drop spot. If you're hiring, please don't forget to attach yourself there Oh, and I also found something cute Yeah, what are you doing here? It's a new Java Next up the appreciation bot. So Our organizer Winston, he has mentioned that he doesn't need any tanking any more tanking. So We should also probably just use this bot to tank the jam creators and open source contributors like We use bundler almost every day and it will be we use factical for our testing and use device and the list goes on so I'm sure you have some people You might have at the edge of your mind that you want to thank So I urge you to go there and put their names up At the end of the day, the organizers will go through each of them and tweet each of them how much we appreciate them And you know since most of us live off internet points, so please keep us alive Yeah, if I could put my cheek heads up there, I would but they don't have to turn and they don't write gems And also I would like to thank all of you here who turned up thoroughly in the morning. Thank you very much Lastly if you want to contact us or you want to post anything interesting, please tweet us at red dot RubyConf Lastly I would like to thank our sponsors again, and I believe pusher Has something to say Is bannon Still outside Okay, I will probably get him in after After I'm in stock Okay, okay. Oh, I guess I'm using this mic. Hello. Hello. Hello. Good morning. Good morning. Happy Friday First I want to say thank you to everyone. Thank you to the organizers Thank you all of you for being here today, even though I know it is the morning of the Second day, which makes it a lot tougher to get here So thank you for being here It is a great honor to be here. I think this is my fourth year here I think and I love coming every time. So thank you Today I'm gonna talk about Taking out the trash This is the title of my talk first. I'm gonna introduce myself a little bit. My name is Aaron Patterson I'm also known as tender love if you don't recognize me. This is what I look like on the internet So my avatar looks a little bit different than me, but that that really is me So I work for a company called the github You may have heard of them. I'm not sure I've been working for the company for about three three months now and it is the first legit company I've ever worked for So like I said, I've been I've been working for the company for about for about three months and it was really weird I had to go down to I live in Seattle I had to go to San Francisco for a week to do like training or whatever and it turns out that at the company everybody refers to each other by their github nicknames and Like in person And this has never happened to me before all these people were calling me tender love and I just thought it was really awkward So I mean if you if you want to call me tender love you can but you know, please call me Aaron That's fine. That's fine, too. I'm happy with my I'm happy with my real name So I should have called this I should have called this talk cats puns and Friday hugs because it is Friday And I love puns and cats so I'm gonna show you I'm gonna show you some of my cats is one of my cats This is choo-choo her her Her tongue is too long for her mouth. So it always it always sticks out. This is a more famous one This is Gorbachev puff puff thunder horse My wife likes him better than the other cat This is he's trying to hide in this photo here like he thinks he is hiding he is not This is choo-choo again. I really I really like her She likes to sit on my desk while I'm programming and she always makes this face Which is like the same face that I make when I'm programming Anyway, so, you know, I I also love cats and code just like our emcee So I want to go over I want to go over a few things that I've learned so far at the conference So first off I learned about some new ruby features We were taught like Matt's was talking about typing and ruby and we learned about gradual typing And I want to so I I want to demonstrate it here today. I have a live demo for all of you So I'm gonna demonstrate gradual typing. Okay. You ready? I did it on stage and now soft typing soft typing Anyway, this is a mechanical keyboard. So if you want to talk about mechanical keyboards, you should come talk to me later Okay, also, I learned that we don't care what what Dynamic typing. Yes dynamic typing. This is very dynamic. I was also gonna do static typing where I don't move my hand the keyboard Anyway, so we don't we don't care about small things, which is why in Ruby 2.5. We're gonna remove the downcase method. So no more no more small things Also, I thought it was like I thought it I was watching the presentation I thought it was really cool that we're gonna have like upcase and downcase and they're gonna support UTF-8 characters. So we have upcase and downcase But I was thinking about this. I'm like, well, we've got up and down We don't have we don't have widen and and narrow in that doesn't like we're missing those So I I put together an implementation of that. So you can call widen and narrow and so look look forward to this in Ruby 2.5 If you want to get this Functionality today go ahead to my github repository. You can get this jam and have those I also learned that there are no no durians allowed at this Conference, but that is too bad because I love durians. So I will eat them anyway and Sorry, I'm trying to work in a durian pun somehow. I Appreciate all of you for enduring my puns. We also learned about teaching people to fish. I'm going to teach you all how to fish Go ahead and look in your email today all of you you have an email from me Just enter your bank account information So also I learned I learned from today At this conference I learned from Terrence that learning a new language can keep me from getting rusty So that was an important thing I learned from his talk today All right, so let's let's move on to the real topic of conversation I'm gonna talk about GC today. I'm gonna talk about GC specifically. I'm gonna talk about Godfrey Chan So I asked him I asked him, you know, what is your what is your favorite memory and he said that it has been collected? His favorite food is pizza and I asked him on the bus this morning. What is your favorite color and he said that it is ruby? Actually said it was orange, but I fixed that for him Okay, no, no, no, no, I'm I'm I'm actually gonna talk about the garbage collector today And this is this is a very exciting topic and I'm so happy for all of you to be here at at 930 in the morning to listen to talk about a garbage collector You know get ready to have a nap this morning. It's okay. You can sleep in so we're gonna talk about I'm gonna talk about the GC and MRI I'm gonna talk about GC algorithms, but I'm gonna talk about them specifically With regard to the algorithm algorithms that we use in MRI So I'm gonna talk about some some GC algorithms, but I'm gonna focus heavily on the algorithm algorithms we use in MRIs GC, but I think I'm also gonna talk about two sides of the garbage collector I think it's interesting when you see people give talks about the garbage collector. They typically talk about Collecting collecting memory, but they don't really talk so much about allocating memories So I'm actually gonna cover both sides of that the GC is the GC is Responsible not only for collecting memory, but also for or also for allocating it. So we're gonna cover that So we'll look at collection algorithms how Ruby actually reclaims memory And then we're also gonna look at allocation Algorithms how we actually create objects in Ruby And I'm also gonna touch a little bit if we have time I'm gonna touch a little bit on the introspection API like the GC introspection API and MRI so you can actually get statistics about Objects in your system So I'm gonna talk first about the collection algorithms in MRIs GC So let's let's describe a little bit what What type of collector MRI has the type of collector we can we can describe it as? Mark and sweep collector it's Generational and it is also incremental so we can use these three three adjectives to describe the garbage collector So we're gonna cover each of those a little bit more in depth But first I want to step back a little bit and talk about you know what what exactly is a GC So what you know we know that it's it's it's a thing that that frees up memory for us But at a high level, what is that? What does that really mean? If you think about the objects that are in your system So if you if you look at the objects when you you're creating objects in Ruby You'll notice that they form sort of a tree thing here So for example if you look at this Ruby code on the left You'll notice that it kind of forms a tree where we start at the at the root And the root references an array which is that that a variable and then that array references a hash and the hash References a key in a value pair so we can think of it as kind of a tree data structure so when you're creating these objects in Ruby you're creating kind of a tree a tree data structure and Let's say let's say we change this code a little bit and we assigned a nil into a We actually cut that line up there with the root Now the root is no no longer holding a reference to that array and hash Etc. So this these objects are going to get collected. So Essentially what our garbage collector is doing is saying, okay We're gonna find everything in this tree that isn't referenced from the root anymore And we're going to we're going to free that memory up So we can think of all of our objects as a tree and we can think of the GC as something Is trying to find find things that are not available in that tree anymore. So Few terms that we can learn from this or we have a we have what's called a root set Which is our vet that very top root root thing we have garbage objects Those are ones that are no longer reachable from the root And then we have live data which is all the all the objects that you can actually reach from the root in that Tree so our GC's entire job the entire job of the GC is to find those unlinked nodes and Then free those up. So when we talk about GC algorithm algorithms All we're really doing is talking about how do we find those? How do we find those unlinked nodes, right? How do we get how do we get those what is an algorithm for finding those and releasing those nodes? So the first thing we're going to talk about is Mark and sweep. This is a very simple GC algorithm and it has two distinct phases Mark and sweep Which is why it's called Mark and sweep garbage collector So if you look at the if we look at this, let's pretend we have an object graph that looks like this What happens is we go through a mark phase and the mark the mark phase goes through walks every one of these arrows and Actually marks that object is something that we care about so it starts at the root and follows each of those arrows through the tree So you'll see that it'll follow through a It'll follow through all these all these arrows until it marks all the objects and then we go through a sweep phase and What happens in the sweep phase is any objects that we're not marked. We actually just free up those objects. They go away And we're left with we're left with the actual live objects And at the end of the sweep phase we go back through all those objects and unmark them all So now we're back where we started and we can continue on with the program So Right now the mark mark flag has been cleared. So the mark and sweep algorithm is very it's very easy It's a simple algorithm But we have some problems with it. It's a little bit too slow We have to actually walk every one of those arrows every single time look at every every object mark Whether or not it's alive and then sweep through the whole thing and another problem is that we have to actually stop the world when we When we do this collection, you may have noticed in older versions of Ruby if you're especially if you're running a long process You'll notice like it'll just stop for a second and then do you're like, oh, it's just pausing Maybe my iTunes is playing too much or something. No, it's not your iTunes. It's fine It's actually the garbage collector just pausing doing the mark and sweep phase and then continuing on So the other problem is that we have to visit objects every single time We have to visit every one of the objects in the heap and mark all of those every single time and this is not very fun So We have to walk every object every time So this one way we can get around this is if we introduce what's called a generational garbage collector And the idea behind the generational garbage collector the theory behind it is that objects in your system will die young So typically typically objects they're going you're gonna allocate them and then they'll just go away very quickly So the idea is that if we divide objects up into old and new Objects so if we have old objects and new objects then maybe let's only look at the only look at the young objects only look at the New objects so the way this Generational collector works is say we have a say we have a object graph that looks something like this We do the same Mark and sweep phase where we say okay. Well, we're gonna mark here. We mark B and D and then We sweep A and C So those go away, but once that happens then we take we take B and D and we move them Generations they go into the first generation right now Let's say the program continues on and we allocate some new objects those new objects are gonna get allocated in the young Generation or generation zero Then we go through the same mark and sweep phase so we mark F and G. We also mark D Then we know that E goes away and then F and G get moved into generation one Oh go transition. There you go And then we do the unmark step again or we clear that mark bit now the interesting thing that You may or may have not may or may not have noticed is that when we're doing that mark phase when we walk We're walking those arrows. We actually didn't have to touch the B node We didn't go to be because we knew that that one was old So we're not dealing with B. So the nice thing about this is that we don't actually walk those old objects every single time So we can speed up that mark and sweep phase because we're only we're only looking at old objects once in a while We only consider new objects frequently, but there is one slight problem with this with this Algorithm and that's that let's say we have let's say we have a graph It's like this B and D are in the old generation and for some reason something happens such that that D actually Allocates a new object E Right that new that new E object is in the young generation And when it's time to do the time to go through our mark and sweep phase We don't actually walk that arrow up there because we know B is old Right. We know B is old So we we don't go through that go through all those arrows and what happens is at the end of the mark phase We say well, you know E is E hasn't been marked So we're just gonna GC that and now we've got this we've got this bad connection here So so your program will crash. We've we've freed up an object that actually is being used. It's actually live so The way that we fix that is we introduce something that's called a right barrier a right barrier and a remembered set And what a right barrier is is when we write that when we write that connection between D and E When that that arrow gets written we keep track of that we keep track of that connection inside of what's called a remembered set So when that arrow gets written we remember that we put it inside a special set so that we know when we do our mark and sweep phase We say oh well, we're gonna go look at that remembered set, too We need to mark E and then E goes into the old generation and we're Free of our we don't have this bug anymore So we have a couple new terms from this we have we have a right barrier Which is just keeping track of those interesting what we call interesting arrows Which would be old generation to new generation arrows and we have a remembered set Which is that list of those interesting interesting arrows, so? Generational garbage collectors are faster It's it's faster because we don't have to walk as many nodes as frequently, but it's not quite as easy We have to implement this right barrier, so it introduces some right barrier and a remembered set so it introduces some some complexity to our garbage collector now unfortunately We still have to stop the world with this with this algorithm even though we're walking fewer nodes We still have to stop go through that go through that cycle and then continue on So to get around this problem We've introduced what's called an incremental garbage collector and an incremental garbage collector is what it says It's something that you can do incrementally and we'll look at how that works the way this algorithm works is that it uses an algorithm called tricolor marking where you have three colors These names are so imaginative I love it all right, so we have three we have three colors white black and gray now White those those objects are objects that we're going to collect Black objects there they have no references to white objects, but they're reachable from the root and then gray objects are Reachable from the root, but they haven't been scanned yet, so we'll look at we'll look at how all these these colors work together So the way this algorithm works is we pick we pick an object from the gray set and then we move it to black We change that over to the black set now for each objects each object that that one references We move those into the gray set as well, and then we just repeat step one and two Until the gray set is empty, and then we know that everything that's left over that's white those can be those can be garbage collected so An example of tricolor marking Let's say we have a we have an object graph that looks something like this We start out with a and f in the gray set because a reference from the root We color those we color those black then all of their references go to gray Then we color those black and now we know that we're done We don't have any more objects inside of the gray set So we can take all the white objects and GC those we can free those up. So those go away now The advantage of this algorithm is that we can interrupt any one of those steps So we can we can take a break at any time So we do we do the gray to black coloring and we can just stop right there We can continue on with the program. So what this means is we can perform each of these steps incrementally, which is why it's called an incremental Garbage collector. So what this means is that our halting time is actually reduced what we what we do is we say, okay? We'll run the program for a bit now We're going to do one step of this incremental GC and then we're going to continue on with the program so Our program halts for less time But there's also a problem with tricolor marking and that's very similar very similar to the Generational problem, which is let's say we have these these gray colors here now We let's say we halted right right now. We're gonna. We're gonna pause the program and We do this we do the first step we pause the program We let the user code continue so the GC pauses and then your code continues and somehow F This F object creates a new object. It allocates something new so it's pointing at a G object now We have this new object now Let's say the GC starts up again, and it says okay well now We're gonna take all the gray objects, and we're gonna color those black So it colors all the gray objects black, and it says okay. We don't have any more gray objects So it's time to GC so we collect that G object, and now we have a bad reference again That G object should have been live, but it is no longer live So in order to fix this problem We actually introduce another right barrier, so we say okay when when we write from F to G We're actually gonna save that off, so we have another right barrier and a remembered set so we do exactly the same trick we were doing earlier we Just keep track of that, and then we know that we can color that one gray later, so Interesting things from this the glossary to learn from this is we have Incremental GC which means that we can halt the GC at particular times. We have a right barrier Again, which keeps track of rights to objects, and then we have a remembered set which is those interesting interesting connections Now all these algorithms the idea behind all these is that we want to actually minimize Tracing so we want to minimize the number of arrows that we traverse We're gonna reduce those as much as possible, and we also want to decrease halting So that's those are our biggest things We want to reduce the number of arrows we walk because that's less work We have to do and we want to decrease halting because if the program halts It means it can't be doing other things like servicing your servicing requests from a user or something now Things that our GC is not which I want to talk a little bit about Our GC is not parallel it means we're we're not able to run this mark-and-sweep algorithm in another thread We can't do that. We can't do it in parallel It's also not real-time Which means we can't we can't run MRI on things like That require real-time feedback for example like I don't know Maybe you're building a robot or something you might you might not want to run MRI on that You might want to use something like mruby or something that has a more real-time garbage collector The other thing is that it's not compacting We don't actually take these objects and move them around and I'm gonna talk about that a little bit a little bit later Those objects when we allocate them they stay in one particular place in memory. So we'll we'll look at that When we talk about allocations All right, so we've covered we've covered the way that we do Object freeing in MRI now. I kind of want to look at allocation algorithms and The thing I want to look at first is I want to look at our heap layout So and we have a heap our heap contains all of our objects now When a Ruby object is allocated we don't actually call Malik every single time We we actually allocate that into a page and the reason that we do that is that Malik isn't free You're killing me I know it's early, but please I'm so proud of this one Alright, so thank you. Thank you All right, so Malik is it Malik isn't free when we when you call Malik at cost time So what we try to do is we want to we try to allocate one large chunk of memory We allocate a large chunk or we call it a page or a slab I prefer to call it a slab though I think we it's referred to in the code as a page and The reason I like to call it a slab. I'll get into a little bit later Now we allocate one large chunk of contiguous memory this page memory is contiguous so and each one of these chunks of memory contains a linked list and Yes, I should advance the slide so each page holds a linked list now This linked list each member of the linked list is what we call a slot so nodes in this linked lists are called slots and Though each slot is actually a Ruby object, so to make this a little bit more clear When we do an allocation will allocate a page and it is just a big chunk of memory like this and When we when the garbage collector allocates an object It'll just put one of the objects inside of the page So each time we allocate an object we get a new object one after another oops Yes, there we go. So we have a linked list inside that's stored inside of this page This list we call it the free list So all we have to do to allocate a new object as we find the next slot That's open in the free list and then hand that back to you as your Ruby object. So If we actually fill up a page, let's say this page is filled with objects when we go to the free list We see oh, it's filled So let's just allocate a new page and now we have a bunch of other places to store objects now Each one of these pages Actually one thing I want to talk about is when we're when we're looking at the top of that the top of this link list Where we actually pull that pull that object out when it's time to allocate an object. We call that our Eden Our Eden so that's where objects are born So when an object gets garbage collected it's actually pulled out it's pulled out from this list So we free it from that we free it from that list and that actually leaves a hole in the page Which I want to talk about a little bit later, but let's say we let's say we actually Free up all of these objects all these objects get gc'd if that happens then we'll actually free up this page So the page will go away so we can free that Oh, I want to talk about this. This is exciting. I love this part. So I want to talk about some interesting allocation hacks now Not every object requires allocation. So Matt's talked a little bit about this yesterday with integers Now not every object and you may or may not know this but not every object in ruby actually requires us to pull something off of that off of that page So I'm gonna try and walk through how that actually works here. So one page one page in ruby is about 16k This is just a size. It's a fixed number at the 16 16k page size now one object in ruby is 40 bytes Okay, so we got a 16k page and we got 40 by 40 by objects now pages are actually aligned there And what this means is that we say instead of malloc'ing the pages We actually do what's called an aligned malloc and what this means is we say we ask the computer Okay, give me a chunk of memory, but I want the beginning address of that chunk of memory to be divisible by some number Okay, so the address of that chunk of memory is going to be divisible by some number and we choose 40 So we choose 40 as our multiple now 40 happens to be the size of a ruby object Now what's cool is if you start at 40 and you print out Print out the binary representation of that number So we're gonna say alright 40 times 1 40 times 2 40 times 3 Etc as if we're walking through the size of those objects, so we're adding we keep adding an object to that list If you look at the binary values of those now if I I'm gonna just show you this here These are the binary binary representations of those numbers if you look at the last three digits You'll notice that those last three digits are always zero Okay They're always zero So what this means is we can actually use those last three bits to have add some meaning to the number So we'll say like let's use those let's use those last three bits to Indicate that that number is something special So What we can do is we can represent we can use that bit to represent integers without doing any allocation So for example, let's say we have the number let's say we have the number two And we want to represent that as a ruby we want to represent that in ruby What we would do is say well, let's let's have a flag of one for example And if we convert to to binary we know that that is one zero And if we shift that over Then we get one hundred and we add the int flag to that and now we have one zero one And if you look at that number Because those last three digits are not zero. We know that that was not allocated in ruby's gc If an object is allocated in ruby's garbage collector, it's always going to end in three zeros And since this one doesn't end in three zeros. We know that it's something special So what we can do is just take that number two encode it like this and treat it as a ruby object And then to decode we simply just shift that over shift that over one So using this information we can actually figure out. What is the largest Fix num that we can allocate without actually without actually pulling anything from the garbage collector So for example, if we do two to the 64 minus one You'll see that's what the binary representation looks like that's 64 64 ones But if we subtract one from that we get a big num class And that's because you know, we had to shift that over one. We lose one bit from that shifting So we we can't represent 63 bits And if we if we do 63 We still get a big num and the reason we get a big num is because the very first bit represents plus or minus Right, so if we actually do two to the 64 Then we get a fixed num So we can see that that's actually the largest fixed num that we can represent Now even if you add one to that you'll still get back a big num Now unfortunately, this is before ruby 2.4. If you go look at ruby 2.4. It's much more confusing. You can't tell the difference They're all just integers unfortunately, but But if you look at the object IDs, you'll notice they are very very interesting those top two ones there Those top two ones have very Big object IDs and it's because we actually use the memory address The address of the object as the as the id So you can tell those two objects at the top those two are Not allocated on in ruby's gc and the bottom two ones are And this technique this technique of using those Three bits those three remaining bits. We call those tag pointers So it's a pointer. It's some number stored in memory But that number has special bits in it. So I I love this hack. I think it's awesome. It's really Clever in my opinion now if you look at uh, if you look at the other objects in ruby many many of these objects Are tag pointers fixed nums or tag pointers? Of course, we don't have fixed nums in 2.4 Floats or tag pointer is true false nil symbols. So a lot of these things don't actually require Allocation from the garbage collector so I want to move on a little bit to talk about some allocation problems and these are these are problems that We've been seeing in our application Now one one problem that that the garbage collector has is we have an issue with poor reclamation So let's say we have let's say we have three pages that look like this They're full. They're filled with objects, right now Let's say the garbage collector runs and we and we reclaim some of these objects. So they go away, right? We've filled we've reclaimed some of those objects now Unfortunately, I mean what would be cool is if we could take these objects and move them Move them around like this then we could actually take this page and free it you go away Now unfortunately, we can't do that that does not happen in that does not happen in MRI We cannot move these objects So what ends up happening is we have these three pages that are allocated with holes in them And we can't reclaim that memory So unfortunately, this causes this causes some problems for us where we have we have Heaps that are too large. They could be smaller, but they're not Uh, the other issue is that we have copy-on-write We've experienced copy-on-write problems and where this comes from is I'm going to explain how these copy-on-write issues occur now. Unfortunately one A ruby memory page is not an os memory page Which is why I like to call those pages in MRI slabs So an os page is something totally different It's a page of memory, but it's not the same size as a ruby page So one ruby page is about 16k on os 10 and os page is about 4k I think on uh, the linux systems we have in production too. They're one os page is about 4k So, uh, one ruby page contains four os pages Now let's say we have a parent process and a child process Parent process fork to have a child process and they're both pointing at something that looks like this So they're they're both pointing at a ruby page that looks like this Now let's say the child process allocates an object sticks it in there now What happens is when we write to that bit of memory the operating system has to copy it So it's copy-on-write. We wrote to the memory. It has to copy it Now unfortunately the the operating system doesn't copy just that that little section It actually copies one os page So rather than copying those 40 bytes, we actually we actually copy 4k So we wrote 40 bytes, but we actually got 4k 4k copy So what would be cool is This is something i'm trying to work on and test Test on our application is if we could group old objects together such that we didn't have as many holes in our pages So for example, let's say we had let's say we had two types of pages One type is a probably old page and another type is just a regular a regular page Now What we could do is if we allocate objects at the time we allocate them if we know hey I think you're going to be old we stick that into the old page Then maybe we can group all those together so they don't get we don't get holes in them like that So we'll allocate an old object. We know it's old when we allocate objects that are unknown Then we just stick them into the normal pages So you might be thinking Aaron How do you know what object is going to be old? It seems hard. It is not that hard If you think about classes and modules in ruby all these things are objects And they're probably not going to get garbage-elected when you do a class foo It's probably going to be become an old object same with modules or constants or possibly frozen strings So we know we can have some heuristics to determine like hey when we allocate this object Let's put it into the old page Uh The other thing I was thinking about doing and I'm not I'm Just testing this now is if we could statistically determine which objects are going to be old and which ones aren't So let's say we have some foo object and 90 of the time we allocate a foo object. It becomes old Then the next time we allocate a foo object. Let's just stick that into the old page Uh Though unfortunately most objects becoming an old object is a rare thing So i'm not sure if this technique will actually work for us But that's why i'm just trying to implement it and see what happens in production Uh, so this will help us efficiently use space So if we we can group all these old objects together then maybe we can reduce those reduce those empty slots without having to move objects We could also reduce gc time with this with this technique because if we know that those objects are going to be old And we mark them old already then we don't have to go through those first two cycles of of Garbage collection in order to mark them as old All right So let's let's take a look at a little bit of some gc introspection tools that we have available to us We use we use a lot of these tools to debug object leaks and Just various gc issues in production. So MRI has a lot of gc introspection tools. So for example, we can look at gc info Now the way to look at gc info is just with the gc dot stat method and you don't need to read all these Just go run that method But one of the main reasons I went through talking about all these all these gc algorithms and pointing out the Glossary and those those particular terms is because each of those keys in this hash corresponds to That particular thing with the garbage collector. So if you know how the gc works when you read all these keys It's very easy to understand exactly what the gc is doing So Now that we've talked about all these techniques go run this run this code Take a look at all the keys and you should be able to determine what each of those keys means Uh, we can also get gc performance So sometimes we'll look at uh the performance of our garbage collector and that's really easy with just a gc profiler It looks the code looks just like this. You can just enable the profiler and check reports for it You can see how long each gc took Uh, so just call this method and then uh, call the report method to take a look at take a look at, uh The performance of the garbage collector We also use a lot of heap introspection So we'll look at what objects are actually allocated and my favorite tool for using this is object space dump all And what this does is it takes your entire heap and actually dumps that out as a as a json file And if you have a lot of objects, this can take some time So, uh, be careful when you use this when you use this tool, but this is one of my favorites to use You can actually dump one this dumps out your entire heap, but you can actually dump one object at a time if you want to You can just use object space dump to actually dump out one object You'll see right there. That's just the json json representation of that And what's neat is if you use this if you use this uh dump method, you can actually see See what happens to the object as you gc. So let's say we do gc.start three times Uh, you'll see that that object down at the very bottom After the three garbage collections It becomes old and uncollectable and marked So you can see you can see how these objects change over time, which I think is very cool Also, not not all objects in ruby have a right barrier and you'll see when you dump out a particular object You can see which objects do and don't have a right barrier Uh, so in ruby objects have three generations as we saw from this Uh, the last one is the third. We don't consider an object to be old until it makes that third generation Uh, so another another thing I really like to use when debugging when debugging Allocation issues is object space go check out all the methods on that that class especially Trace object allocations. I like to use this one. I like to use this one a lot If you call this if you look at this, uh, look at this method, you can actually turn on Tracing for objects so that every time an object gets allocated it remembers Where it was allocated what file what line So if you're having troubles determining where the some object came from Enable this flag and then you can ask it. Hey, where were you allocated? Um, hopefully am I I am out of time. I I don't know yet. I think so. Am I am I okay. Thank you. Thank you so much. I Oh Please use this hashtag If you enjoyed the presentation also, I have stickers of my cat. So come say hi to me I know I'm not supposed to have stickers. Apparently that's not cool anymore, but I do So come say hello find me later. Yes. Thank you. Thank you Questions from the audience. I know it's early, but you can do it Hello, yes, yes I don't think so. I mean we probably could though. I don't I mean, yeah, I don't see why not There's I don't think there's any significant overhead We could try it Seems seems like a good idea. Yes I've got a question. Yes. Hey, so you mentioned that Classes and modules are the objects that probably will become old and will not be garbage collected What if I use a lot of anonymous classes? Does it make it harder for gc to do its job and I should avoid it? No, I don't think so if you use anonymous if you use anonymous classes, that's fine We can tell so When we want to divide those when we want to divide those up we can tell like okay, this is This class was allocated via, you know class foo We know it was actually allocated in the source where if you're doing an anonymous class you're doing like class You're doing like class dot new or something or module dot new So we can tell the difference between those two those two types of allocations So I mean feel continue to do that feel free to do it Anymore questions? Okay. I did it. Yay We have Samir Samir is a student at Pune So while he's setting up Do you guys know what the object oriented way to get wealthy? So I've been going through the speaker's list ever since it was out And one thing that stood out was that I'm like the youngest speaker here today So I think this photo here over here describes me very well So it's a great honor to share the stage with some of my role models and It would not be an understatement to say that I'm here to just Yes, these guys speak I come before you as a cat amongst giants and I hope you will forgive any mistakes that I'm with So you're probably wondering who is this guy? And introducing myself. My name is samir deshmukh also known as wardrobe on github and twitter I come from the incredible country of India Which is Home to the lonely deserts of Rajasthan The serene and snowy Himalayas The emeralds in the Indian Ocean that are the Lakshavadi pylons And the monument that is described as a teardrop on the cheek of time the Taj Mahal Specifically, I'm the from from the city of Pune, which is a city of about six million people It's also known as the Oxford of the east because of around two million students studying in the city In about 700 colleges So, um, I'm also an undergraduate student in computer engineering at the university of Pune Most of my family has mostly consisted of doctors. In fact, my great-grandfather was a doctor My father my grandfather is a doctor. My father is also a doctor But somehow I turned out to be an engineer and look like this not too long ago I also play in a band called cat kamikaze. You can check out our music on soundcloud I also love cats And I like the Japanese too. So it's got kamikaze I am on the core team of the ruby science foundation So also known as sigh ruby The ruby science foundation is committed to making ruby a viable language for data science and scientific computing Which is something that ruby has been lacking in for a while now It's very popular for web development, but not in this domain as much So, uh, we basically make tools and infrastructure and open source libraries for this purpose Uh, I was first introduced to sigh ruby as a summer intern for google summer of code 2015 Later that year. I also received the ruby association grant from the ruby association And uh, this year I'm an admin for desock 2016 um, administering, uh, the admin the Manage managerial duties for sigh ruby and also mentoring a student for improving data science tools in ruby So the talk my talk for today will be scientific computing in ruby In this talk we'll go over a few tools that you can use In ruby for data science and scientific computing And I will follow up my slides with a short demo of these tools The first tool that we'll go through is called the i ruby notebook i ruby is a browser based ruby repl shell That is mainly used for interactive computing So this is what an i ruby notebook looks like as you can see it's running in your browser And there's an input shell in which you can put in ruby code and renders it very nicely in the form of html javascript to css You can also put in a form Inside the notebook and you can accept Input from users which you can use for input into any ruby script that you have in the in the notebook The next gem is called n matrix N matrix is an n dimensional array object similar to numpy and python It helps to interface n made it helps to interface ruby with a few high speed c libraries such as atlas or lapak for linear algebra calculations So this is roughly what an n matrix allocation looks like n matrix is not A dynamic object like say ruby's array, but it is of a fixed size So in the first argument i'm saying that The n matrix should be of size 2 by 2 that's a four size n matrix In the second argument i'm specifying the actual elements And then i want a matrix of 32 bit floating point numbers. So i specify the metype option as float 32 So n matrix supports eight data types currently ranging from 8 bit integers to 128 bit complex numbers And it also has three storage types Two of which are sparse storage types and one is the normal dense storage type By itself n matrix is more of a ruby container is more of a ruby wrapper over c arrays But what makes it truly shine is that It exposes a very powerful c api which can be used by anybody for Interfacing these internal c arrays with a variety of high speed c libraries So you can basically create a plugin for n matrix with n matrix as a dependency And use this internal representation and the n matrix api for creating your own plugins Currently we have plugins for four major libraries atlas and lapak are mainly used for linear algebra FFTW is for fast Fourier transforms and the GNU scientific library or GSL as you know has almost every mathematical function one can think of The n matrix is also coming to j ruby by august and it will have the exact same api as the n matrix for mri And it will use jblast for linear algebra The third library is called niaplot niaplot is an interactive plotting tool It basically generates interactive html and javascript plots that run in your browser and let you interact with them So this is a very simple sine wave plot generated by niaplot Uh, niaplot also sports a somewhat similar plugin-based architecture similar to n matrix It has map niaf for creating map visualizations niaplot 3d for three dimensional objects And bionia for Realizing relationships between genes The fourth library is called daru. So for most rubies from all over the world Daru stands for data analysis in ruby but In the national language of my country that is hindi daru also means alcohol I hope daru The library becomes as important to ruby says daru the drink is for indians I hope both of them become equally important to tonight's conference party It's a so daru is basically an library for analysis and cleaning of data You can say that it's primarily used for data scientists who are interested in analysis of data It provides some powerful indexing functionality And it also lets you read and write data from many different sources like csv files xl files SQL databases and even active record It works very well with wild data So it so happens that almost 60 percent of our data scientists time Is spent in cleaning data and bringing it to the state that That can be analyzed by conventional means And daru tries to make this very simple by providing functions for doing that It also has a lot of statistics functions which work with wild data as well The libraries that I've quoted right now niaplot i ruby and n matrix daru is able to leverage these For things like efficient storage of data visualization and interactive computing So daru primarily performs its functions via two data structures The first one is called a daru vector It's a one-dimensional heterogeneous array object And each data data point in the vector can be labeled independently with daru's indexing functionality The second object is called the daru data frame This is a two-day spreadsheet like data structure that can be indexed both on rows and columns I'll switch to my demo now So this is an i ruby notebook As you can see you can put either markdown or a code over here Now I'm going to create a form So I can put in the number of rows that I want for a matrix say four columns four And I want the number to be three inside it So to turn the hash and I use this hash to create a matrix A short demo of n matrix Now I want to create a four by four matrix Or consisting of 64 bit floating point numbers So I specify the shape as four by four and I specify 16 elements over here Now I want to create an n matrix with eight bit floating point eight bit integers So just for demonstration purposes, I'll use 129 over here If you store 129 an eight bit number, it will be reflected as minus 127 You can access any element in the n matrix with a simple ruby array like api So I want the 0th row and the third column So which is minus 127 You can also assign numbers So 0th row and first column I want to be 56 and do that But as I said it cannot be expanded like ruby array So this gives a range of Now I want to use the atlas plugin for performing some fast computations and we'll compare this with the ruby matrix class for To know actually how much the speed actually increases So let's benchmark this Now over here i'm creating an n matrix Of size of these sizes five to 200 And I do a dot product on them for matrix multiplication. So let's benchmark this now So as you can see it's instantaneous The for size 200 the speed is 0.01 seconds And i'm doing the exact same computation n matrix On ruby matrix, sorry And the time turns out to be quite a lot So as you can see it's magnitudes faster For the same computation Now let's do a small example where I'll solve a system of linear equations using n matrix By the way you can also use iroby notebook to represent latex equations Right in your browser like this So i'm representing the coefficients of this equation this way in an n matrix of 32 bit floating point numbers And the right hand side In a matrix called rhs Now I use the atlas solve function And it gives me the solution Now let's head on to a demo of nap plot So now I want to plot a simple sine wave similar to the one that I showed in the presentation Here i'm creating the data for the x and the y axis I create a nap plot plot object and add it to the line graph and add a line graph to the plot object Now I want two waves on the same plot So what I do is I want a sine wave I create data for that using the math sine function And I want a cos wave And I create the data for that with the math cos function The sine wave should be a line plot Which I add with specifying the line option And the cosine wave should be a scatter plot which I add with the scatter option The color over here is red and the color over here is green So as I said it's an interactive plotting tool You can actually zoom into this graph If you hover your mouse pointer over one of these dots It will show you the exact coordinates on the x and y axis So now we want to plot two graphs which interact with each other and change according to Input that you put into one graph So let's create some hypothetical data So I have four band names here And I create a nap plot data frame which is an internal data structure for storing data And I create So there's a popularity between 0 and 100 assigned to a particular band in a particular country Now I want to plot a histogram Which has the of the popularity of each band And this next plot that I want is a bar graph Of the frequency of Of the number of times the band occurs in the data frame So I just do it with add.df and I add both these plots to the nap plot frame So as you can see both these graphs have been displayed here Now what I want to see is the number of bands that have popularity between 0 and 20 in any country So I can just select it this way And as you can see the bar graph has changed to reflect that So if you see megadeth is three And in order to verify that this is correct Let's filter this data frame and you can see that megadeth occurs three times here one two and three And each time the popularity is less than 20. So this bar graph actually reflects exactly that So next up we'll have a small demo of daru so The daru index provides indexing functionality for The the the rest of the daru data structures. I'm creating a very simple index over here And you can select an element in the index with the box operator Or you can even select multiple indices by separating them with commas So now you want to create a data frame. This is a very simple case of a data frame I specify a b and c as the hash keys and you can see that they've been reflected as a b and c as the names of the columns of the data frame indexing also supports Most more type of indexes called multi index data index and categorical index Multi index used for hierarchical indexing of data So now over here. I'm creating a multi index with tuples and I use this multi index As as for indexing data for this vector So now I want to select the sub the sub indices of a comma b. So I specify it here And I get cdp and now I want the sub indices of just a So specify a and I get bq which is exactly what is here So now the datetime index is another index that is used for indexing data on timestamps Here i'm creating a datetime index from a range starting from the beginning of 2011 Up to the end of 2013 separated by three days each I create another hypothetical vector And I indexed it on the datetime index As you can see each day over here is separated by three days And now I want to select the data from first of january to the 10th of february Which can be done like this as you can see the last date is the 9th of february Now I want all the dates in 2012 And now let's take a hypothetical data frame, which has average temperatures of these given cities And I want them to be ordered by name and temperature. So I specify the order option as name comma temperature And it needs to be sorted on the temperature. So I specify temperature as the argument to sort Now data frame also has a where Based where clause based squaring syntax similar to error or active record and You can actually specify where data frame where Where the temperature is less than equal to 25 and it will show it to you instantly Plotting is also very simple So over here I'm specifying a bar graph and the x-axis is the name and the y-axis is the temperature And it gives me a graph. That's it for my demo and I have cool syrupy stickers So you can come say hi to me and I'll give you a sticker And I'd like to thank red dot ruby con for having me over This is the first time I'm stepping out of my country and I was quite blown over by the amazing infrastructure of singapore And the very hospitable and inviting nature of its people So hats off to singapore for being an amazing city and hats off to you guys for being an amazing audience Thank you so much. Thank you Samir Any questions from the audience please So it can export the diagram to either an svg Or a pure html, but if you export to an svg you won't be able to interact with it So it will just be a static diagram So Samir, I have a question So regarding your Your scientific do you do any scientific research? With the libraries that you have just Planning to Suggestions for any of us that's simple to implement Well you can start with the plotting Thank you so much. Thank you So right now we have a break. We are a bit early so We can go out and mingle. I I'm not sure if there's still food outside, but please head for the coffee Our coffee is really good and some of you are unaware. We also have tea at the station So, please be back By 1115 for the talk by Constantine Obviously when talking about celery Who was here yesterday for joe's talk the lightning talk? Yes, so that actually said that celery isn't that important. So this Talk doesn't know it did not say that celery isn't that important. It said it's not the reason that people leave I also think that this talk that i'm giving is not so much about celery but more about development and leadership or rather How we do things at trevis And I think it's an important aspect of that is that I'm trying to break a taboo here A taboo that people have been breaking previously Very famously with the talk pay hashtag and Some news articles and blog posts lately. It is a great taboo to talk about celery. I don't know how it is in Singapore I know in europe and in the u.s. It's not something people generally talk about With their friends with their colleagues But it's not just that it's also not companies companies I found Don't really talk about salaries either I mentioned that was the talk pay hashtag where People were tweeting their salaries their job some details and that Gave a lot of momentum to having a conversation about salaries There's also bots that do that anonymously so You don't need to reveal your salary And there are some companies a very small number that are aggressively open about their salary structure Most well known probably buffer who have published their salary formula and not just their formula They've actually published a complete list With names and everything of all the employees and the salaries they're paying The buffer also is trying to be a very very open Company about this which is quite impressive so recently they saw themselves forced to Let let people go because they couldn't afford from their budget anymore to pay that many salaries which also Brings us to to the to one important aspect of this presentation, which is looking at salaries from the employer side As well as just what people are making which actually is the site that as a co-founder at Travis I've Had to deal with more often than the other side But we're actually is in a salary and I think that's that's where one of the The major conflicts between employees and employers comes from or the major friction points or interpretation differences is the different view and interpretation Of what the meaning of a salary is Who knows these people? Yes. Yes. Is that a big thing here? The big bang theory So that's interesting. Let's talk about their salaries Does anyone know what they make? And how they're paid A lot Yes, correct. No, not correct depends on who you talk about of these five So they all renegotiated their salaries last year initially the three Main actors renegotiate their salaries To one million per episode Uh, so How do you pronounce it? Gaelic Gaelic Parsons and cuckoo playing Leonard Hofstadter Sheldon Cooper and penny Last name unknown or at least I don't know um Got a deal of one million per episode in august 2015 at a time where hellberg playing Howard wallowitz and Kunal Nayar playing Rajesh We're getting paid 100 000 per episode Which caused the two to try to renegotiate their salaries And they were aiming for getting salary parity with the other three Which was denied to them. They got a raise which is undisclosed at this point, which is supposed to be in the mid six digits But essentially the salary Negotiation ended with the studio's threatening to write them out of the series And this makes you wonder what does this mean? Does this mean That one's a 10 times actor 10x actor This is this is a phrase from that's very popular especially In the bay area this concept of the the 10x developer And this comes from from a strong belief In an actually misused word in that context. I think the belief in meritocracy That uh merit defines you're standing in an organization This rock actually has been changed for one That believes in collaboration, so I didn't want to do like github shaming or anything But in the bay area There there is as I said this strong belief in the 10x developer Quora is a great resource in general on such topics If you want to learn the difference between a full stack 10x generalist and unicorn programmer There are there other like if on this topic 10x engineers, there are the really great questions like What do mediocre people do in silicon valley? You gotta wonder as some of the questions are why are the best programmers 10 times more productive than mediocre programmers? But paid only three times as much another interesting question um Very interesting to me as well is how do bootstrap companies hire talent? I've actually once was asked by someone In a in a bar like you didn't take we see How do you pay salaries? um anyway Nothing against vc just saying there are bootstrap companies and they do uh pay salaries I actually work for one of these I actually founded i'm a co-founder of one of these bootstrap companies Who here has heard of travis ci? We bootstrapped uh found it in 2012. We have 38 employees at the moment That was my last count from slack or this morning I think we have some top talent. We're based in berlin, germany We're actually a Or try to be a remote first company. So we have employees In eight countries of 16 different nationalities. Also again, I tried to count it this morning. There might be something of um Other interesting things to give you a feel we we have a 54 percent women In our company in our engineering team. It's 50 percent. It's a one-to-one split with With a very wide range of backgrounds And we were actually planning to keep hiring in the future And salaries is an important topic for us From the company perspective Salaries currently account for 46 percent of our spending. I think in most companies it's actually higher At least most software companies, but our infrastructure costs are also very uh massive But that on its own makes it the biggest cost center if you just look at it from a budgeting standpoint So any conversation that we have there? Even if it's just changing it by a small percentage Has a big impact on our company finances Um salaries have actually For a long time been quite a mess at travers To be honest, we started bootstrap. We started with not essentially having any money And we started hiring in europe And specifically mostly in berlin, which by western standards is a really cheap city and Salary was based on So what do you need? And that has actually led to once we started hiring overseas for instance a very uneven Not well thought out distribution and You essentially got a raise when you said oh money is really tight. I need more money But no one ever ran like oh you've been here so long great work So we've been having a discussion for quite a while in the company on how to change that And we came to the conclusion and the result Is that we got rid of salary negotiations At traverse ci if you want to work at traverse ci if you're going through a job interview We do not negotiate salaries with you Instead we have a sinatra app That will tell us How much we'll pay you why do we believe before I tell you more about this app Why do we Why are we trying to to get rid of why have we removed negotiations from this process? well We think that negotiation skills don't reflect your value to the company which is the employer's approach to To how you decide on salaries if you remember That uh the three main characters were at one point at a salary That was 10 times the salary of the other main characters in the big bang theory that was not based on them being 10 times better actors This was based on them being 10 times more valuable to the production of the show So your negotiation skills are not the main value you provide to the company unless of course you're hired as a negotiator Then that's that's a different discussion The employees view on salary is different It's not it can be the value argument can be used in a negotiation or in Figuring out if it's appropriate, but the employees view is usually in financial needs base And your negotiation skills also don't reflect your financial needs But most importantly As one of our core values at Travis we care about creating a diverse and welcoming working space And we believe that salary negotiations harm underrepresented groups There is lots of evidence for this for instance women people of color Generally underrepresented groups Often are impacted from things like imposter syndrome. There has been a study for when Women and men look at a job at and there are five requirements listed and For women one of the most common approaches is like i'm not so sure about this one requirement I'm really bad at this I probably am not a good fit for this job and most men approach it with oh, yeah, I check up three of the five I can totally do this This is not any individual person. This is just a general Thing that is more dominant in underrepresented groups And there is also statistical data to prove the whole thing So salary negotiations are very likely to cause pay gaps within your company So these are some of the things We wanted to tackle And we've been working on this for over a year and we call it the Travis CI salary framework We finally rolled it out released it this march company internally And this here Is the very first time we're actually talking about this publicly so All the things i'm telling you Are not really known outside of Travis Let's first look at how again how other companies have solved this Buffer also has an app. I don't actually know if that's Sinatra or not Their approach is very very formula based. We also have a formula, but I get to that in a second And your plan actually access their app publicly You can go to buffer.com slash salary and then you can select your role experience location And it will tell you exactly What you would make when you were to join a buffer But let's get back to our our framework. So buffer was a big source of inspiration and figuring these out because they also have public explanations of all the things So we developed this framework for a full year and Very importantly to how we have Travis approach these things This is a conversation that everyone in the company Could take part in so a lot of this was actually discussing aspects of what is a fair salary for instance Or what should be in a salary? so we decided that We want to pay by value and we want to pay by needs by needs So we want to take both these aspects into account and we also want to Come up with generalized rules that apply to everyone And that also everyone could question. So in a way you can negotiate salaries But you cannot negotiate your salary. You can as an employee at Travis CI say I think This pay difference is too big or we should in general pay higher salaries And then you can give a proper reasoning Or we should our increase our salaries because the euro is taking a nosedive because the UK is leaving the EU or something like that But you can say I'd like to make like 500 bucks more a month. That's not an an argument you can use That's a screenshot of our app This is Confidential You're the first people seeing this outside of the company Let's talk about What goes into this? Are people taking photos? Is this being recorded? So let's talk about value. How do you measure Value or how do you quantify What someone provides to the company Buffer does that with a factor. So they say you have an experience factor of 1.6 or something That is something we we started off with with playing with that But we found really hard to what does that actually mean And then everyone talks about senior developers, but senior is such an overloaded term in the industry where there again is a massive mismatch between Employers and employees Uh, similar to what does a 10x developer actually mean? So we instead decided to Take an approach where we have levels And then on these levels we have certain expectations that are quite meta That when we have an employee at a certain level We expect them to fulfill these expectations one way or another so If you had the discussion How do you define a 10x developer if they even exist? I don't really know how you place them on our levels, but Let's just pick a level. Let's pick the the level 10 of our software engineering career path and that has expectation like Shows an intuitive grasp of situations analytic approach only use in novel situations, which is basically You see a problem. You already know how to fix it without having to analyze all these things It's self-motivated to the point that they create new work for themselves and sometimes others And understands business requirements does not just understand but also shape the big picture So these are very vague and then you're supposed to work with your team lead at Travis CI on a regular basis mostly through one-on-ones setting specific goals for yourself um to fulfill And reach the point where these are expectations the company can have towards you We currently have our engineering career path defined up to level 17 We do not have anyone at level 17 The main intention is To show people that there is room to grow While at the same time staying a software engineer and staying at Travis CI Because this is actually one thing we see happening in the industry so much It's like people reached this point where there's senior developer and damn what I guess Not just about salary. This is actually about Your professional growth career development and so on and not just about your bank account statement And then you often see one of two things happen Either people go to a different company to do something new and fun or people become managers And in both cases you lose your best developers In one case you might swap them out for potentially a bad manager um If if someone wants to be a manager like I'm not I don't want a bad mouth developers becoming managers like if someone is really passionate about that sure But becoming a manager because it's the next logical step in your career path is a really really bad reason And these levels we've constructed or tried to phrase them in a way That employees should level up about once a year So that it's also clear how often people can expect salary raises And most importantly we don't view this As a performance review that you get once a year That we can use to justify to not give people a raise But instead as a program for us To grow people internally If someone does not meet the expectations for the next level That needs an investigation of what's going on Is this Some productivity issue do we have other problems here that we need to address? Or are our expectations wrong? You do not fix productivity issues by paying people less The other aspect we take into account is needs We use a generalized needs model based on location This was actually a discussion initially where We assumed that the bigger discussion would be should you pay by location or should you not pay by location? Actually The opinion at Travis was pretty unanimously that you should pay by location The bigger question was How do you determine that those differences? But essentially even if we Wanted to not pay by location just pay everyone on level 10 no matter where there are the same salary We could either not afford that Or we couldn't hire in California essentially But then we have the goal had the goal that if we pay by location differences We want to be scientific about this. We want to be reasonable about this And we don't want to have someone Basically get a worse deal because they're in a cheap or an expensive location We don't just want to pay people in San Francisco more than everyone else essentially So how do you do that? There's one interesting statistic. That's a big Mac index. Has anyone heard about the big Mac index? So big Mac is a great thing because it's in so many countries And it's essentially the same supply chain, but it's different cost So you can compare countries Based on what the big Mac cost. We do not use this for salary determination by the way It's also it's a bit unfair to have India in there actually because it's chicken meat in India Which has different costs associated There is different conclusions you can draw from this like Big Macs are the most expensive in Switzerland But that doesn't tell you actually much about Switzerland because if you then compare that to Or Norway for instance is the second most expensive It's the cheapest in South Africa But if you compare it to how long it actually takes an average worker to work to get the money for a big Mac You see that Switzerland is still pretty good and actually Hong Kong is the best where it takes on average 8.7 minutes of work to get a big Mac Whereas in Ukraine in Kiev Ukraine. You have to work for almost an hour There are actually way better statistics to reason about cost of living one of my favorite is Numbaya, which is actually something that we used So we use Numbaya. We use the international labor organization. We use the bureau of labor statistics And we use shadow stats, which is basically going like oh those Bureau of labor statistics statistics aren't actually that great to look at This Numbaya is actually a crowd sourced Database that gives you very detailed insight into into living costs for instance for Singapore it uses 6,300 something entries You can see what a McDonald's meal costs as well, but also disposable income rent and so on When I when we started investigating I built a little web app Where you can basically enter which city you live in where you want to move how much you make And then you can calculate which salary you should aim for when you move there It doesn't work well for Singapore because that data on there is outdated and some models in there are outdated But works quite well if you want to move from san francisco to Berlin or the other way so Say you make 130k in san francisco means you should aim for Somewhere below 60k euros in berlin if you want to keep the same start of living That's living costs, but also Market rates are a big factor as well There there are some websites like glass door Where people can can Enter their salaries and then it can give you an overview of a location Pay scale is similar also Really nice to know they have like fifth and 95th percent or I actually think I think this is 10th and 90th percentile and Pay scale has that as well for the US Thanks to the open government campaign under the obama administration You can actually just access the data from all the visa applications. So, you know What people are getting paid when they get a visa for the US or a green card Important here is that you only compare data points from the same source Because the visa data points for instance tend to be on the high end of the salary spectrum So you shouldn't use that data point on its own But you can use it quite well to compare two cities and the same is true for our glass door and pay scale Where salaries at least in the US where I can compare to other sources Are more on the lower end So I use that I did a factor that's a comparison between Berlin and that city and Created a splatograph of the biggest cities in the world To see your correlation between market rates This is market rates compared to Berlin and this is living costs compared to Berlin And you actually see that almost all the cities are in a very nice line. You also see that Singapore is quite the outlier. So Living costs are significantly higher here than in Berlin, but salaries are Not proportionally higher as well And then besides that what we took into account is I think I'm running bad on time good so what we also said Or what what was more like the conversation went like this Um, so if we take living costs into account and rent and all these things Shouldn't we take income tax into account? and The conclusion was yes, we should but you can't really calculate the income tax for everywhere in the world And then okay, that's right. And then I spent a weekend playing with some ruby code and turns out you can Calculate the living car the income tax for everywhere in the world. I created a ruby library. You can gem install income tax You can tell it which country you're in. It doesn't do tax deductions or anything Even though someone emailed me and they want to do like a tax filing program for you. I don't know. I think they Actually never got back to them. Anyway, um, so you can tell it how much you make or salary a country and we'll tell you about gross income net income taxes and Important for us. You can also tell it a net income Like the income after taxes and it will give you a gross income Interesting things I learned there in molly.  You can also bring a goat and use that as your tax payment Um And from that we calculate The the salary levels we have a baseline per country that's based on market rates And then a city adjustment for expensive cities that's based on living costs So we do take market rates into account for the us But we don't take market market rates into account for san francisco But we do take into account that the rent is really high in san francisco Here's actually the coat. I promise coat. There's the oh, there's it's been cut off the result Basically the rounded we have our own rounding method to Be more favorable to our employees so it's uh taxes applied on top of the Before taxes we apply of someone's working part-time Before taxes is a monthly value times 12 monthly is the the base value Um Plus the increase for the level you're at Times the level plus the city adjustment exchange to the local currency Yep That's essentially it if we would pay less than zero. We just pay zero We don't make people pay And with that we've actually calculated the rates for 3536 cities in 209 countries for four of these countries We also have 92 regions you might wonder 209 countries there aren't that many countries. I recommend to you this youtube channel I also found myself in that income tax gem to for the first time specify What this gem sees as a country and stating that this is not a political opinion, but just trying to model income tax best Currencies are very tricky employees that are Not in the u.s. Or Germany can pick their own currency we have Conference allowance converted to the currencies. I imported a set a list of currencies with exchange values Which I didn't know when importing this currency list included gold so If you work at travis ci and we pay you in gold you get 4.45 ounces of conference allowance per year I assume this is how you go home And we've rolled this out. We've used that for all our new hires for to replace salary negotiations We're very happy with it. It reflects what we set out to solve. We think the rates that come out are competitive based So there are some relation to market rates. They're comfortable. We think people can live of them. They're fair this has to do with the Rules applying to everyone for instance, they're feasible They're within the capabilities of our company and they're prospective. So people know there is A path to take and most importantly, I think it's a model that everyone Participated in and can participate in shaping how that develops in the future Thanks I don't know if there's time now otherwise hit me up later or treated me email me I also brought stickers But I absolutely know what you guys are doing So Can you hear me? Yep. Yeah What's your plan for bringing existing employees across this? Have you done it or is it going to be a gradual thing? So we actually started doing that So we've brought on So all of our engine Nearing team I think is now on the framework So we have this concept of people being on the framework and people being off the framework We started rolling that out. We don't have all the career paths done. So we have the career path done for Software engineers designers Support we're working on our administration career paths right now So that's why not everyone is on this now, but the majority of Travis CI is paid in accordance to the salary framework Yes, so we've actually discussed this releasing this public. First of all, we want to actually Talk more about this blog about this not just to like show off what we did But also to get a conversation started that goes further than just Travis We still need to figure out What work needs to be done so that we can actually release our app Which would probably also be quite interesting for people out there Yes, so This is uh, where do I have to slide the currencies are tricky So what we actually have is we have a Travis CI exchange rate Which means we picked exchange Behind the scenes, which is back end. I'll be speaking a bit about front and back end like back end Uh, a bit towards the end as well as front end. I don't know what this emoji was how to represent back end So I just flipped it I tried to search for it emoji for front because there was a back. So it should be there should be an emoji I spent some hours actually a day trying to trying to find this But I couldn't so I thought that hey, I should actually try to submit my emoji proposal And I ended up here, but I didn't have time. So Yeah, we are this is the front end emoji So today I'll be speaking Many progressions that have happened in the latest version of rails rails 5 One of the things is it has gotten a nice logo and this homepage. So yay for the designer who did that Which was after 10 years using the same one Uh, most of my references that I'll be speaking today are based on the block that we run on On my company, which is called as Which you can find here. We have around 60 plus Blocksy blog entries right now for all the various changes which have come in The latest rails version So let's begin first thing that I want to touch up on is prockets So sprockets is that part of rails which is used for actually serving your resources Your js and css resources There's a wonderful talk on how sprockets actually works if you want to go deeper into that Which was given at the this this year's rails con by rafael. So you should check that out I want to focus on some of the main things which help you out Which are the new additions right now in sprockets the sprockets come consists of all of these things which is sprockets sprockets rails sass rails exit js coffee rails, which are like peripheral gems which actually comprise of the asset pipeline Uh, so one useful addition which sprocket has seen is that of manifest.js So previously in your older versions of rails if you would like to have you know, use a different kind of asset Like let's say I have my payments page and on my payments page I want to have different styling and not embed that completely in my Application css. I could I needed to do this this kind of thing wherein my pre-compile needed to have payment css Advertising css etc And then on my page where I actually need to use that I would use this kind of tag Uh rails 5 Comes now with manifest along with support from sprockets wherein you can just define this kind of file It is by default generated on rails 5 apps And then here you can define Which file you actually want to pre-compile as well So instead of defining that list which is not kind of handy Like you always need to define that and make sure that it's actually in the pre compile pre-compilation So that it actually finds it in production in your apps You can instead do something like link And you can actually use this manifest file to define dependencies on all of these multiple different assets So we will say like Link tree and link all my images. So instead of specifying them I will say that hey, I need to access these images and when pre-compilation happens They should also go along like from this In the process of assets pipeline I can do link directory and link and so and so forth So here I see here you can see that I am doing link advertising css and instead of specifying that in My pre-compile and specifying it over here in using link After I do that I can then go ahead and start using that in my erb or Also in some other directory in some other asset as well So if I do this Like rails will like sprockets would automatically pick that up and it will autocom like Send it through the pre-compilation path Next useful addition which sprockets has got is that as folders as modules So if you if you notice over here, I have this foo slash index.js So this is a kind of a technique which is pretty popular in node based apps Where in you have this directory and you specify index.js and instead of having Including this this asset like asset path foo slash index.js instead of this you can directly just use foo.js and it will go and Import the index file. So this is pretty useful if you want to define Something like this So previously if you wanted to you know import all of your different assets in your directory What you would do is you would do something like require tree Require tree in that particular directory and what will end up happening is it will import all of those in an alphabetical way Problem with this is Problem with this is if you have such kind of a file jquery ui.js and min.js Problem with this is it will import alphabetically and then This file will be imported before Its dependency and you'll not you don't want to have this kind of scenario So instead of this now you can just import index.js and there you can specify the order in which you actually want to import a particular file So this is pretty useful The next thing which is pretty useful again for debugging debugging purposes is source maps So in the previous versions of rails on rails 4.2 if you try to do this You would see that On a request on your development side You have all of these resources being loaded So individual files Every single resource that you have it will send a new request for fetching that particular Resource also like jquery button etc On the latest version rails now actually Maps that similar to how you have in production. So in production you actually have just a single file Which is compressed and minified. So now in In the rails 5 apps you'll start seeing something like this which is application debug And what this means is it is it is trying to map the same behavior that you actually see in production Which is something that you would like to see Because it So that you don't have any surprises when it actually is how it is actually working in production So for so since you have this comprised thing like Compressed thing that is also being served in your development version Whenever you want to debug something it's pretty hard to actually find a particular line You know something goes wrong another is thrown from a javascript You don't know where that particular line is because it's not compressed So to actually find the location of that particular error or something like that We can now start using source maps So source maps are actually defined by this mapping this directive that you see over here So let's say I have this file application js Sprockets will create this mapping file that you see over here Which ends by just map And this mapping file will then be provided by these directives in the original file For see it for js. It is this way and for css It'll again provide a different kind of mapping file And this is pretty useful. So like In your request now you'll see these two files being served But on the on the browser end browser will detect these mapping files and then you will actually You will actually be able to find the mapping of these particular So if you want to actually trace back to some log or some error being thrown you can map that back to the actual sources again The next useful thing which At least for me, which is pretty exciting is be able support for sprockets How many of you use or no? es6 es15 A lot nice. So es16 has a lot of these Amazing things which i'm pretty excited about to use have started using Like the arrow functions lexical binding Classes and and has object or literals and a bunch of different things So by default now in the latest version sprocket ships with Babel support. So now you can actually start using es6 by dot es6 Extinction So this is pretty exciting again. I wanted to show this by this And I tried to search for this but it was like hey There are so many different things that you can actually show this for like rainbow unicorn poop as well Or something like this. So internet is pretty awesome If you try to search for it The other thing that I want to speak about is this 12x improvement that sprocket has seen in the pre-compilation process of asset pipeline so there's this nice blog which Richard Schneem's Which he goes by schneem's has written on heroku. So you should check that out which actually provides the whole listing of whole process of how this speedup has occurred so Here's the here's the actual flow of things that go into the pre-compilation process like you have your raw asset You try to you try to actually process that Based on which kind of you know sass or coffee script And then you try it is bundled together like all of your different files are bundled They are compressed then they are given a fingerprint Like a digest that you see after a particular asset which is for cash bursting and then they are minified and so on so forth In the previous versions of in the previous version It had actually had a bug if you see over here. This is the function which is used for actually finding the digest So when you pre-compile it tries to you know store that particular asset in your tm like asset tmp asset cash if you have heard of that So for actually finding so ideally you would not like to go through a whole compilation process because it's pretty slow To actually getting that asset so instead of that we have these fingerprints and then you use that to find that if that resource is already being Processed and is it if it's either already in the exists in the cash or not and if it does then just return that Problem over here if you see the path is being provided over here In the previous versions it used to pick the absolute path The issue with this is if you have if you use something like Heroku or if you're using Uh, like Capistrano in case of Capistrano every deployment will actually create a new directory So if you're using this Your cache is always going to find a new path. So it's not going to Refer to the original resource. So you are actually missing the whole point of Having accessing the cache versions. So this was fixed by uh, this fixed again by richard and he added this unload asset which actually tries to find the actual paths properly When it does the whole compilation for this digest and finding it actually in the cache Uh, the next thing I want to speak about is sri, which is sub resource integrity Uh, so it it has been added as an experimental feature right now. It has been present present for a while Uh, so what this allows us to do is we can append integrity through as an option to our application whichever resource that we want And that will create an integrity hash Uh, so now why this hash is useful is so let's assume we are have we have a page which has multiple different resources like I'm not always going to access a resource which is in my project like I would like to access The jQuery or bootstrap which is being accessed from some third party cdn or so and so forth In that case, I need to make sure that whatever the the resource which has been provided to me It actually matches the one that I wanted to access and something has not been injected into that particular resource So in that case, we have a use this use of our sub resource integrity Which makes sure that whatever resource we are actually providing actually matches the digest of the original resource Which you meant to actually access from a particular domain So you can start using that right now All right, uh, that's all about rockets Hope everyone is following me The next thing I want to speak about is all about headers in Your page So this is a typical header or the response that you see when you you know send a request to a page This is coming from The chrome browser it shows what are the response headers like you have you have your e tags date content type, etc So what I want to speak about a bit about is The how you can use them How you can use or manipulate these headers so that your resources are actually cashed on your browser in So one of the good uses of these headers is e tag So if you see over here, uh This there's a header which is which says e tag and it has some digest And then we also have this last modified over here. So using these two headers, uh The browser can negotiate with the server that hey Whether I have the latest resource on my particular like in my cache or not In the previous versions you could use like in the old like right now you could use it with fresh wind or stale And what essentially this will do is it will first of all find out all the items and then it will generate an e tag for those items And when you send this as a request So if you see over here I'm sending a request to find these fetch these items On the first request it will return 200 as a response On the subsequent request if I pass this e tag which was generated over here This e tag if I pass this to this request again It knows that hey, this was my e tag. I'm passing this in in this header if none match It sees that there is some e tag object Which is being passed to me and it tries to find that on the server And if it does match then it returns with 304 modified Not modified and just use the existing resource that you already have in your browser So this is useful so that you don't actually go to your database and try to fetch Your resources and slow down your request. So this speeds up That part of the thing So rails now Has support Jeremy added support for strong and weak e tags. What this means is So these are two kinds of e tags that you can actually have in your requests What this means is if you have a strong e tag Your request when you send a request it will try to do byte by byte matching So this is useful when you have some kind of a png or you know, you're streaming a video Where in you want to see hey this resource which was previously sent to me does it actually match To the resource that I currently have This is not so true in case of pages you are generated like all the default pages that are generated in rails app Which are dynamic and in which case they are not always going to change That's useful to have this kind of weak e tag Right. So in this case, it is just going to create a digest of that Of the resource and it is going to match with the previous digest instead of doing byte by byte matching of the whole Response because this is pretty useful to you know, not go through that render process to find out if the resources changed or not Another useful helper which has been added Is http cache forever So let's assume we had this home controller over here, which just serves this welcome tag Previously like if you if you always try to access this page What will end up happening is it goes through processing render and all that multiple different things the whole cycle of particular Serving of a reason like html page This happens for every single request Instead Now you can say cache forever It comes with a risk But now you can do something like this http cache forever public true And render the response that you want to actually send And if you do this for the first request if it will serve it in this way like processing so and so forth In the next request it will just reply with 304 What is actually happening over here is when you do public Caching it will set the cache control over the resource So your page will actually be setting this cache control max age a hundred years or so So use this with caution what this is useful is let's say you have a page Which is really static and it doesn't change a lot and it's never like it's you know that it's not going to change You can use this so to notify the browser that hey Don't always try to compute this resource just use the one that you already have Use this with caution because the only way to you know Undo this or get a fresh resource is doing a hard reset on your browser cache The next important thing that I found or that is that I found pretty useful was having support for custom asset headers So let's assume we have this request. This is a typical rails request wherein You have all these headers these are not visible thus but just take my word that there are lots of headers over here If you see over here If I want to specify a custom header in my request in my response Usually you can easily do that by saying response dot headers some header and then try to set some value over here And then this value will be served from the request problem with Problem with this is you cannot do that easily Or previously you could not do that easily in case of your assets Your assets only had or your dynamic assets only had this one header Which is static cache control and the only thing that you could actually manipulate in this case was this header Which will only set the cache control over here This is useful again for caching purposes. You will say that hey To the browser whether you want to cache it publicly or not and for how much time you want to cache a resource Before actually getting the new one problem with this is Sometimes like in case for me what it end up what it what it does is it some of the browsers do not actually Just depend on the cache control. You would like to have other headers like This is something that I got from page speed, which is a google tool for analyzing your How your Pages are doing So if you see over your page speed was warning me that hey expiration was not set for a resource So you actually want it it was saying that for some of the browsers Expiration header is also useful along with the cache Cache control so you I should be setting that But I could not do that easily since we can only specify a cache control as a specific as I said so This is a new thing that you can now start using On rails 5 when you can specify public file server and then It actually uses all of these custom headers like cache control you can specify for cache control You can specify for expires or you can specify other different, you know for controlling It so and so forth for your Serogate cache or cdn on so so and so forth So this is useful if you are actually using something like Fastly or if you're on heroku In case of if you have your own server like engine x this will already be taken care of by on the engine x side Uh and now if I after I've added this expires header In my like in my config Like the result is that you'll stop seeing that warnings and then you'll actually see start seeing that header which is being set to here This was actually done by yuki. So thanks to him for that The next thing I want to speak about is caching So apart from these things which you can use like headers or e tags for specifying Not to compute the resource you can also use a bunch of different things on the caching end of Action pack or action view There are a bunch of different ways you can actually perform caching in rails a page caching action caching etc You can find them all on the rails guide Today, I just want to speak about The new additions which have come into the latest rails release So one uh important work like important change which has come in is that of partial rendering from cash So if you see over here, uh, I have this to do collection And what I'm doing is I'm rendering that as a partial And here I have this cache which is using this to do So when it wants to actually like render this it will render it for that many number of times And then for every single cache it it will try to Uh actually fetch your cache entry for that this to do that you see over here Which is expensive if I have 100 of these it is actually going to send hundreds of requests to fetch This cached version of to do So instead of this now you can start specifying this cached true When you're trying to render the partial This was basically influenced a lot from multi fetch fragments gem And now you can start doing this and rails will Identify that hey, I want to actually render a collection and then Not do these multiple requests to the cache and just use read multi to Actually fetch all of these entries together in one single request Which gives you a lot a lot of boost so you don't have to do that back and forth to the cache And this is something that you'll see in the end result in the logs that hey It just fetched it in one single go Uh the author of this library like the original library which uh Actually, which which was from it which was influenced by this In in his usage usage it actually improves the performance for these kind of scenarios by at least 72 percent. So which is pretty good The next thing Which is kind of on the peripheral is that of action mailer fragment caching So now you can also use like make use of caching in your mailers You can enable this by perform caching true again for action mailers And then in your mail fragments what you can in your mailers what you can do is you can start doing this cache do And then it can also if whatever part you have of mail which is you know actually dynamic You don't need to do that. Uh, you know pre compute all those values. This is pretty useful because Uh In your mailers you don't have to end up spending so much amount of time to actually compute all of these things which You know are going to just be this part of the particular Mail is actually being going to be dynamic. Everything else is going to be seen So you can start using this now to use cache in your mails You can see over here how it is uh After you run this request it says that hey, it's going to fetch the cache digits for that particular part of your mailer Uh One another addition which is kind of useful which goes hand in hand with the fragment caching or collection caching Is that of support for collection caching on result sets? So active record relation now has uh support for cache key So previously when we were saying cache do you have to actually provide So we'll see an example for that. So if you notice over here I'm trying to find this Result set when I'm saying I need to find all the users who stay in city Miami The thing with this is if I'm not changing this particular query If no records were added or if no records were deleted This result is always going to be the same if nothing was changed This result is actually going to be the same so I can catch this particular result Now there are multiple different ways uh previously you you had to do that manually for yourself like for a result set You you needed to derive How you're going to specify this cache key? Our rails now has the support for cache key on these collections So what it does is behind the scenes it tries to find that based on the updated at timestamp and it has some restrictions But based on that it By itself it tries to find this cache key And then you can start using this cache key to actually you know cache users and it will automatically start using that cache key Instead of you specifying or you trying to generate a cache key for that collection, which is again pretty useful So that it gives you again this use of caching over whole resource collection of resource Finally I want to speak about layouts and tags Which is coming towards the end of how your pages should be structured So I want to get to first or I want to first get to refer to this async and defer which is kind of attributes that you can specify over your tags So here's just a legend of that I've picked it up from an example, which is here somewhere So here the green part is actually parsing halted downloading of a script and a script execution So generally when you just have this script tag Script tag with application import your Usual import what will happen is It will your page is parsing parsing parsing. It comes to your JavaScript include tag Then it will halt for a while it will stop rendering your page It will try to go and fetch This since it is a different file it will send another request and for this time it is halting Which is not pretty good because for that time of download nothing is happening on your web page After it is downloaded it will then go and actually execute that js which was imported and then it will really you know it will Again start rendering or processing the html parsing But this is not pretty good because This affects your page speed if your resource is slow your file is huge This is going to affect how your page is actually rendering Other ways that of specifying async If you specify async what will happen is this resource will Download so when it sees that hey this tag was async it comes over here Instead of stopping the parsing it will continue parsing. It will try to find this parallely And then it will stop again the the html parsing and execute that particular JavaScript or css resource The third thing that we have is defer Here what happens is the normal execution will go as it is whenever it finds this differ or It finds this javascript tag it will go and download it But it will not stop normal html parsing. It will do that towards the end So based on your usage How you're actually importing your javascript or css. This is pretty useful for Speeding up your page actual page paints This is useful for your page speed again So there's an ongoing pull request right now to have this by default in rails You can follow that over here. So this will actually be pretty helpful by default if it's enable in your real saps Towards the end, I just want to speak a bit about page speed So you can use page speed this tool we just provided by google for finding out different things that you can improve And this lets you know about how your placement of js and css imports are How you should import your forms analytics how you should actually have your analytic stacks and so and so forth So in a typical application a dot html file you will have a bunch of different things So I just want to end with this that How you should actually place all of these stacks Based on your execution it will actually slow down or speed up your painting of a page if you see over here I have my style sheet tag by default style sheet and js file js tags will be in the head In your new apps you should usually try to move the js towards the end so that the execution is not stopped You can also specify async and differ again so that execution is not stopped for your html parsing You might also have something like this which is for your Loading of external fonts, which is something. I don't like but you will need to have in your apps So you you should actually try to move that towards the end or actually try to use open fonts and just embed them in your assets asset pipeline Google actually says that you should have this style sheet towards the end So that it is not blocking again for fetching that file But usually this is pretty useful again because you don't want to have the page like you know All black page and suddenly you have that whole style coming So this is useful so that it appears that your page is actually pretty fast So the style sheet actually should be above So based on your usage you should have Your scripts execute towards the end of your page on page load and so on so forth And yeah, you can combine all of these different things that we have discussed today To actually speed up your page Which is something I like in the latest release of Rails. That's all I have for today. Thank you Thank you I would hate to hold you back from lunch, but does anyone have questions for him? Okay. No, everyone is hungry. Yeah Before you go, I have to make a few announcements. It's more reminder than announcements Uh So no food and drinks allowed Please be back by 140 fs. We have a surprise by one of our sponsors for sure And after that we will begin Sayani's talk on sense and sensibility at 145 shop Please don't forget to Visit the appreciation board Thank you Ben from for sure Has something to tell all of you Cool. Hello. Hi there. Thanks Cool So this is just going to take a few minutes. Um, but I just wanted to kind of ask Show you something quickly. Um, so Me and hamilton, uh, we're from pusher. Hello. Um, and we've come across and we're sponsoring Helping sponsors this which has been brilliant. Um, but so we've got a prize But we were supposed to have a raffle and we're supposed to gather names over the last couple of days But we kind of forgot about that So what we've done is we've So we've got this prize which is a bv8 Right, so this is like a kind of remote control. Um, bluetooth like very kind of cool little mini robot Um, so we want to kind of give this away and the way we're going to do that is we're going to have a live real-time Prize draw. So the way we're going to do this is I want you to all visit rcrdrc.herokuapp.com On your phone on your laptop or whatever you can And hopefully that'll work. Uh, but and what you should see is you should see, uh, a big circle Is that working? Cool, right. I'll give you a few minutes So the more people we can get on this page the better Um, uh, it should don't like it should manage up to a few hundreds Hampton, I think we're gonna need the Plan b Oh, no, no Right, so if everyone what we're gonna do is um, we're gonna change the colors of all the circles in this room And one person is going to end up with a big red dot and that person has won the bb8 Right, okay Yeah, no no no changes the css so, um We'll be checking I should have thought about that Okay, so what I want you to do is I want you to turn your laptops or phones towards the front So that I can see the the circles and they should all be Be white cool. We can see those Right, so I'm gonna select a winner and hopefully this will work Right, okay, so they've all gone black. Um, if they're connected so Hang on a minute. Hang on a minute. Um Right, so we're gonna cycle through some colors here. Okay, so now it's kind of flicking between different colors And now we're gonna introduce sounds to turn up the volume on your laptops Okay, so we're doing colors and sounds and we're gonna just randomize that a little bit And then one person is going to end up with red dot. Hopefully Is anyone got it? Oh, there you are How come everyone everybody can hear me? All right I am so so happy to be here This is my third time that I'm here in the red dot ruby conference And the first time when I was here. I was just a beginner Trying to learn the ropes or web technologies the ruby programming language and the rails framework But it has been a phenomenon and fun journey since then because of many of you in the audience the ruby community Of singapore and beyond whether it's people like winston Who has been so crucial about programming and teaching are people like huanito You know like spreading the love and laughter Or even getting some free hardware like this in red dot ruby conference just last year It has been a fun journey But being a beginner is not just a story of an individual It is a story told repeatedly By generations of engineers experienced engineers whether it is The integrated circuits of the 1960s or personal computing of the 1980s or even more recently the web and the internet Engineers have always taken on something new something unknown something messy Have kind of played with it and made it so usable to the people beyond That it almost became an extension of who we are And so I thought I should start with a quote The wheel is an extension of the foot The book is an extension of the eye Clothing is an extension of the skin Electric circuitry is an extension of the central nervous system Today, I hope to share with you something that is an extension of our very own senses Starting with the sensors themselves and some of it I have right here I want to be taking you down from the extraction layers from the physical world the real world All the way to the application layer, which all of us are familiar with And I want to share with you exactly what is changing in each of these layers So let's start with the sensors their pressure sensors light sensors humidity sensors They are becoming cheaper and more compact than ever before, which means it's just much more accessible to us The next layer is the wireless communication stack, which the sensors will be using to transmit their values And all of us must have heard of nfc or even wi-fi Bluetooth and 4g which is kind of long range There are newer protocols coming up such as bluetooth low energy Or even 5g that will be coming up in a few years and these are especially Important for tiny sensors low power sensors for machine to machine communication The next layer is the access layer So all of us who are web developers here are very familiar with these protocols the http the html Jason and so on I also want to share with you two more protocols in the application layer, which is the mqtt and the co-app And these are very important for machine to machine communication once again And of course above these we have the web of things layer which all of you as rail developers or web developers will be playing with Now for today's talk, I will be specifically focusing on the bottom three layers focusing on the layers that will be communicating with the Physical layer the analog layer with the digital layer the application layer So before I get started, I want to talk a little bit about what is an internet of things platform So a show of hand how many of you are rails developers? Wow, that's about 70 of you I I agree with you not all of us are web developers. So we might be playing with ruby a lot So as web developers as rails developers, I'll be talking about this framework called thingspeak So if you go to thingspeak.com You can actually connect your sensors and play with it. So it is a hosted platform But if you want to look at the source code as developers, you can also go to github And you can also download the source code. So just a word of caution The source code in github is about a year Old so it will be based on ruby 2.1 and the rails version is 4.2 But nevertheless, I still feel that we'll be able to Catch on a lot of how the controllers are made the models are made and get a lot of ideas So essentially what is an iot platform? So it has four key features number one So you have a sensor out there in the wild and that kind of communicates Periodically to your ruby on rails server. So that's a key thing about what is an iot platform The second thing is When does the sensor send these values? It can be triggered by event which can be by a human And it can be by periodic like every few hours or so The third key thing is what do we do with the sensor data after it comes to your ruby on rails server? Typically, we will plot charts and graphs. So that's a very key function of this The last attribute is hooks and I think as rail developers We are very good at this like when we get a post request or a get request We might trigger a rake task or we might generate a pdf or an email So these are the hooks and triggers I talk about so now that you know what is an iot platform I want to be giving you three demos in the course of this talk and hopefully with these three demos They are tangible enough for all of you to take back something home Get excited and start playing with the physical world So I have three demos here. We'll be using wi-fi wireless protocol for this purpose I have three kinds of sensors or acquit actuators and being a red dot attendee I've used emojis as you can see the first one is a switch The second one is a proximity sensor and the third one is a temperature So let's get started with the first one. It will be event triggered So before I start with the first one, I want to be talking a little bit about the wireless protocol system So we are in today's world very familiar with Two kinds the wi-fi which is like within a room or the cellular which is like 2g or 3g So it's very important to know where is the device of the sensor and where is The kind of the central device that catches all these values in the case of wi-fi. It is the router So I have a little router here. So I've kind of flashed the firmware and connected it to the ssid of this little guy And here is the router and this is the sensor. So in terms of cellular, it is the huge cell tower Now what's happening in the last few years is this We are having more and more chips, which is the connectivity chips available to us So this is an example of a photon, which is a dev kit and over here I have a wi-fi chip made by broadcom bcm And it's all done up for you in a tiny little pcb that you can explore I have another example This is based on as a stm 32 microcontroller the switch is connected to the microcontroller And that's where I flashed the c program To kind of say how it will communicate and then of course it has a wi-fi chip on board Which is talking to this little router here All right, so that is the thing and with this wi-fi it kind of talks to my ruby on rail server, which is running in my laptop All right, so are we ready to see what it does? Okay, so i'm going to go here and this is completely wireless So let's see i'm going to turn it on Of course All right, so Let's go on and i'll show you the channels this time It's going to the switch So this is where you will see the public switch here the public view. So this is how Let me see whether we got it. No, we didn't get it. Oh well Well, even if you don't get it I'll come back here and then i'll show you the channel settings So what happens every time I turn this on and off? It basically communicates with the rail server right here and when it communicates it gets a post request So let me show you the api keys right here. So this is the api key you have here And then the channel settings over here You kind of set it up and then you say that in this sensor node right here You have eight different fields and one of them is a switch so let's turn back here and This is the api key Let me move on and show you where this can be used this little switch How many of you have heard of amazon dash? Many of you. So this is actually using the same Broadcom wi-fi chip as the amazon dash here So what it does every time you kind of switch or push the button It connects to your router at home and it basically creates a line item in the e-commerce cart of your amazon And the cool thing is a few months ago amazon even released the iot button And the iot button is available as a developer kit. So go ahead and play with it aws iot button So that's a brief about this But I want to go to the firmware and the rails code where the communication happens So I've basically kind of flashed the firmware right here And even though this is c code, I want you to notice three things about this Firstly, we have slash update Secondly, we have the api key right here And thirdly, we have the field which is the on and off of the field And of course, it's communicating to the whole thing which is my laptop here and the port So with these three things Let's go on to the rails and this is something I think all of you are familiar with So what happens in rails with slash update? It goes to the routes rb and it is kind of defined as Channels model and then the method is post data. So when it gets this data It creates a new feed. So feed is also a model and of course it checks the api key And then field one it kind of creates one line of the feed So that's the meat of it all So let me go to the second example. That is the periodic part of it And this time it will be not triggered by a human being with switch But it will be triggered by proximity or in terms of every interval So let me go on to show you this little thing So here I have a very similar case. So this is the ultrasonic sensor that you see here And ultrasonic sensor basically means that When you have a line of sight obstacle, it can measure a distance And this is a little different previously. I powered it up with a power bank But this time I'm powering it up with a lipo battery So if you kind of sleep and wake it up every few hours, this can last for let's say a month or so So this is an example of a proximity sensor, but it will not be triggered by me this time So let's see an example. So I'm gonna go to the channels And let's see proximity Let's go to the public view and let's see what's going on here And let me turn it on. Let's see whether it works even if it doesn't work. Well, it doesn't work All right, but I will show you something Let's see once again whether it works or not No, all right. So I'm going to show you right here And this part I want to show you the data. So if you click here Now things speak will give you How it's you can able to kind of communicate with it So if you want to do a post request you can do a slash update here And this is where you they will actually generate the url for to write in the firmware and communicate with the rails And then this is the feed json that you can see here. So I think it might work. Let's see. Why is it minus? Ah, it is working. So you see here It's minus one Let's wait a bit and you see 17 19 All right, so a bit far away And there you see So if we come back to the graph right here Number one if you go to the public view there, you see the plots are being charted with this little guy All right, wait a bit probably have to be very very near But you see if I show you one of the queries that it's kind of working with Where is it right here? So this is an example of a post request that's being done with the field It's minus one. It's very very nearby So there you see it's giving minus one. So it kind of updated the chart Now in this example, let's look at the data here because the iot platform is all about data So what i'm going to do is i'm going to export it to csv file and As you can see minus one minus one and then 19 and then minus one right here So it's immediately available in your csv format and of course it's also available as json right here So let me go back and explain a little bit about the rail structure. So Thingspeak uses things like device and whenever gem and many of the gems that we are very very familiar with But i'm going to be talking about three models That are very very specific for iot platform one of them is something called the channel So channel is something like this is a sensor node So it might have one or more sensors. So all these are related to one channel The second thing is all about the feed. So whenever this Sensor or more sensors kind of transmit something it will record in the database as one feed item And lastly of course is the api key So let me go and also say that api key in the terms of thingspeak is basically a 16 digit alphanumeric number And finally is the feed controller and this is where the data the meat of the matter is the data Here thingspeak, which is i think is really good is Kind of exporting five different types of data html of course is for the view json is very important if you want to be using it and kind of hooking up with apis with other types of applications And of course there's csv if you want to download and do some spreadsheet manipulation and of course text So let me talk a little bit about application. So yesterday if you Remember got free stock. He spoke about smart can So it's the same thing here. So imagine what can happen if you put a proximity sensor Not in a garbage can in a room But let's say in the huge garbage can underneath the buildings And there are many of them and public utilities personnel comes and collects the garbage And wouldn't it be better if each of these is Has a little bit proximity sensor connected to a cellular communication And it just gives a ping when the distance becomes Smaller than a threshold and then it says that hey, you know the logic six people or the The public utilities people go and go to this area and these are the exact Containers garbage bags that needs to be emptied. So that's the application the practical application of this So I have based my example a lot on things speak now There are nine platforms I've played with and I think you can also play with it You do not need to install rails or anything. These are all hosted platforms and then nine of them And whether you say ibm microsoft aws Google they each have a iot platform of their own and out of these guess what two are made of rubion rails So you can imagine some of you in this audience in a few years will be part of an iot platform Building team working with electrical engineers on the other side and interfacing with the sensors While you all will be working to collect the data and make sense of it So inafruit io is one of them and they also are basing their platform out of rubion rails I also want to talk about standards that are being done because standards are so important because right now iot Is a very very new technology. There are so many messy things going around It's crucial that we also follow standards and follow the conversation. So there's web of things standards by w3c I love to subscribe to their mailing list and kind of look at what they are doing They will talk about api endpoints and restful url things that are we are very familiar with it and also some new terminology So go and subscribe to the mailing list So this is the last example i'll be giving and this time i'll be using a temperature sensor And instead of http i'll be using mqtt. So how many of you have used mqtt? So that's like the last Layer of the application stack of the internet protocol and if you use mqtt That's that's awesome. I see a few of you like about five hands in the audience So before I go on to the example, I want to explain why mqtt So like I said http mqtt or even ftp dns these lie on the top layer of the application layer And mqtt and co-app are becoming increasingly important for machine to machine low power communication And the exchange model is a little bit different for mqtt It is based on publish and subscription as compared to http which is a request and response And I think as web developers we are very familiar with this So what publish and subscription model does is that you can do a many to many propagation So you can imagine many sensors connecting to a single mqtt broker and that kind of publishes Are subscribes to other reels service Now here's the interesting thing why is mqtt and co-app especially suited for low power sensors two reasons Firstly look at the header size http has a huge header size as compared to mqtt or co-app And secondly http is also based on ASCII or text whereas mqtt and co-app is based on binary So that really really does a huge effect on your battery So remember I said if you're doing a prototype you're having it at home Sure, you can plug in the battery power and kind of plug it back But if you're trying to do something urban like you know waste management system You have to ensure that the battery probably lasts in the order of the years And that's where using protocols that are made for low power sensor systems machine to machine communication is so important So let's go on to the last example shall we? Here I will be using something different. It's not photon. It's based on esp 8266 And it is the first sub three dollar usd chip. There's a wi-fi chip. So I have a little Uh temperature sensor here. What do you think is the temperature right now when I'm going to on it? any numbers 19 24 22 Who is right who is right? Let's see So I'm going to go here and go to my channel and And let's see So once again, I'm doing it every three seconds, but in reality probably you want to be doing every three hours So let's see. I've oned it Oh, there you go. It's happening You know the stage is a little warmer. I think So there you go. It's it's good. So it's about 24. Who's at 24? Winston Thank you miss. I owe you a drink anyway. Thank you for what you did for the community All right, so that's 24 23 and and So, uh one thing about data is that you might want to have a huge dashboard in your office And that's where web sockets come in also very handy. So once again, it did matters what you do with the data So I want to talk a little bit about mqtt. So in terms of mqtt Like I said, it is a publish and subscription model. So I'm running a mosquitos server Uh, which is a mqtt broker in my laptop here. So once again, I have already Kind of flashed a firmware in my microcontroller here and the firmware So here you will also see how the mqtt protocol works API key slash field which is very similar And then of course, I'm going to my host which is my laptop right here And I'm basically doing a client publish And it's of course talking to my router here And then my rail side will basically subscribe to this mqtt Sensor value and what it does is it will also check exact