Hello everyone My name is Guillermo Rouch. I'm the creator of socket AO engine AO and the founder of a few companies in the past You can find me on Twitter on Twitter or a comment slash Rouch G and I Occasionally log about the things that we've been learning at our different projects on Rouch G.com My experience in the last five years has done a lot with Node.js JavaScript and a bunch of open source frameworks In particular what we focused on early on was making this idea of real-time Collaboration and real-time communication namely chat applications very easy to write so our hello world application When we introduced socket AO back in the day Was basically the most minimal server that would allow you to emit events back and forth and with this event You would be able to create any sort of application, but particularly it was really easy to create a chat application So not only would you be able to emit events? You would also be able to broadcast events and then we started adding functionality relating to rooms and subscriptions And things that come up always in the context of messaging systems But something that we noticed over time was a lot of what we enabled very easily also needed a lot of extra functionality a lot of extra patterns and a lot of new ideas to evolve to become maintainable and scalable In particular when you're working with potentially larger code bases When you're working with larger teams when you're working with Applications that are growing beyond the scope of what you had originally planned So this talk is going to be about the main ideas that have changed Both in the JavaScript ecosystem and in our own thinking about how to design this systems Not only it's just chat applications, but any sort of very Basically real-time a live application So the changes that we've noticed Especially those things that we would do differently if we were starting over again Are in basically three key aspects one of them is the definition when we started This project and we said we were gonna solve the problem of creating real-time applications No one was quite sure what real-time really Referred I think everyone had a slightly different idea then the second big change especially over the last year or two has been the Emergence of the new language features the emergence of really sophisticated build systems That have shaped our code bases differently and then finally and perhaps more importantly the programming model that we use To basically create complex UI that is constantly updating The way we think about data flow the way we think about Making local changes to the application that are synchronized later on have all changed and I'll discuss these three concepts for which I've published three essays over 2015 so All of these concepts that I'll be discussing today. You can Expand on by going to these websites So in terms of the definition one of the most important things to me has been Try to really understand what makes a real-time application real-time or what doesn't make it so and I've come to the conclusion that there are two key aspects that make a real-time application Remembering these two ideas throughout the development cycle or even throughout the process of you coming up with a Product that you want to make real-time are really great indicators of whether your application will be as Successful and as fast as possible so the very basic idea is data updates have to be Communicated to the user with minimal latency Essentially the user has to feel like the data is alive that the application is alive that everything is changing really fast and secondly This data changes Have to happen without the user interacting with the application in order to claim new data So if the user is suspecting that the data is not alive and they have to think about pressing a refresh button Then your application will not be as fast will not be as real-time as possible. So these two ideas That the application needs to communicate with minimal latency as fast as you can possibly make it and That the changes to data have to happen without user interaction if possible If it doesn't have to do with input if someone else changes the data those changes up to be communicated without a forcing a UI Interaction basically boiled down to the up your application being fast and self-updating So provided that you maintain these two concepts throughout the development cycle your application will be real-time Self-updating is something that we solved through soccer day of pretty easily and pretty successfully Because the idea of pushing data from the server to the client Which used to be really complicated really it involved writing Pauling code involved Mixing and matching transport code with your own business logic. We solved really elegantly with introducing events but What has happened throughout this process is that we've sacrificed and we've been unable to define really what makes an application fast so the idea of exploring what makes an application fast is really interesting because What has happened in the development of this really rich self-updating? web applications has been that in order to make The application feel real-time once it's loaded with sacrificed how long it takes to load initially so The idea of a single-page app that has been getting a lot of popularity over time Basically boils down to the idea that you render just one page which contains actually no data. It only it's a shell on to which scripts styles and later on when those scripts load data are loaded on to so this is an extraction of a popular JavaScript framework and the pages that it produces As you can see the markup is actually less than the Google Analytics code This is the entirety of the single-page application once you analyze the code that is shipped to the user and this is a very clear example of the idea that When we're actually trying to enable the creation of this real-time applications There is so much to load that that time that it takes to load initially has been Sacrificed and it's a really bad idea to do this because if we think about how TCP works That initial load time is actually the most crucial to user experience So if you're trying to make your application fast you have to take into consideration that a TCP connection will start slowly When a new user comes to a certain URL or opens an application the TCP socket will start consuming data slowly as to avoid congestion This roughly describes over time Here the x-axis represents time how much data at typical TCP connection is able to ingest Nowadays eight segments at a time is pretty common in most deployments Roughly it means that the 14 kilobytes that you first ship to the user tend to be the most important In trying to seek this idea of creating a really fast application So Therefore if we're maintaining the invariant that we want a fast and self-updating application The most optimal real-time application will necessarily render some or all of its data To this on the server and convey to the user for example through pre-computed HTML so If we have a server-side rendering JavaScript is still very fundamental on the UI side of things In particular JavaScript is what it's going to enable us to later on when that code is loaded when that initial markup is loaded from the server to pick up on the interaction and This idea of masking latency will be enabled to us So the idea of masking latency is very important because when we're communicating We are we have a fundamental boundary. We have a fundamental limitation of how fast data can be exchanged In Physical terms that would be the speed of light So it seems in at first sight that maybe it's not as relevant to us But if we consider even just like trying to communicate from the west coast in the US to the east coast of the US We find through a simple Backup of the envelope Calculation that the best possible case scenario tends to be around a hundred milliseconds with everything considered a hundred milliseconds is actually quite a bit is The threshold at which a user considers something to be immediate and that's of course the best case scenario and that and there are Greater distances to consider So how do we overcome this? How do we? Defeat this limitation. So one thing we can do is debunk general relativity and invent faster-than-light communication Which would be awesome if anyone wants to do it. Let's hack on it later on Or the other two things that we can do that are very very relevant to That JavaScript base user interaction is one we can make optimistic changes to the UI and Two which optimistic changes as we'll see are very very important to chat applications in particular and two we can enable prediction so Optimistic changes means zero latency or the latency of how fast you can update the UI how fast you can update the screen basically Prediction means negative latency. So these are the two fundamental ways that we can ignore that intrinsic network Time that it will be spent sending packets back and forth So this is an example of iMessage. I particularly like this example because I was talking to myself not because I enjoy talking to myself, but as you can see if you can clearly see that the first message is Optimistic because the echo the gray message comes later on So the network round trip actually took quite a bit of more time to the user is a really great feeling though to see that Reaction immediately on the screen. So basically there was zero latency from the time I pressed enter to getting that reaction on the screen and then even the idea of Actually conveying how long it's taking is enabled by a different part of the UI So this is a very successful example of optimistic changes to the UI The basic idea again is To the user input will react immediately So the problem this creates though and that weren't existing in any sort of like server rendered exclusive Application is that now we've introduced the idea of the version copies of the data and this actually increases significantly the complexity of applications because now we introduced The problem of conflict resolution, which might not become apparent when you're starting to create a real-time application, but essentially There's a there one way to think about this is you have now like a git repository and The server maintains its copy where each message is that commit and then you have each user that Basically subscribed to this data And they are able to perform changes directly on their own copies of the data But then those copies of course need to be reconciled later on so One thing to and perhaps a very important point here is that when you're creating a real-time application You have to consider not You have you have to reframe the problem perhaps as a syncing problem in terms of what your application is enabling There is a copy of the data that needs to be there is a source of truth of data that needs to be synchronized with another source of truth as opposed to merely thinking about Messaging basically it's not just about like sending the message across it's about Well, is that message as being sent resulting in a successful synchronization of state between these two points and This is particularly relevant to the previous talk as well because the example that I like to give is in the IOT world a lot of the demos that I see recently in the Socario community a lot of them have to do actually with hardware with devices with robots all sorts of What commonly known as IOT? So the example that I usually give is that of an IOT light bulb which doesn't sound like a good idea But I've seen a few on Kickstarter. So basically I fully internet connected light bulb and you are empowered with the switch And these two things are meant to communicate over the internet So the first thing that you think of is I'm gonna send a Jason packet from the minute this from the moment the switch Happens to the light bulb. So one way to map this Commonly is what you want to do is you create a restful API and and you say turn on this light bulb but now what can happen is that Messages can be it can fail to be delivered and they can fail in very understood ways so very clear ways and At the same time the way that a physical switch works is that The user can actually make the change they can are they're moving something and they're getting that immediate reaction However, the message might not have been delivered at all. So now you end up with a switch that says it's on but the light bulb is not on So what we really want to do is we want it to synchronize that on state between our optimistic source of truth to the eventual device or server that we're communicating with so We can now understand the real-time system is one that rapidly or as fast as possible synchronizes sources of truth that can end up being divergent and This has implications that Have to be very carefully implemented in our applications So for example, one of them is retrying on behalf of the user if that message failed to be sent Immediately we need to keep retrying on behalf of the user we don't need to necessarily present an error right away for the best user experience and Another thing that's very common another type of failure that maybe one doesn't consider immediately is The client might think that the message didn't get to the server So he keeps retrying but the message had actually Arrived, but we had we didn't get the acknowledgement. So we didn't get like half of the round trip So a lot of China applications will have this problem where The user thinks that the message was not delivered actually was and then they end up with multiple copies So the solution of this is actually pretty straightforward Which is we need to consider that new message that we created as an object perhaps with its own UID That's client-generated and then the operation in the server side of things nice to be idempotent So basically whenever we keep retrying if there was already a message with that particular UID the rest of the messages get discarded As far as the second point that I want to talk about the code and libraries How has the ecosystem changed since our initial release in terms of what our tools look like and what are the tools that we are now Enabled is so it's not primarily talking about for example like the emergence of a new syntax But what does that new syntax potentially enable in in the context of a real-time application? so In this regard the biggest change has been the introduction of Eggman script 6 What this means is that if we're like considering our documentation on our home page Where we use Eggman script? I guess three five four We would rewrite it and use new syntax. So here I'm highlighting the things that in particular are interesting in That they're preventing a lot of bugs and that they are making Our code base a lot more robust So some of the things that ES6 introduced have actually made our socket your code base a lot more a lot less error prone The tool that we that we use is Babel Primarily because Babel draws a very clear distinction between what features you're incorporating into your code base For example, if you want to turn on the all of the ES6 features Or you want to turn off turn on some of the experimental ES7 features or if you want to turn off for example Module packaging or if you want to customize the transpilation options is very flexible in that regard so One of the more important changes is modules are syntax. So Whenever we would be writing a socket your application we would use require which is something that I know JS introduced We would basically call this function which is a global with strings Now What usually happens in the context like I was saying earlier with the bundling of this rich applications is Something you will use something like graph certify that goes through your code and tries to extract What modules you're using from that code? So it'll try to do some sort of a static analysis to see what modules you're using and trying to get them from The node modules directory and package them all together But what happens is since require is a function call it allows the developer to introduce expressions That are actually environmental it depends on the environment on the execution context to be evaluated. So For example, if you introduce a variable in this require calls that system, which is doing the packaging It's not going to know how to resolve this dependency So by having a very predictable Syntax-oriented and easily parsable dependency graph We can now start introducing the concept of not just pushing data Which we've been doing with soccer AO, but also the idea of pushing code So if you understand the map of your modules and how they are composed together The future is clearly in considering code and data as basically the same thing if you're Changing the your chat messages or if you're changing a part of the style of the application those two things can actually be transported in the same exact way and this will further reduce latency in this case by Potentially fixing bugs or changing UI or exposing new information The second thing it has to do with something that has helped us create more robust less error prone code which is the introduction of syntactical scoping so Let over var is what I referred to when I mean that var is probably going to be obsolete And we're probably going to be using mostly let or const Depending on your programming style an example is the traditional Forlub where if we were using var the I variable would still be defined later on and potentially create conflicts Finally the other important changes whenever we would use a an anonymous callback We're probably gonna use a new fed arrow syntax In particular because it enables us to access the outer scope much more easily it prevents Bugs that usually have to do with this being a null or undefined And perhaps even more importantly We write less code so we don't have to assign temporary variables like self equals this and so on and Finally the third and perhaps most dramatic change has to do with The programming model so how do we put all of this together? How do we put data and UI together? And presented to the user in the in the best possible way so the traditional approach that we would use is had to do with listening to some event and Manipulating the DOM directly in that callback so in the first example that I showed in the chat application we would Listen to an arbitrary event in this case chat message. We would create a new element and inject it into the DOM What normally happens is as your application evolves that same event might be relevant to different things for example if you wanted to maintain a Counter of how many unread messages the user has You would have to update that same code to say okay now I need to perform this other mutation of the UI and also What's perhaps more concerning is another part of the application might have changed this? new elements and Your code would actually work in an erratic way In fact one of the features that jQuery has is that it silences errors which has been used By a lot of users to perform item potent changes to the UI. So if messages to doesn't exist it'll ignore that operation So what happens is your application will keep growing and a lot of Components might be introduced so your event listener code has now Not only mutated on directly But also read the state out of it in many cases because you're not necessarily Synchronizing data or maintaining a copy of the data then what happens is you end up maybe relying on data attributes on your html as a way of doing bookkeeping of code and another Thing that is really relevant when you're thinking about a real-time application is That you're maintaining a subscription to the server Since you're not just getting the data once you're expecting data the server to give you data in the future You've basically done an allocation on the server side. You're basically Having resources that are keeping track of changes that a particular user is interested in So if the if like for example the UI had changed through other part of your code Then you will continue to listen to chat message events So it's important to now have the idea of tying the Observation of the data to the actual rendering and perhaps to even the actual visualization of the data So for example, I've seen a lot of applications that render many many many chat messages That are actually not in the viewport. So if we're maintaining for example a Log or a history of those chat messages and the user is able to edit them We might not want to actually be subscribed to every single message that has been communicated in that room We may only actually be interested in the ones that are rendered and the ones that can actually be seen Soon enough. So for example, if you scroll quickly, you actually want to Have changes happen very fast to those other messages so and Not only does this Sound like a much better architecture because your UI is now informing what data requirements you have It also saves you resources and potentially money So the basic the basic idea is data is not fetched until the UI actually needs it The solution to a lot of these problems really have to do with the way that you're declaring Your UI the way that you're mapping this real-time data sources to the things that are presented on the screen And the solution to all of this has to do with rethinking the programming model. So By having a very declarative and functional oriented UI, which I call pure UI We can see that we're gonna improve the way that not only We create our code bases and maintain them over time but also how the communication happens in terms of for example if your application changes because of the requirements of a product designer and you need to update your code and presented in a way that you can get received feedback on it or Basically render your UI independently of the server instead of for example launching a Node.js server to synthesize fake events that are coming from a real-time source How can you actually very clearly see what a certain UI would look like with with a certain data in place? So the basic idea here is that we will declare our UI as a pure function of application states so this is something that People have already been doing but we have sort of not being systematically applying In the context of our own programming In fact designers already have been thinking this way because when we create representations of what the UI and what the application is Gonna look like they're essentially these Describing discrete states of your applications. They make entire copies of the UI and then they present them to the developer But once we implement them, we don't really quite follow that approach Recently I was tasked with the creation of this video player That powers the entire wordpress ecosystem so wordpress.org and wordpress.com Websites have received this new video player that I coded in a very particular way to experiment with this programming model So what I did is I created a very clear mapping between data source and UI I didn't Manually specify transitions between states trying to follow that line of thinking of what a designer would do so For example, if a transition happens in the video player where the video finishes uploading and we were displaying a State where it says oh the video is actually still recording or the video is still being uploaded We wouldn't define manual mutations to the DOM We wouldn't ever say okay now we're gonna remove this element now We're gonna add this element in the way that we were doing when we're discussing the chat example Basically that function that defines your entire UI is called again and it describes the entire UI again depending on that in the on the parameters that were Past and like I said, this is very clearly mapped to how designers think so Recently for a project that's coming up in the Sokario world called Sokario evolution This is basically a dashboard that represents our activity in real time so our activity in terms of commits our activity in terms of How many people are accessing the CDN and a lot of other interesting tidbits that? Basically paint the history of the project and how it's evolving over time as you can see when I actually decided in a graphic design tool I Thought about okay. This is what the mobile version is to look like this is what the iPad version is to look like This is what the web version is to look like not only does it change in terms of for example like the actual Width and height of the components sometimes like it requires that you've rendered the data differently Like for example the numbers are rendered differently and so on So what I was doing basically as I was designing I was it was almost like I was coming up with two functions With the same function and only the width parameter was changing. So if we imagine this as plain JavaScript code In fact, I we could even disregard the usage of CSS media queries all together because we're thinking about receiving a parameter and altering our result are altering our output as a result So in this case, I am defining one piece of the component Now what what I observed as well is that this idea of being very declarative and Mapping out all the states that your application and your UI can be in it's not necessarily new. So for example CSS style guys have been doing this for a while This is an example of the GitHub Primer code base where or if you go to bootstrap, you're gonna find the same They're able to represent the actual implementation of their components Almost like you would do on a graphic tool where you say, okay this is what the button looks like in this way and in this way and in this way and in this way and in this way and this has been enabled because we can think of of HTML as a pure stateless Functional programming language in the sense that when we're passing the attribute disabled It's like passing a parameter to this function and when we're passing the class attribute It's passing the small true parameter. In fact the functional approach looks a lot more clearer to me and What I started thinking about as well is we can take this idea of extrapolating and extracting what each Component or part of your application can look like in a very very fine-grained way. So if you consider the I message example We can think of that message Single message as its own function its own component and then say this is what it looks like for all its possibilities for example if The message is sent over SMS. It looks a certain way if the message is sent over a Regular I message connection sent this way and this is what happens if it was acknowledged and that's what happens if You're not the person Creating the message and so on and this is roughly what that Function would look like once again. We're just mapping data very clearly to UI We're returning a representation of widgets with information in it. That's mapping the supplied parameters So it's now useful to think of the work that we're doing when we're creating our applications when data is changing over time as defining a map of Our application not just a map in the sense of okay I'm saying that for this set of data the UI is going to be this but a map by which you can actually navigate Almost the complexity of your application So this was my map when I created that particular widget. I set out to say okay This are all the states that I imagine my application will fall in so for example This is what would happen if you hovered the toolbar or this is what would happen if in the black Once if there was an error or if the video couldn't be rendered because the user was under age So what I was doing essentially is asked answering questions. What happens when this happens? What happens when this new data is received? What happens when the backend decide to make decides to make a change? So I started thinking about like all the possible Common things that one associates with this kind of widget and This is what normally in the conversation between a Person that's envisioning the application and the one that's implementing it's what we normally think of as requirements What I did next as I created that map of my application in a visual way was I try to map it concretely to The code and the implementation in the same way that for example github have done for their UI style guide so in particular I focused on this one example of This particular widget as I said This idea of having a map is useful too because as you're discovering new states that your application can be in it's almost like Zooming in and getting a higher level of resolution into into that which you want to work on and Then I proceeded to define a JSON object that a static JSON file that mapped parameters To the different states that I was interested in rendering so for that tooltip I said I'm interested in seeing what it would look like when the tip is in the middle What happens when the tip is on the left side and so on? so The actual implementation of it actually resembles the design that it did in that I I'm without actually executing any code without actually getting any data from the server. I'm able to Simulate what the application is going to look like later on so this is for example the in the UI what would happen if you press that plus button is you're able to change those parameters and simulate different behaviors and different Situations that your widget or component or UI can be in so for example I had to consider the possibility that if the if that little tip was on the left side It actually had to look different or if the that Position parameter went all the way to the right had to be adjusted as well So this is a tool that's very interesting because it allows us to also think about Potential use cases potential data changes potential user interactions that have not happened yet But we must be aware of to create a very versatile application So I started thinking about our role in the creation of this real-time applications as having two parts one is the design itself which you do by trying to answer the questions that you might imagine are Necessary for the creation of the application and the second part is actually a discovery of new states that You didn't you didn't you couldn't possibly think about in the beginning or you just your experience or personal Thinking about that particular problem did not allow you to see originally and as I was thinking of maps I started thinking about this one map which is What Christopher Columbus was thinking that the earth looked like in in the 1500 in the 1400s? Which is as you can see pretty inaccurate He thought he was setting off to a place that pretty much didn't exist in a world that didn't exist But regardless of what everyone was saying as a matter of fact The map where the book where that map was obtained from said that his maps were actually a hybrid of fantasy and delusion it actually allowed him to Basically accomplish his goal so successful discovery of what holes exist in your application successful discovery of or Accomplishing that which you set out to do can actually happen through random behavior so if you set off to Interact with your application and maybe put it in states that are randomly introduced which is a technique called Bus testing then you can actually improve your code Without actually getting to the point where like a user needs to either complain about it or further requirement needs to be created so I'll show you another state of the application in which the video was converting and So cario Channel was established to communicate the progress of the conversion of that video What I did was in my tool that showed me that rendering that allows me to for example customize the width and height of this Video I can now introduce randomness into my testing So if the key of that data ends in URL I decided I would source images randomly from the internet to see how my video player would behave this particular technique in fact was so useful that That progress bar actually had to update because I didn't know would actually how it would behave when there was a light background When there was a dark background So this idea of introducing random walks of that state space Gives us a heads up in our implementation What this model also allows us is to have a much better understanding of Completion so going back to this example something that's actually missing here is a state of What would happen if the sender doesn't realize that it was an error We go back to the map. There is no even understanding here that an error could occur So and this is why he insists that this model also improves communication almost creating a protocol between That person that defines the requirements and that person that implements it because we can now say okay you have to implement the State when the message has not been delivered. It's being retried and so on and This also leads us into a better understanding of correctness because what we know from working in a lot of rich real-time applications is that a lot of errors can occur over time such as Being disconnected from the internet being in a reconnecting state or for example The message we know has failed to send because we timed it out and it has to be manually retried so this enables to actually now look back and Look at our initial map and then look at where we ended up in terms of adding new states and contemplating new possibilities in the UI and We can we can say okay. What was my understanding of the complexity of this project? Was it accurate or was it not accurate? and it also allows us to look forward because Essentially the work is never done This is the map that Christopher Columbus created after his successful initial trip ended up still being highly incomplete, but it was inevitably a Good positive step into the future. Thank you so much. I think we have time for questions, but Hi What do you think about Dom going functionalized or few of the concept of using the functional programming? with Dom in terms of being stateless and immutability So can you repeat the question? What do you feel about the Dom going functional as in using the functional programming paradigms when we are manipulating the Dom? Yeah, I think a lot of the problems that we've incurred into For example, like why were we mutating the Dom directly in our initial examples have had to do with the? APIs are naturally Imperative and non functional so it's almost like the hello world of the web Doesn't lead you to this kind of workflow and I think one of the solutions has to do with Maybe not making it functional, but maybe having frameworks ensure that you're programming in a certain way Even with a lot of frameworks that do enable this kind of programming you'll find that You can stay away from functional programming patterns very easily you can still incur in the same kind of mistakes and that's why I Referred to more importantly as to the programming model and not so much to a specific frameworks Because we have to think in a functional way to accomplish this and in a way that is side-effect free So whether Dom is the rendering target or whether it's something else actually is also not as relevant Because you can you can imagine that once you have a certain mapping between data and UI Then the rendering target of that UI can be many different things. It can be the Dom It could be WebGL and so on but again, I think a lot of it in the in terms of the JavaScript world will have to do with a Part of it discipline and a part of it frameworks that ensure This kind of constraints. Yeah. Thank you Hey back here back right back Yeah, oh, yeah great talk. So your idea of making things fast is by making the better design and Providing data only when it's required, right? These are the two main approaches that you've talked about What about Machine learning and also understanding how the user behaves And then preempting data. Is that is that also a possibility? So the second point that obviously didn't go into which is we can have zero latency through optimistic updates But then we can go into the realm of negative because we can present to the user data that they still have they haven't requested yet Because through machine learning, for example, we could understand their patterns of behavior one that's actually fairly straightforward to implement that doesn't require machine learning, but it still allows us to predict Involves doing collision detection between the cursor and the and the target that we know will be fetching some data in the future So if the mouse is heading the direction of a certain button, the data for it can be prefetched In fact the declarative model would allow would allow this in a much better way Because you would simply define Attributes that have to do with like okay, so this piece of the UI is interested in registering a collision detection with the input to enable Prediction or some sort of like data retrieval and caching ahead of time a part of the Prediction model also has to do with the server side of things. So a technique that's interesting is analyzing the referer Data on the server side when resources are requested to push them with HTTP to Before they're even requested. So for example most applications will Load index dot html and subsequently they will load logo dot PNG now You could manually declare on your server side of things okay every time I push this push that or You can actually remove it one layer further away and you can say okay So if logo dot PNG is consistently getting required through referred data in the headers and it should be headers Alongside this page then we can assume the user is going to request it and push it automatically So I think prediction is definitely the next big frontier Although I wouldn't necessarily go there until a lot of maybe more pressing issues are solved like the idea of Pushing code or treating code as we treat data in terms of how we convey the updates to the user I think in my opinion would take higher priority on that Hello Awesome talk so as you said like describing the UI in Discrete states, but we need to switch between those states based on the user interaction right so normally How we would do it like add or remove a CSS class But in your case I saw like throwing away all the markup and like creating the widget again was was that correct? So the basic idea is that you're never changing a class you're saying okay Here's my entire UI and now because of this parameter having changed the class will be there or not and then the engine Which in certain cases for example for react it could be the virtual DOM for ember It could be template at the template engine knows which parts of the templates are dynamic so they can do basically they can perform the changes on your behalf and this is a very powerful idea because The way that I think about it is kind of like manual memory management and garbage collection A lot of the code that you would normally write you no longer right But in addition to that you're now declaring the mapping between data and the complete representation of the UI which allows you to basically visualize it and Change the data and see how it would perform even for example with without setting up a actual real-time socket You would say okay show me what the UI would look like with this certain parameters with this chat message having failed to be delivered and so on I Have another question. It's related to socket. I am so the next version of HTTP HTTP, too So it brings a lot of great things like server push and all so how is that affecting the development of socket? So that's a great question HTTP to already has for example like the functionality to have a bi-directional stream And in addition It does something very interesting, which is like every request and every response goes over the same socket Which enables like very efficient for very efficient restful requests and response cycles. It also compresses headers Which means that One of the features that WebSocket was known for which is minimal framing basically minimal separation between messages and a very lightweight protocol you can still do with HTTP as well because things like user agent and Host and all of that once the connection established it becomes part of the compression state and are not sent over and over and over again so With this in mind what we have to think about is that to send data from server to client? It's not as straightforward because it assumes this idea of Resources that are known ahead of time or are going to be requested by the client in addition in terms of socket AO we always try to use the best possible transport for every requirement or every connection or every For example in the presence of proxies we have to like drop down to polling and other techniques, so HTTP to if it's enabled it's something that we can add in basically optimizations to the code for But ideally it's transparent to the user because what we find as well especially in in terms of applications like this is The event approach Happens to be happens to fit really nicely into the majority of applications you might create So it's not that your interest is so much in resources I think of it as being interested in events for example when you request the initial data for The chat application then you will be interested in events that alter that data set that you're interested in so Get and get and post are not Maybe as interesting as if HTTP to supported something like watch Where is the idea of watching a resource which is essentially what we're trying to do here is we're saying I Need some initial state and then whenever that changes we're gonna get some event We're gonna alter it and then we're gonna repaint the entire screen Thanks. Hey hi here Really inspiring talk by the way One question around the pure UI So you mentioned that you have thought about all the states that the video player can be and that way you can write a Pure function which represents all those states, but the moment you sort of start composing them to build more complex you guys The set of states that you you can be in eventually grows exponentially. How exactly so so in those Sins the moment you compose such pure you guys to build more complex you guys How exactly even you can test or maybe think about what all states the application can be right? So that's a very interesting idea right because like it really shows that even when we're in the design process We're dealing with something that can grow so much that is it would be even overwhelming to see My vision for that would be that you can actually navigate it as an actual like Google Maps We're like you could actually zoom in and you would have a new eye that has an idea of like resolution of what you're seeing So maybe you're not interested in every possible state or maybe they're actually infinite, but this idea of Clearly there they're there being a clear connection between the size of what you're dealing with and the actual UI that's showing That complexity that connection. I think would be really useful to understand for example like how complex your application is now in terms of composing so what I mentioned about going fine grained is We are basically defining a function Arbitrarily perhaps but for each part of the widget or each part of the UI that we consider to be a unit of its own So in the case of my map I said okay the tooltip is going to be its own thing Perhaps I could have said the tip at the bottom is also its own function So you sort of arbitrarily decide what what things are going to be the building blocks of this UI now Naturally, I think they map very well to how you think about the application So if you again go to the shoes of the designer or the person is envisioning the app They're probably going to think of like okay. So when I hover this this new thing is going to show up So potentially that new thing will be its own new function that will return some new piece of UI In terms of the state space being very large potentially infinite That's why I think it's so important to have the ability to play around with the UI to see Random data being fed into the UI An example is in in the little video that I showed about the width and the height being altered When designers think about apps they should think about breaking points So they say this what it looks like for iPhone This is going to look like for Android if you look at the emergence of different mobile phones and resolutions It actually be daunting to create an entire map for that and even if they did There's still scenarios where responsiveness is desired because people are dragging and draw like dragging the browser screen itself Right, so it's almost like you want to see every everything and That's why I think the idea of fast testing to be really interesting because like you could see this thing change by its own And then potentially perceive a problem and stop it and correct it in fact Something I didn't mention here was if we're thinking about this Very clear mapping between the entire state of the application and its corresponding UI time travel becomes possible I had a particular issue with this implementation where for a certain time of the video Like I was watching a video and so like oh the number just exploded and it had to do with a bug in my logic of using like Numbers in JavaScript. So basically I was getting like scientific notation for this one conversion of time into like humanized time So the only way that I could effectively Debug that was I had to go back in time and see like okay. What's the particular data? What is a particular number that was giving me problems in the conversion to this string? So that's the other benefit is you can Basically in this discovery process stop and say okay, like let's look at this and like let's develop it further The other fundamental idea is that when you're developing Instead of thinking about like fake data or like for some of when designers think about like lorem ipsum and like filling everything with lorem ipsum Now we can see how we could actually populate data from like real data So if you're developing a chat application, you can actually like develop like feed it real text that people could be writing in Different languages. So essentially giving it a more real-world testing approach Hi So you talked about pure UI mostly from a design point of view How does this affect our coding style and the coding architecture? Yeah, so Like I said the main thing to consider here is that The way we think about the sign and the way that we think about programming this UI should certainly match in the sense that if if you if you for example like see a map of that application and You've already implemented it and you haven't done it in this way Getting to a particular state of the application would actually mean for example like loading it Then perhaps if it's a chat application You would go to a certain thread and then if you want to simulate an error you would have to for example Simulate a network connection error or turn off the internet or something along those lines, right? so it becomes really difficult to actually establish a really good connection between what what the Applications intended to do and to actually see it do that particular thing. So as far as coding style goes It's perhaps less important. Like I said, you could develop this using ember templates right But what you have to maintain is this idea that there will always be a mapping between a certain type of data and a certain representation in a way that's pure so basically rendering that one time and time and time and time again calling that component or Triggering that function call should always return the same thing and that's basically the idea behind Purity in this is that there is no for example in that In the chat example, I'm not calling the Asia I'm not like getting the data from like an Asia X request in the context of that function first I started with the stateless view and then I separate that very clearly from how the actual data of the application during the life cycle of the application is retrieved something to concern in terms of coding style is accessing the parameters or declaring what parameters the function needs in terms of for example Saying the way that this piece of data is gonna be obtained will map this resource is very important because a common problem with applications is they will request way more data than they're actually rendering on the screen right so like You make an API call to a restful endpoint to get you and you only need to render the name of the user And you actually get the entire profile, right? So that's a very common approach like actually rendering UI you get way way more data Than you actually need to render so at some point in your application You're gonna have to define okay in order to when I'm gonna call this function that's gonna render a certain UI component I have to think about like okay, so that particular name maps to this particular object on the server side So perhaps what we're gonna see in in the next Years is we're gonna either be evolving rest to a point where It understands the precise needs of the UI in terms of like I only need this key or I only need this or the evolution of an entirely new protocol that Has this idea of getting it only as little data as possible and then it's staying subscribed To that to that data changing in the future So and that I think will only be enabled if you're maintaining this Approach or it could be enabled in other ways, but this is gonna make that transition to a more Efficient future a lot more easy and painless First of all huge fan What I wanted to ask you is that what are the What are currently the Challenges that you're facing or the problems that you're working on it for socket dot IO some some Unsolved challenge because they're on 150 developers here. So like someone might have an idea Definitely, that's a that's a great question. So glad you asked So one of the most important things moving forward is we want to migrate the entire code base to ES6 We want to make sure that we're not relying on we that we're relying more on Eggman script in the sense of or or or for example like what browser APIs not through npm modules But directly assuming that the API exists and then statically analyzing the code base to Polyfilling those features in so when I was mentioning that one of the big barriers to creating an actually fast application Not just fast in the interaction, but fast from the very instant that the user loads it Is you have to minimize the Footprint you may have to minimize code size So if you're loading like 300 kilobytes of the UI framework and you're loading a hundred kilobytes of the real-time framework And you're in addition at loading your business logic that is maybe perhaps shared between server and client That's gonna have a very detrimental impact on performance on making that application fast so what we want to move towards is a Minimal code base that assumes all the most modern features are already there and then selectively polyfilling depending on browser capabilities and producing different bundles so I Was recently in a Presentation where I was talking to developers and of a large deployment of soccer AO And they had found that a lot of their users Because they were in the analytics space like collecting analytics from websites a lot of their users were still using i6 So prior to that conversation had actually thought about like just dropping i6 because it's oh it's too old But that kind of changed my mind that we still have to have a path to supporting the past We still need to have this ability to polyfill for example web socket into the socket your code base or through ngin.io and Not making assumptions about like every future being there So that's one of the major like next steps for we haven't started working this so that's something we're gonna start working on Very soon and contributions would be most welcome and that's probably gonna be expected for 1.5 and Around that time the next big goal for the project to is as I mentioned this idea of only Retrieving data when it's observed is a very powerful idea. So We might want to in the future think about something along the lines of Direct observable Support inside socket your core now And this would be more in the realm of like 2.0 But the idea would be perhaps that you could Selectively say I'm listening to this Event or not right now that involves like you maintain do you're doing your own bookkeeping of events So you would basically emit an event to the server to say that you're interested in an event And then if you're not interested anymore, you would emit another event So it is essentially possible right now, but this what I suspect is a very common pattern Which is okay now. I'm interested in this. Oh now. I'm not interested in this anymore we can make a lot simpler and The API in particular that would be relevant to us for this is observable now observable is still in the early phases of discussion in terms of Specification so we wouldn't want to Invest too heavily in something that then is not gonna play nicely with the rest of the ecosystem But that's one of the next frontiers for us perhaps Observable could be even a more powerful starting point for Any application in general perhaps more so than event emitters now event emitters are great to have in core because on top of them You can implement virtually anything including streams So streams we never supported directly because it would increase our clients at footprint and you can easily bake it in through a Plug-in to have stream support directly as a basically soccer event The hands you a stream object now with observable things are different because if we have an API that It's both in modern web browsers, and then we can complete it later on then It can tie directly into for example UI frameworks. So if we wanted to render a List of chat messages, we can just tie it. We can just basically pass this observed event stream to to the UI rendering and then like it all becomes a very easy to glue together In fact because observables are lazy They're they already would do the job of like being good at resource consumption if The UI doesn't need it anymore. It cancels the subscription to that observable value and then The resource on the server is cleared. So to me that's very exciting because it solves a lot of problems It solves the problems of perhaps like coding with asynchronous streams of values It could be a very powerful standard for composition perhaps way more so than promise and Again, it's something that can Bring so that you're closer to being a more complete framework in relating to UI without trying to do too much and staying very minimal But yeah, I mean we have a very very very large list of things that project that I showed earlier evolution to basically our Dashboard for showing progress of the project because progress in the project of soccer deal lives in so many different places For example, we recently announced soccer deal peer-to-peer which is a project for Essentially applying the same idea of soccer deal falling back to different transports, but for peer-to-peer So if we can we establish a webRTC connection if we can't the peer-to-peer goes through a server Now it's becoming increasingly hard to discover All the different modules that we're maintaining we also have implementations of the client in so many different programming languages now And so one of the purposes of evolution is to communicate that progress and that like universe very effectively to users Something that comes up a lot and opens those projects is like you go to a certain module like oh, it hasn't been updated But maybe it hasn't seen a different branch and so Communicating it's a meta communication of the project itself is very interesting another Thing that we're developing is soccer deal insights. It's basically a middleware so you go IO that use insights and What it'll do is it'll aggregate information about the socket your connection and send it over to another socket your server That's going to do the analytics aggregation What we found is that socket yourself has a lot of it's collecting a lot of data that is super useful Even for business goals So how many people are connected to your website right now is a question that for most people here They they either can't answer it very easily or they would have to go to a third-party tool like Google analytics Whereas the data actually is already in your server the number of people that are that have opened Unique socket your sockets is already giving you a very good idea of the activity that is in your application even once they're further What events are being sent most frequently so that allows you to have a very good idea of What parts of the of your application are actually more popular what parts maybe need to revisit Need to be revisited in terms of scalability.  So I like this idea of like using the tool like I Analyzing the real-time framework in real-time. It's pretty cool. So those are like the big things like big picture things that we're working on in the next perhaps year so most of them have so either a github issues or We're constantly recruiting volunteers. We have a slack channel in which we have created a specific channel for each of these projects So evolution insights and in the core channel we discuss the next steps in terms of working with us mentioning with Browser support and optimizing in the code base and I can script 6 and other tasks like that So it sounds like a lot and I could keep going but I mean there is work on all basically in all parts of the Project so if you're interested in UI design and you want to like Contribute that part of the application that's or the applications it would be super useful to get help there We're we're currently developing a new documentation system as well that we hope to roll out Very soon that will be integrated with github so that people can update it very easily And we're gonna dog food a lot of the techniques that I described today in terms of how we communicate data changes To the client how we even communicate code changes to the client So the idea of code and data being sort of the same becomes very clear when you're working out with for example like github Websites are based on github push because when you push something you want to see that reflected on on the screen it doesn't matter if you changed the code part of the Project or if you change the data part of the project. So we're gonna be Using this technique for our documentation And that way we can also increase the rate of updates. So right now. It's way too manual We would like to get in a lot more comments from the community around certain topics around Different API's and best practices and patterns that they found Also, since we have a support for such a wide range of frameworks like unity 3d Titanium there are so many clients right now that it's almost like each of them requires their own universe of documentation So our goal is to bring in and have one centralized resource for all the all this different sub projects Each of which have their own documentation with their own API's with their own best practices with their own troubleshooting and and so on and this is why it's taking us a while to come up with this because We've seen that the problem is actually quite large to tackle But it's probably gonna be one of the most exciting announcements for a lot of people Just one more question One of the I'm really sorry. We've been like you can just take all of these questions offline.  Yeah, I Drunk a lot of South Indian coffee like filter coffee. So have unlimited energy basically Away special thanks to Sequoia for getting him here I mean talking about this coffee the whole conference is I got here I message all my friends in the US. You have to try this coffee. Yeah, so if you want to be really productive just say Well, you're probably already doing that But I'm gonna communicate at home as well so but Do we have time for more questions or I? Think we can do one last question, but from someone else Because I had seen yeah, I had seen him raise his hand Firstly, thank you for making a life easier with soccer at IO and mongoose and everything else My question is basically so let's take an example of Twitter web What would you want to load in the first 14 kb of the data that you're sending to the user? What would I load? Yeah, for example for Twitter web What would be the first things that you would yeah, so that's a very good question because the answer is actually non-trivial What the main idea is that we want to present the user the data that they're interested in right and Nothing, but the data that you're interested in the web here has a really amazing advantage because when you're using hyperlinks The the way that you get to the application is already Informing what data the user is interested in as opposed to like applications where like I tell you hey open snapchat It's different for me if I give you a link to a particular Portion of the application so in the first place and the most important part of it is you want to Preload the data that's related to that particular URL Now if you have a framework in place that does seamless server and client rendering That's easy to accomplish perhaps But if you don't what you might want to do is you might want to for example in line The data that the application will need once it loads So it's not as optimal as rendering HTML and CSS with the particular data the user needs But perhaps an evolutionary approach Instead of rewriting your entire code base to be pure functional and do server-side rendering and like all of that You can embed the JSON data that the subsequent script is likely to load Based on the URL now. That's one approach Another one is a should be to that someone asked about so You know that a certain URL with perform certain JSON requests for example, so you can use the HTTP to or speedy push feature to say, okay Here's a URL and here are the other resources that your this URL is invariably gonna need and there are some ideas for automating that but even if you did it manually it'd be really easy because you essentially have Like in express for example, you say rest that sent you can now say rest that push and specify a route So that's a great way of like and this is why I fundamentally really like HTTP to because it gives us an evolutionary path to improving our applications without like recruiting 20 new programmers to work on the hottest latest framework now The ideal solution and the one which I don't have a perfect solution for yet Is you actually want to render as much as the user's probably gonna see so you could say that maybe you have to render Default of the screen size that you actually don't know yet. How big it's gonna be right so There are certain challenges there that would if we want to like really optimize it would need to be looked at But one thing to consider when doing this seamless rendering approach is Since the input model is different Between a pure server-side rendered application and one that also uses JavaScript In particular when you click something on when you press a button or when you start typing The implications of what could happen are different So if if you're using a traditional server rendered application like Wikipedia, you can safely render everything from the server and Whenever you click the browser is the one that's in charge of Deciding what's gonna happen next so that's why the browser comes with a history system and loading Mechanism they saying that the page is busy and so on but now that we have we're including JavaScript in the client Maybe a lot of people render anchors that actually have That go nowhere or buttons that need a lot of code to be loaded to actually work So that's why I don't like the term isomorphic Applications because it's never actually gonna be the same nor should it be the same So when I say server rendering say part of all of the page Because in certain cases even if you rendered all of the page you'd be doing a disservice to the user experience because Certain things are just not gonna work and the user to the user is not gonna be clear why they don't work Oh, it's because it's just who hasn't loaded yet. Yeah, good luck So one has to be mindful of even What would happen if only a part of that html is being rendered since a lot of web browsers kick into Progressive rendering mode. So my advice would be to have Basically code paths that say if it's being rendered on the server Maybe exclude this portion of the input and load it later And again in the declarative model that would be easy to do because you would say Ignore server attribute or something like this on a portion of the widget and then when the entire UI is being rendered on the server side that's excluded and when the client side picks up all Of those things that had that attribute are now rendered So it's a seamless way It's almost like you would imagine that you see the server rendered page and it's missing a few things that Require a JavaScript and then you seamlessly see them pop in the minute that script finished interpreting and loading and Interpretation time is also something to consider as well, right? So the advantage of a server rendered application that's giving the user the data they're interested in has the other advantage that Interpretation time is actually not free, especially on mobile devices It can take a very long time. Even if it's been cached That's why it's not only about the transfer time It's also about interpretation time. It's also about What the code is actually doing which could be potentially slow like dipping is as in terms of CPU It's an expensive operation Especially if it's you're dipping an entire tree of server rendered widgets with a new Tree that the client side has created so one has to be Mindful of not just the code size in terms of network transfer, but also like what's actually happening once it's loaded Yeah, all of the questions are playing, please We should Thanks a choir for getting him here making sure he And thank you for that very interesting. All right. Thanks a lot guys and if you have any Feedback about the talk or any questions about the pro it's a video project that I mentioned just email me at route to g at gmail.com with the subject JS food and I'll get back to you as soon as possible. Thank you