Thanks for having me. Hi everyone. So my name is Francesco Serini. I started off my career as an intern at the computer science lab, so with Robert, Mike and Joe. And, you know, being very passionate about airline, but not just airline per se, but a lot of the features which came along with the language. And I'm going to get back to that later on. Having left Ericsson, I went ahead and found that what is today become airline solutions with offices, well, pretty much all over. There are five offices right now, three more opening this year, focusing on building distributed massively scalable systems. And, yeah, the reason I'm here today is, well, to bring a very, very important message. When C++ came along, everyone came and told you, hey, guess what, you know C, learning C++ is easy. Java came along. And when Java came along, they came in and told you, hey, guess what, no C++, learning Java is easy. Come over to the dark side. What are the Scala people doing now?  Learning Scala is easy. And on and on, I keep hearing, you know Ruby, guess what, learning Alex here is easy. Beware. And I don't want to scare you, but, you know, when you're learning C++, you need to change the way you think and reason and how you program structure your programs. Java, going from C++ to Java, the same. And Scala, you know, from OO to an OO, functional paradigm. And very much the same applies to Alex here. You know Ruby, the syntax might be familiar. It might make it easier to get into Alex here. But that's not where the journey stops. And, you know, the real power of Alex here is, in my view, I think the concurrency model. There's a lot of functional paradigms, and I think Dave has covered them really, really well. What I'm going to focus on today is, you know, the concurrency and how you start thinking concurrently. But I'll also be covering the distributed aspects, the architectural aspects. And I'm going to do that using Erlang examples. The good thing is, you know, and I'm going to use Erlang examples from the last 15 years. So, by explaining how the Erlang runtime has evolved, how the virtual machine has evolved, and how, you know, the more powerful computers we got, the more powerful VMs, the more features we got, we started doing things differently. So the way of thinking in the Erlang world has evolved in the last 20 years. And it will continue evolving and continue changing, because the problems we solved 20 years ago are very different from the problems we're solving today. And, you know, I do a lot of talks, and for once, I can say I've actually not prepared one single slide for this talk. Actually, the cover and the last slide are the only ones I've prepared. All the other slides are taken from other people's presentations. Okay? And I will try to remember to acknowledge those presentations when we look at it. But, and the reason is, well, looking at the architecture, if you go back 15 years and you start looking at these architectures, that's where those presentations came from. Not only, this isn't the first time I've given this talk. The first time I gave it was in 2009 at QCon in London. And, you know, quite a bit has changed since. The name of the company called Erlang Training and Consulting, we're today called Erlang Solutions. Lesson learned, there is never name a company after the services you provide, because as soon as you do that, a couple of weeks, months later, you start providing new services on top of that. The logos changed. We, well, took in a graphics designer who, you know, convinced us to pay them a lot of money, because the logo wasn't elegant enough. And so, you know, this is what we've got today. And, yeah, my titles also changed as well, but it wasn't in this slide. Now, this presentation I gave a few months after I'd returned, you know, from a very first of Oskon. So, the first Oskon Open Source Convention in Portland, which Jose and Robert and quite a few others attended this week and flew down for LixirComp for, and I just come back from Oskon at and at Oskon during the keynote, Tim Bray, who at the time was the director of web technologies. So, in front of thousands of people went in and flashed this slide right here. And what it showed was, you know, Joe Armstrong sitting on a Royal Enfield. It was a motorbike he discovered they're still producing in India and then adapting them so that they would conform to EU regulations when it came to, when it came to, what's the word I'm looking for, the exhaustion, so that, you know, they would conform to EU regulations. And he went in, he flashed up a bit of airline code and then went ahead and with the following quote.  And an airline does require some fairly serious mental readjustment. At that point I was just like, okay, Tim, I'm watching you. But, you know, however he added, you know, having spent some time playing with this, I tell you, if someone came to me and wanted to pay me a lot of money to build a large-scale message handling system that really had to be up all the time, could never afford to go down for years at that time, I would unhesitate and choose airline to build it in.  So, you know, after having given this quote, you know, someone sitting right next to me goes, yeah, yeah, yeah, yeah, the syntax really turns my brain inside out.  And what he failed to do is see beyond the syntax and actually look at the semantics. What Tim Brain was talking about here was not the syntax. You know, what really requires the mental readjustment in the way you do things is the concurrency model. And you need to start thinking concurrently because that's how you go in and you start structuring your programs. Was it anything new for me? How many of you know how I first got into airline? There's a few, but I got into airline in university at Uppsala University in 1994 in the parallel programming course. The teacher came in, you know, waved the first edition of concurrent programming in airline, said, this is the book, read it. These are the exercises, do them. And that was it. And off he went and lectured into the horrors of parallel programming. Dead blocks, mutixes, semaphores, you know, critical sections, race conditions, you know, all of these things, which really scared us. Now, we started working on the exercises. And in this exercise, we had to write a simulated world. We had carrot patches growing, and then we had rabbits going around eating the carrots. And if a rabbit ate a lot of carrots, he got fat, he would split in two. If there's a politically correct Sweden, yes, here, we're dealing with. So if the rabbit didn't get enough carrots, he would run out of energy, would starve and disappear. Then we had wolves, which would go around and look for rabbits. If a wolf ate a lot of rabbits, got fat, would split in two. If a wolf wasn't able to catch any rabbits, he would die. And there was also a lot of intelligence displayed, you know, if a rabbit saw a carrot patch, he would broadcast all the rabbits within a certain range, just saying, hey, come eat their carrots. And the same, you know, if it saw a wolf, hey, warning wolf, wolf, and all the rabbits would start running away. And the same with the wolves, if a wolf saw a rabbit, it tell all the wolves, hey, there are rabbits here, let's go chase them. And it took me 40 minutes, sorry, 40 minutes. It took me 40 hours to get, you know, this lab in, completed with documentation, had a nice tickle TK screen and you had dots going around. So it wasn't that much. And what, you know, every carrot patch was an airline process. Every rabbit was an airline process. And every wolf was an airline process. And I, you know, it was really easy to model, really easy to conceptually think of. And when, you know, and I clearly remember going in, these were on HP workstations we were using, which at the time, I think, allowed a maximum of 128 threads. And I remember going into the workstation and typing in ps-ef, expecting to see a thread, an OS thread, for every, or a job for every rabbit, you know, for every wolf. And instead, there was only one, which was, you know, the thread of the jam, which was the virtual machine we used, the airline virtual machine we used at the time. Jam standing for Joe's abstract machine. And that's, you know, that's really what I thought, oh, this is kind of cool. You know, and then I think that much about it, until a few months later, we had to solve exactly the same problem with an object-oriented language called AFIL, invented by Bertrand Mayer. And, you know, for that exercise, it was exactly the same exercise, exactly the same specification. Only difference is that they paired us up. So there were two of us working on it. And despite reusing a lot of the algorithms, despite having solved the problem once already, and it's my view, an engineer never really understands the problem until he's actually solved it. And once he's solved it, he'll think of a million better ways to do it. So we reused a lot of that knowledge into it. And despite there being two of us, it ended up taking us 60 hours to do. So three times the amount it took us to do it with our language. And that's when I started thinking, in terms of there's the right tool for the job. You know, and there are wrong tools for the job. And that's just in itself is a completely separate talk. But what I hadn't realized, the whole concurrency model, what you have is three things. And they've been covered already, but you've got process which don't share data. Processes do not share data, they communicate through message passing, and they monitor each other for failure. And I think those are the three key ingredients, which give you this whole concurrency and the whole ability and the fact that it split the processes from the underlying OSFRED. So see the early virtual machine as an operating system on top of the operating system. So, you know, left university, joined the computer science labs, had a great time there, after which I started working with Ericsson's training and consulting arm, helping projects start using airline. And at that point, you know, the year was 1996. And they just started the AXD 301 switch project. This AXD 301 switch project, I think was, was one of the many airline success stories within Ericsson. It consisted about one and a half million lines of code. That was the first release. I think now it's way beyond two million. And about half a million lines of C++ code, that's a bit of Java, JavaScript. And the way the approach takes the AXD 301, and the way Ericsson approaches any airline project is designing by prototyping. They never start. So when you've got these large corporations, and they kick off a project, they'll have 100 developers. They'll usually maybe have 200 people on this project. And there are about 100 developers and testers, and then there are 100 other people, no one really knows what they do. But I'm sure they're important in some shape or form. And when you put 100 developers, you know, you need to make sure that what you've got is solid. And so before even, you know, committing to these vast resources, they'll always start designing by prototyping. I mean, it's not good enough, you know, to quote one of my mentors, Mike Williams, it's not good enough to have ideas. You need to make sure they work.  So just like any other, well, this was the first major airline project within Ericsson. And but you know, this, this this kept on going. What they did is they went in and started deciding, you know, how do we actually model the concurrency? How do we model the setting up and tearing down of these calls? So the XD301, it's an ATM switch. And you set up calls and you tear them down. That's a simplistic view over what was happening. And remember, this was the first time we had access to a virtual machine which had an incredibly powerful concurrency model. Back in 1995, I think the limit, the VM limit was 30,000 processes. You could have 30,000 processes running in your airline virtual machine. Compared us to, I think we had the 128 threads, OS threads. I mean, the delta was huge. And so we started, the question, they started asking themselves, the question we're trying to figure out in this proof of concept is, how do we use this concurrency to our advantage? We, we had no idea. No one had ever done it before.  Now, the goal of the XD301 is that for every processor pair, you had to have, you had to get in 30,000 calls. That was the goal. You needed 30,000 calls to be handled simultaneously in a pair of boards. Okay. You're following me here.  So they went in and the first prototype, they basically did six, you know, fairly complex finite state machines, each running in a single process. And worked well, you know, took a few milliseconds to set up the call, a few milliseconds to tear it down. The problem is, if you had six processes, that meant you could only have a maximum of about 5,000 calls in your system at any one time. So didn't solve the problem. So they got a pretty good programmer who started merging together all of these finite state machines into two processes. And so that resulted in two processes per call.  So someone came up with a, hey, you know, let's merge these two finite state machines into one process and have only one process deal with all of the calls. Then we go in and we actually save the state in an ETS table. So, hey, set up the call. The process would go in, it would do all of the complicated calls set up. Once it was done, it would save the state into a table and then take in the next call. So, you know, two problems arose right here with that approach. The first problem was, you know, setting up a call, all of a sudden became a sequential bottleneck. It became the sequential operation because you could only set up one call at a time. You had one process setting up all the calls. So you got a bottleneck and that, and the second part was that the code was so complex it became completely maintainable. You really had, yeah, you didn't want to go there. It was just, you know, maintainability is critical when it comes to telco products. You've got, you know, 80% of your costs are usually support and maintenance. So that wasn't an option. So someone came up with the idea is, hey, how about having two processes per call transaction? So, you have one process when you set up the call and then when you've set up that call, you store the state in the table and then you terminate that process. And when you're done and you need to tear it down, you sport off a new process, make that process, tear down the call after which you store the new state in the table. And that's really where, that was the point where they started thinking concurrently and they started embracing concurrency the way they should have. And from the two process per transaction, your processes became cheaper and cheaper and they eventually moved to four to five processes per call transaction. You know, that meant that at any one time with four or five processes per call transaction is that your limit is at any one time you could be setting up about five, 10,000 calls. But that, you know, the number of simultaneously open calls all of a sudden, you know, became dependent on your memory and on your physical constraints of the machine, your networking, IO and so on. And that's the way we do it. You know, that's the way to think. Set up a process for every truly concurrent activity in your system. And that was in this case pulling up the call and tearing it down. And that's the architecture that ended up sticking with today. So this is on the concurrency side, on the distribution, your very old telecoms that had an active standby. So every time you set up a call, you went in and then communicated to the device board. And then there is that set up the end-to-end connection. When you were done, you sent back responses. And those responses were copied both to the active board and to the standby board. So in case the active board terminated, all the calls which were currently being set up would be set up successfully with the results being copied to the standby which would then automatically become the active in case of failure. So you know, that's how you got your no single point of failure right here.  So this was the first kind of the first major project which worked. And where it started embracing concurrence in the concurrent way of thinking. About a hundred developers and I think it's still selling today all over the world. Now, if we fast forward another five years, this is another very interesting case study where doing a consultancy, I got called, this was my first customer outside of Ericsson. So I went to Paris and started working with a group of developers who were doing an instant messaging proxy. So at the time you could have a maximum of about 512 simultaneous open socket connections on a single machine. And they wanted to increase. So that meant that if they started running Jabber servers, there was from Jabber Inc. There were open source servers, which were available at the time. That meant that for every server you could only have 512 users on a single server.  So this proxy would handle all of the TCP-IP connections from all of the clients. And then channel them through one single TCP-IP connection to the Jabber server. You're following me here, you see what I'm doing here. And this is how, so I got called in, you sent me the code ahead of time, I went in and I reviewed it. And this is what I discovered right here. So this is how they set up their whole concurrency model. Now this team was a very talented team, but they'd been working in C++. They were thinking in terms of OS threads. And what they'd done is for every client they had two processes. One to handle the inbound messages and one to handle the outbound messages. And then once you'd receive a message, you sent it to another process which would handle the decoding. That process would decode the message after which it would pass it on to a third process which would deal with the state. So it would do all the error handling, it would do whatever rerouting needed to be done. From there it passed it on to a fourth process who would encode it again and then send it off to the process which was managing the socket connection towards the Jabber servers. Can anyone see how many simultaneous messages can we handle using this architecture at any one time? Sorry. This was pre-multi-core, so there was only one thread. But think in terms of concurrency, think in terms of the virtual machine handling a lot of messages in parallel. The total is three messages, well actually four. We could be receiving one message, then we could only as there was a process which was handling the decoding, we could only decode one message at any one time. One process handling the state, we could only have handled one message at a time, we could only encode one message at a time and then send one message to the server. So any one time we could handle a maximum of four messages in parallel, so not that efficient. And this is the way I've seen a lot of users think when they're coming from the C++ world but also from the Ruby world, because you're thinking in terms of threads, you're not thinking in terms of lightweight processes. Remember, lightweight process processes are cheap, it will take you sub microseconds to create them and they consume very little memory. So you know try to think concurrently and in this case, you know truly concurrent activity was dealing with the socket right here. So we went in and refactored the code and we had one process which was connected to the client and we used that process to call library modules instead. So what did we do? That process would first call in a library module to decode the message. It would call a library module to do all the state handling and then it would call a library module to encode the message. Once it had encoded it, it sent it to the green process up there which would forward it on to the Jabra servers. So all of a sudden we went from being able to handle four messages simultaneously to being able to handle 512 messages or about 500 messages which was the maximum number of open sockets that always allow us to have. And just by going over to this architecture back then, we got a 400 percent increase in throughput and a large, you know, not large reduction in memory. Certain things had to be serialized and that's why we had the blue process right there. So every time we need to serialize something which required a particular order, the terministic order, we just send the message and use that blue process to synchronize. You following me here? Yeah. Okay. So, you know, this is the way, you know, we started reasoning in around 2000. And, you know, the goal here and what we did here is, you know, in even when we were stress testing it, we couldn't get it beyond a certain number of, you know, I don't think, I think we couldn't get it beyond 70 or 80 percent CPU. And what we done is we'd actually push down all of the bottlenecks to the underlying hardware, operating system and external dependencies. Which is really, you know, when you're dealing with Erlang, when you're thinking with Erlang, and that also obviously means, you know, the Erlang virtual machine, the beam emulator, you need to start pushing everything, you know, pushing all of all of your dependencies so that your bottlenecks are outside of your system.  So, we moved on and, you know, this team left this company and took ownership of, I think, one of the first major open source Erlang projects, eJabberD, which I think at its height, and probably still today, accounted for about 60 to 70 percent of all of the Jabber-based, XMPP-based traffic. And they started working on a distributed XMPP server. You know, back in 2008, you could manage about 30,000 users per node. We've gone in and we've actually forked that project into what we've now branded here at Erlang Solutions, MongooseIM. And earlier this year actually reached one million simultaneously connected users on a single node. Now, eJabberD and even MongooseIM is very much the way we used to design systems in, you know, in 2002, 2003. And we did that by clustering.  So, state across them. So, what would happen here is that a connection would come in and by the load balancer would be re-routed to once, assume we had three eJabberD nodes running. These eJabberD nodes would have a fully replicated MNESA database. So, you know, you'd have, there would be, yeah, the data would be exact copies on all of the three nodes. The load balancer would come in and it would forward everything to a process. Now, whenever you logged on using your eJabberD client, a process would be created which lived for as long as your session kept on going. So, every time a client sent a message, it would be sent to that process which handled your session. Then, you know, the checks would be done here. You wanted to forward a message to someone else. The client would check, okay, is this user on the same node? No, he's not. Okay, let's forward it to the node, he's actually on. So, we go for the client to server and then at the client to server, it would then forward on the message to the final client. So, we would basically have different hops in between the servers. We'd also have, you know, chat rooms, we would have federation and session management processes as well. So, but here, once again, this is the way we did it in 2002. Definitely not the way we do it today. Now, start imagining all of a sudden you've got one million connected users on a single node. That means you've got a million processes running, not necessarily doing anything. So, that's just wasting resources. The second problem, what happens if the node fails? You get a million users being kicked out. What happens if you get a million users simultaneously trying to log on to your system again? They hit that node and that node. Maybe a good load balancer, half a million here, half a million there. Logging in is the most expensive operation when you're dealing with XMPP. So, that would cause that node to crash. That node, so all of a sudden you'd have one and a half million users trying to connect on that node and you get a cascading failure. Back in 2002, you didn't have to worry. You know, you may be, in worst case, you got 30,000 users logging on again. If that, you didn't have to deal with the scale you need to deal with today.  This here is a slide from a customer of ours. They approached us in 2007 and wanted us to build a scalable instant messaging and email gateway. So, using the predecessor to smartphones, so what they wanted, what they wanted was, using, what they wanted was to provide connectivity for email and instant messaging, which was highly optimized for mobile. So, it meant a, you know, security.  When you lost connection, that was the first thing you had to think about when you're dealing with mobile. The second is thing you need to deal, think about is bandwidth. So, you want to reduce the protocol, reduce the overhead and make sure that, well, you compress your data and then only share the necessary amounts. And the third thing you need to think about when dealing with mobile is battery life, especially, this was 2007. So, you know, we've come a long way in the last 10 years, but it was still early days. And what we did is, you know, this is what the architecture looked like. Your mobile connections, your mobile phones, your cell phones connected to a set of front-end nodes. These were all airline nodes. They would receive their request and their goal was just to manage TCPIP connections. They were stateless with the exception of managing a TCPIP connection. They'd receive their requests, they'd parse them, and then we'd send an airline term to one of the three back-end nodes. Okay? So, whenever one of the back-end nodes received the request, it would spawn a new process which would live for as long as this request was being handled. So, try to picture instant messaging. What can happen? Well, you send the message. The message gets received by the front-end node. It gets sent to the back-end node. A message, a process is spawned and it keeps, it handles that message and the process lives for as long as it's done whatever it has to do. And in the case of instant messaging, what you would most probably do is, you know, you'd handle the message, you'd log it, you'd, you know, make sure that the session was still alive, and you would then forward it on to, I don't know, it could be AIM, it could be Google Talk, ICQ, or MSN, so Windows Live. Once, you know, ICQ or Windows Live or whoever you're forwarding it had acknowledged it, you send back the acknowledgement saying, yes, the message had been sent, the process would send an acknowledgement and went back to the cell phone saying, yes, you know, the receiving part has received the message and at that point you would terminate the process. And all of a sudden, what does this mean? That meant that you were receiving 20 status updates from your buddies, that meant 20 processes all being handled at the same time, all being processed at the same time. When I talk about making a mental change or a mental shift in how you do things, not have one process which handles your session where you then serialized all your 20 status updates from your buddies, no, you have 20 separate processes each dealing with a status update.  Yeah, I'll show you in a second. Any questions on this architecture? Okay, to add to this architecture what we had here, we had three, you know, we were not sharing all the data. The only data we were sharing was the session data. So if a user logged on, we had a process which could go in, it would authenticate them, it would then create a session record, store it in amnesia database and then, you know, send back and yeah, okay, you're logged on all well. Now this record in amnesia database would then automatically be duplicated to all of the nodes. What does that mean? If we lost a front-end node, we'd get a million users logging on again but they didn't log on. All you had to do was set up a million new TCP IP connections on the remaining nodes and you had load balancers which would throttle and make sure that you wouldn't get a wave which would come in and sync your front-end nodes.  No, here's a ready to log on. His session is active because you're duplicating the session across all the nodes and so we just send back in a crowd and yeah, you're logged on without having to do any of you know, the complex analysis and you're contacting back-end servers, database lookups. We just send back an okay, which would be a very easy, you could handle hundreds of thousands of these requests per second in this cluster. So you're losing a node, a couple of seconds, everything had come back to normal and your whole peak of re-creating the sockets was done and also if you lost a back-end node, all that would happen is a request coming in from a front-end node would be routed to one of the two other nodes. We had two level hashings so that we wouldn't go in and rehash all the requests. So all the requests going through those two nodes would still be going through those two nodes. When going in that route, all of a sudden you lost that node, the request was sent to one of the two other nodes, the session record was still stored in the distributed table, you just look it up and you'd handle that node. So it was straightforward, fairly straightforward, and so there we removed, you know, with this architecture, the single points of failure we had with, you know, the architectures which we were dealing with around 2002.  Now remember this is mobile, every transaction was at least for HTTP requests, free destructive database operations, seven log entries, so there was a lot of complexity and business logic to it, a lot of XML parsing and so on. And so contractually we had to show that we could handle 15,000 transactions per second and that included killing machines off, so pulling network cables out, shutting the power off machines, the other, the remaining hardware still needed to maintain and handle that load. And the way we used to stress test our systems was fairly simple, you know, back in the days we started low testers and we got the system to run to about 100% CPU. That meant, you know, we got the system to become CPU bound and we'd follow the performance and as soon as we started seeing a degradation in performance we'd go in and look for the bottlenecks and using different techniques which, you know, we don't have the time to go into it now but using different techniques, I'd go in and I'd go in and, you know, we'd go in and just remove those bottlenecks and, you know, the goal was, you know, to get a stable throughput. So no matter how many simultaneous requests were going through the system, your throughput had to remain constant so you didn't want any degradation of throughput as more users logged on to the system. So just to give you an example here, if you had about, you know, if your system handled a thousand requests per second and you had a thousand requests going through your system, each request would take one second, right? So if you now increase the number simultaneous request to 2000, the only difference you should see is the latency. So every request would now should now take two seconds to handle. And so that's what we mean by constant throughput. You want your system to be predictable on the heavy load and know exactly what happens and you want it to behave. And that's what we've done with all systems in the past. And I think, you know, this is an old old example of a benchmark on your web server, which John stroke did and I think he published it in his PhD thesis, where you're looking at the throughput here, which is 800 kilobytes per second through the your server. And what it does is he measured the throughput, you know, if you had 10,000 connections, 20,000, 30,000, all the way up to 80,000, which in 2002, when this, well, this measurement was done, was pretty much the physical limit to how many processes, you know, your standard Linux, your Linux machine could handle. And you notice here that irrespective of the number of connections, the throughput is always the same. And this is, this is the pattern, you know, we're used to dealing with. Now, what, yes, much, okay. So what was happening though in 2006, however, 2007 was multicore was coming about. And so, you know, we started stress testing the system and we had load machines. First thing we did is we crashed the firewall.  We started loading again and eJabraD cluster, which we're using in the back end, wasn't strong enough. So we had to put more machines there. We started load testing again and we managed to crash the out and load balancers. So we got rid of those. We started stress testing again and we just couldn't create enough load. So we had to add two more load machines. And we got to borrow some F5 load balancers, which worked much better at the time they were the worst, well, the worst meaning the best out there that they really were able to handle the load. But even there, we still got them to crash every now and then. And it was a nightmare. We thought it would take us two weeks to stress test the system. Three months on, we still weren't done, because all of a sudden we were dealing with multicore. And we hadn't anticipated the power that, you know, the airline virtual machine, the beam emulator or multicore environments would unleash. And we were stuck dealing with TCP-IP congestion, memory spikes, higher starvation, OS limitations and a lot, lot more. So it became really, really tough. So, you know, this is where, you know, we move on to a couple of years ago, about two years ago, where there's a web framework which tried to simulate Rubion Rails and, you know, called Chicago Boss. And, you know, there's a company called Concurix up in Seattle, which at the time was focused very much on optimizing the airline VM for multicore. And they went in and rewrote a Rubion Rails application, which at the time was running 30 requests per second.  You've got, it will scale. So they ran it on two cores. They got about 40 requests per second. They ran it on four cores. They got 40 requests per second. They ran it on all the way up to 64 cores and they still got 40 requests per second through. And that's when they started scratching their heads and, you know, came to the realization that, you know, they were being hit by Amdahl's law, which in layman's term tells you that your fastest, you know, your program will run as fast as its slowest component, in this case the slowest component being the sequential code. And what they discovered is that they were serializing requests through a single gen server. So you had a lot of workers all doing all the work concurrently. So each worker was serving a web page concurrently, but when it was time to send everything back to the web servers, they all went through one single, one single gen server, which created your bottleneck. They went in, removed the bottleneck and look at that, you know, from 40 requests on two cores, they reached about 800. On four cores, you know, they were over a thousand after which they hit the next bottleneck and the next one. So there's still a lot of headroom, you know, as you add more cores to the machines. But, you know, another thing you need to think about here in the message here is you're dealing with multicore architectures. You need to now start thinking terms of bottlenecks in your systems, because the bottleneck is what's going to stop your system from scaling. All right. Okay. And, you know, what does a future hold? I think, you know, this is React. This is, you know, from a Bashow presentation and if any of you have been on a React or Bashow presentation, you know, if the Bashon please don't show the ring they don't get paid. You know, that's a standing joke, but it's basically using consistent hashing to distribute across a cluster of nodes. So that's one way in which we're looking at, you know, so I think we've solved the concurrency aspects where you need to have a process for every truly concurrent activity in your system. We're now aware of the bottlenecks which happen in your system and a lot of tools are being built to find these bottlenecks in your system and give you warnings and help you try to well, remove the sequential code from that process and try to parallelize it as much as possible. So, you know, what we're dealing now and I think the next big challenge we within the Erlang World have, you in the Elixir World have is distributed architectures, massively distributed architectures and so consistent hashing, you know, the way React does it is one way. Robert earlier mentioned SD Erlang which allows you to create clusters. It's based on the concept of process groups and then there's one last concept which I still haven't found the slide for which is looking at Erlang on Zen. So, you've got instances of the Ling virtual machine which is not the beam but they've written their own virtual machine which runs in a Zen instance on the bare metal and what's happening there is, and I all recommend you look, go in and listen to Stu Bailey's talk, a keynote at Erlang user conference, they've taken a so the link switch which is an open flow switch and are integrating it into the Zen instance. So now you can then fire off, you know, tens of thousands of Erlang on Zen instances which start communicating with each other using software defined networking where connections, you know, based on your layer two and layer three traffic will tear up and you know, pull down connections, you know, based on your traffic patterns.  And, you know, that's really where we're heading now and, you know, those will be the types of problems we're thinking and solving but once again, this will require another mental readjustment in how we do things because, you know, being able to fire off Zen instances in a few milliseconds and tearing them down and optimizing traffic within the means will start architecturing our systems in a completely different way. And structuring our code in a complete, all right. I think we're out of time. So I'm around all day. I don't buy it. Feel free to come and ask questions, love to answer and brainstorm.  I'm writing a book with Steve Winovsky, Designing for Scalability with OTP. And what we're looking at is what happens behind the scene when you're dealing with OTP and how do you structure and how much you think and reason. And, you know, everything in there is, if not as important, just even more important for you working on Elixir because it's the same concepts, it's the same principles. And, you know, we're trying to put all of the lessons learned from the last 20 years into this book. So, well, others can use them. All right. Thank you very much.


G'day, I'm Lieutenant Brian Payne from Carradine's Fire Brigade and around about 1.30 this morning Paterson River and Carradine's Fire Brigade attended a large structure fire in the Seaford area. As we approached we could see a large pillar of black smoke. On seeing the Gravity Zone, which is a child play centre, the main warehouse was well alive which contained trampolines and foam pits. At the height of the fire we had 100 firefighters, two aerial firefighting vehicles and 10 pumpers.  The fire was contained to one building due to the hard work of crews who protected the adjoining premises. Other specialist vehicles that attended such as Scoresby Hose Land, Frankston B.A. Van, Daniel Hazmat Detection Vehicle and Marilyn Catering. Gravity Zone staff were unseen at the time of the fire and the cause of the fire is not Jim Suspicious.


You're welcome back. Liberal Party has condemned the invasion and occupation of party secretaries, including Tugs and unauthorized policemen drafted from the government house and also ahead of the IMO State Governorship Election Schedule for November 11. The Liberal Party has also been under fire for fixing the price of its expression of interest and nomination forms at 25 million, a whopping 25 million. While joining us to discuss the crisis at the national level and also preparations for the primaries in IMO State is retired Colonel Chinyere, or B, Liberal Party Chieftain. And we're hoping also to be joined by Chinyedu Amadi, Liberal Party governorship aspirant in IMO State. Thank you very much, Colonel, for joining us on the program. Thank you very much. Okay. Let's just begin from the national scene. There is a Lamidia Papa who is insisting that he is the authentic or the acting chairman of Liberal Party right now. Is that a consensus decision that gave him that position or he's just arrogating powers to himself? He addressed a negative power point, so nobody gave him any power. So what's happening at the national level now? Who is leading Liberal Party right now? It's supposed to be a bully, but the people now came and brought thugs, and then when they brought thugs, if they were bullied, he would find his way through the battle. Because the way they came, they are coming to destroy, they are not coming for peace. Because when they came in, they went upstairs and started shouting, level, level, level. I don't know why it's level, there's GDP, there's ABCA, but why level? Everybody is out for level. It wasn't like this before, but immediately they saw that PTOB is their headache. Everybody is attacking level now, which is not supposed to be like that. It has been peace, peace, peace. Like the president to be PTOB, he said everybody should be calm, like they said let's move to courts. And we are right in courts. So I don't still see why they are fighting us, why we are still in courts. I don't see the reason why they should come to level headquarters. Only God knows who sent them. But they are not one of the laborers, and they are not one of the Obedan's. So I don't know where they are coming from. Okay. Only God knows. Before now, you were supporting a PDP candidate in the 2019 general elections, and now you are laborer. What changed? When I supported PDP, that's article, I worked with 36 states with article and PTOB. I was going to prison, prison, I played football for him, I know how much I spent. But when I saw that there is nothing coming out of PDP, and I was there that very day they paid 100 million for each ticket PDP that time. But when I see there is nothing coming out of, I didn't say anything that will come out of PDP and article. Because it always do, when anything they happen, he will not go back to Dubai and forget about us. Even when we went to Supreme Court, I was there shouting, we need our mandate. We went to her eye-neck and shouted, but the person we are fighting for does not care about the promises. So that's why, and when I saw that PDP as the captain, I followed him to the cap as well. Okay. I understand that we're being joined by Honorable Chinadu Amadi, a Labor Party governorship as parent. Honorable Amadi, good evening and welcome to the program. Good evening. Thank you for having me on your program. Okay. A Labor Party in a most state, first of all, the expression of interest form is very, very high, hoping 25 million we hear. What informed that, first of all, and how are the people taking it? Is that a question for me? Yes. It's for you, Honorable Amadi. You are in the race. I'm sure that you spent that much. How are the people taking it? I think the matter has been resolved, and Labor as a listening party has actually moved on from that form and pricing, you know, it was reviewed downwards to 15 million, and that matter is done on dealt with. So we're now on course with the primaries for Labor Party. And how many of you are standing in for this position, running these primaries to get the ticket to contest as Labor governorship candidates? I didn't hear you clearly. How many of you are the contenders under Labor Party? We are 16, presently we are 15, who lost one of the aspirants, but it's our home for animal doom. They are so raised in peace, and so we are 15 on the cause. Okay. Let me go back to Koro. You said that you left the PDP because you didn't see anything coming out of PDP. You are now in Labor because the prospects are higher and all that. But you are in a state where the incumbent is APC, and he also is qualified to stand again to run for governorship. What are the chances of Labor Party in a state like emo state? We already know that emo state, you know, before we got the governor that is sitting right now, we still do not understand how it went. What do you think will happen? How do you think Labor Party will fare in that state emo state? Yeah, in this emo state, I have been here for complete six months now that I came back to emo state from Abuja. And what I'm seeing in emo state, the war is no more politics, it's now war. And the war in emo state now has become survival of the fittest. And then what I'm saying at my age, by June 16th of the 74 years, I'm not seeing anything in Labor because the way they are looking at this is no more politics. It's no more let me come in and say, even when they are fighting a battle that is either we win or we fight, it's a war. But let them remember, I'm asking their parents that they are coming to fight somebody who said that it will be over my dead body for somebody to take that seat. And me and you know the way he came in last time, from fourth position to governorship. And what I'm saying now that somebody who can fight him, that is General Jack Ogilewe, nobody else. Because this is the man that is a general in the army and retired. I'm not praising him because I'm a colonel. I'm praising him because I just met him once at the airport and I know what is capable of doing. And I know he's the only one that can fight this fight. And at least if he comes in, he has already, and the present governments are shaking now that is a general in the army that is going to fight this fight. And then we can have peace in our way, but we don't have peace now or what we have here. And we have a new state anyway. If you go to the secret, you will know. These people will come and say they are the new one. They will go to court and put court in junction. And the next one will come and say they are the old one. This is politics. This is not where we are in the army. This is not where you are fighting a war, a Kuma war. And fighting a war is either one out of these 15, he's saying it's only one person that will march for delegates and it's only one person that will fight. It's a good thing that Honorable Amadi is here. Honorable Amadi, she has just said she throws in her cup for the general. But you are in the race as well. What are you bringing to the table that you think will mark you out? Will stand you out among your pairs? Oh, thank you for the question. I will start by trying to address some of the positions of the, oh, sorry.  Can you hear me? Yes, I can hear you. Go ahead. So address the question about insecurity, that's one area that needs to be properly addressed when it comes to governance in the states. So right now she pointed out the way the incumbent came into power is of course a challenge and that has agitated the spirit of the immolates. They've never found it comfortable within themselves to see such a thing happen in the state. And that is one of the factors that is triggering the insecurity issue in the state. Now, that coupled with the high rate of unemployment in the states on record, you would know that the state has the highest unemployment rate in Nigeria with a thriving population, nearly three million. That is a time bomb on its own. So, but I want us to look at emo states and we see a smaller model of Nigeria in emo. We shared similarities with emo, the problems in Andreas is similar to those of emo. We look at insecurity that is ravaging the country. And of course we have the general as the president of the country and yet we couldn't solve the problem of insecurity. So now, similar to emo states, if we want to solve the problem first, we have to give the people demanded the kind of government that they want. That will give them peace of mind. And I tell you, I give you assurance that the moment you do that, you would have about 50% of these tensions going on in the state disappear and the rest will be on how to deal with the insecurity itself, the remaining part of it. And that has to do with revamping the economy, reviving the economy, creating and employments, you know, going back to the farmlands, creating a great industries, creating manufacturing industries, getting the youths busy. They say the idle mind is a devil's workshop. The youths have not engaged and said the negative biases because there's nothing to do. So, the core of emo states lies more in the economy, rather also looking at insecurity, the bad part of that insecurity is first the injustice, then secondly the high rate of unemployment. You deal with these two things, you will have emo states on proper cover. Then you will now follow up with intelligence monitoring. I tell you with two drones in the forest of Osu, East and West, you have a security center, you can monitor the activities and begin to address them or mobilize your security agencies properly to go, not to go in blindly, because it's when they go in blindly that innocent souls are taken up on courses, more grievances in the society. So, that is the problem of emo states. And I think it has nothing to do with being a general. And if you want to look at examples worldwide, we can take Ukraine and Russia and we see a president who used to be a comedian who is now fighting the war with Russia and doing so very well with it. And lastly, I would say that once you become a governor or a head of state, you become a general yourself because the security agencies line up with you. So I don't believe in the fact that the incumbent is a lion. I think that is an over description. We need to properly align things and look at security from the point of intelligence monitoring, putting innovation and technology into security issues and addressing it properly, then creating a viable economy for emo state. And that is a real point we have to focus on. Once we get that done, emo state will get back on course. And I think with my exposure first, my exposure walking in the oil industry, running various ventures, real estate, IT, farms and also running heading an organization that looks at capacity being for youth and women. I know that I have all the qualities. I'm already looking at the vision of emo for the next 40 years. Where do we want to get infrastructure to? Right now we're sitting on an infrastructure that is 40 years old. It's being beat by Dr. Samu Mbappe. We need to get him ready for another 40 years. And that's where we're going. I'm youthful and energetic. I think out of the box I'm very analytical. Thank you very much. Let me ask you a question. Just a moment. I know that the contenders may be up to 16, like you have said. And whoever emerges after the primaries, I know all of you will queue behind that person. The contenders include also the immediate past chairperson of International Federation of Women, Lawyers, Association, FEDA, emo state, indeed, Vale Keoma. And so many other people. But the issue here is that if somebody emerges, and you are going into the election, there are two opposing forces against you that will be working against you. The incumbent, who is APC. We also have PDP, which is likely to have a consensus candidate. Remember that Ihe Doha, Mekai Ihe Doha, who was the governor for a time and now left because of the Supreme Court ruling. Now someone else who is sitting there has said that he's no longer contesting. So it seems they are going to have a consensus, maybe in the person of Senator Samuel Anyahu, who is the national secretary of the People's Democratic Party, PDP. So these two forces, a major opposition party in the country, having a consensus candidate, and then you're having the incumbent, who is of the APC. So does it not give you worry that Labour Party may come a distant third or fourth in the election? What do you think that you can do to make sure that it will turn in your favour? Let me start with you, Connell. Ask a party, what are the chances of Labour knowing these two other parties standing with you? I want to ask you a question. It's talking about security. Do we have security in human states? The answer is no. On the 18th, I was shot on my leg as they had the wound. I would come shot for House of Assembly in my own local cabinet. It doesn't matter. And I'm just asking a question. CHD, Honourable CHD, this condition must take need of security. Without security, let's see when they say sit at home. So many people were killed. When they said every Monday people should sit at home. Let's just try to wrap up, Connell. Let's just try to wrap up. We're concerned about the chances of Labour Party. I know that all this, you will iron them out when the primaries are over. But the chances of Labour Party, whoever emerges. This is the only person that can savage human states. Come for security. Come to, when you talk about security, this is the man that knows what is security all about. I'm a retired Connell. I know what is security is all about. We are security conscience. But here, human state has no security. And I believe that nobody can fight this man if he's not General Ogulewe. He's the only one that he did it in other states and I believe that he will do it in human states. Because he's the only people like they are bringing all the soldiers that is not even real. When you even tell them it's critical, they don't know what it is. At first they don't know what it is. Then it's only his own soldiers that will ask these questions. And we now flush all the soldiers that are coming up because it's going to be a war. It's unfortunate that we may not have time to deal with all the issues that we need to deal with. But we would just like to wish Labour Party good luck in the forthcoming elections. We do hope that you will put all differences aside. You will be able to elect someone at the primaries that can wrestle power as it were from the incumbent and from the major opposition party and form a new or give a new breath, fresh breath as they say to emo state. We do hope that that will happen. But if it doesn't happen, we do still hope that emo state will be the better for it, whatever outcome the election is going to give us. Thank you so much Honourable Chinadu Amadi for coming on the programme. We wish you luck as well. And also, Honourable Chinadu Amadi, Labour Party Chief 10, thank you so much for coming on the programme. Thank you very much. Thank you. Well, we'll be talking with the Chief 10s of Labour Party in emo state. One is an aspirant and the other one is a supporter of an aspirant. But both of them are Chief 10s of Labour Party and they're talking to us about what is going on in emo state and their hopes are high. We keep our fingers crossed, we see what comes out of that. But we pray for a better Nigeria and all the federating units of Nigeria. And that's how we're going to wrap it up on the show tonight. See you again tomorrow.


I love to Accelerator because I got to learn a lot of things that I wouldn't get to in a traditional classroom setting. You're always moving, you're always learning. They make it really fun and hands-on and that's what I really like. We learn a little bit better when we're doing stuff and not just sitting in a chair. These classes aren't offered at my school and so it's a great chance to get to learn new things. I was really inspired by the people working in different fields. It really made me think about my future. I really liked the classroom but I also really liked the social aspect of it. It was really fun with all the games they had set up and meeting new people. I really enjoyed the waterfest. That was fun. We had bubbles, water balloons, a waterslide and sparklers. That was fun. If you value learning but you also value having fun and enjoying yourself, this would be the right place to come. It doesn't feel like school because you're doing so much fun stuff. Accelerator is stepping outside of the traditional classroom and learning something new.


Hey, I can see that Rohan Prabhu is joining us on the Zoom link. So we should be able to start the first talk of the day pretty soon. Hi Rohan, hope you're great, doing great. So folks. Hello everyone. I'm Rohan Prabhu. I'm a director of engineering at Jupiter Money and today I would like to talk to you about how we handle BII and SPDA reduction. So before we begin, a bit about myself. I was one of the first employees at jupiter.money, so you know, being there from the beginning, being in tech for about eight years, give or take. I was a co-founder and CTO of MITRE.IO prior to this and before that I have some experience working at Google and Amazon. So coming down to it, what is BII and SPDI, right? So BII is what we call personally identifiable information, so information that can personally identify you, like your phone number or your PAN for example, anything that you would use up as a lookup key and can narrow down to a specific person. So that's what we call BII and SPDI or sensitive personal data or information is information that cannot pinpoint to you. But you know, if it was released as part of your information, it could be something sensitive that most people might not be comfortable the world knowing, right? So you're an income group, your marital status, community, etc., are examples. So another question is what is redaction, right? One might think that it's about, you know, when you get your PAN or your other, you see a couple of X's in the number, right? So that masking is not what we call redaction. So redaction is basically when we put in data in a way that is highly governed and most of our microservices or most of your data stores that use or process or collect this BII or SPDI data points, they do not, they do not store the data in plain text, right? And so why do we need something like this? So prior first is because we're in the Pentax space and we also onboard customers to open actual accounts, we are required to handle sensitive information as part of customer KYC for example. The other problem that we face with this is because of microservice architecture, we've got multiple services, everybody having their own data store and then just putting in all of the data that they receive. And so it's very easy to lose control of their customer's information is being stored. So governance is actually what is the real motivation behind BII redaction. It's the basic idea is that anything that might be BII or SPDI relating to a customer, we bring in together and then we store it in a place that we can tightly govern. So probably like a single store with a very high and elevated level of access or at the same time, you know, you're also talking about when this data is in the application space, it does not rely, it does not reside in plain text. So even if you were to accidentally log it, you were to accidentally transmit it, you're still sending out something that's really not decipherable or not something that can trace back to the actual BII or SPDI element. So the basic schematic is this, right? And this is quite common across microservice architecture. So you can see that at one place, you have a microservice and then there's a standard gateway through which different applications or your customer endpoints interact. And when the data comes in before it actually hits the application within the application realm itself, we have a redaction layer. So in this example, you can see there's a pan number coming in and then the redaction layer does its magic and it basically turns it into something that's kind of garbage. It's not something that's very legible. It's not something that you can make sense out of. And within the application realm, that's what it's dealing with. Within our schematic, we use DynamoDB to store this and this intelligible form of data is what we call as a tokenized form of data. And of course, I'll walk you through more details in the coming slides. The other thing that we also need to do is some form of talentization. So we partnered up with a couple of other financial services providers and each one of them wants their customer's data to be stored separately. And when I say separately, it doesn't necessarily just mean two Dynamo tables, but it could literally mean two different AWS accounts itself within which DynamoDB is running. So this is kind of an example where because of the reduction layer, we are in a position to be able to put in a persistence, which talks about where is the actual tokenized form of this data going to be stored. But when it comes down to the application realm, the application as you can see in the lower right corner, it has a tokenized data. It has the data of all the user together across different partners as well, but it's an adapted form, right? And the actual data that was being stored in our folios, it's only there in those tenantized options across two different AWS accounts or two different DynamoDB levels, depending on what kind of talentization we want to achieve. So the implementation is quite simple. There were multiple ways to do this. What we decided to, the route we chose to go was put in a reduction layer, which is essentially in the serialization and deserialization layer. And that's kind of because that's a common interface points, whether you talk ingress or you talk egress, right? You're calling a controller somewhere, you've got to deserialize all of your incoming request objects when you are making an API call, you have to serialize those API objects. So that kind of seemed like logically right place and also the simplest place to implement something that would perform our data level manipulation just across the whole application. What we do is your serialization and deserialization layer we use Jackson, is when this process is happening, we basically pick out the important data elements from the data and then we tokenize them. And when we say tokenization, we basically use a secure hash algorithm and using that to generate a hash and then we store it in our folio. A folio is essentially a place where you store the token with what data it actually tokenized. And this all of this tokenized data goes in DynamoDB and on DynamoDB we have strips here monitoring. So if anybody tried to get access to it, anybody tried to get changed their elevated access to basically get access to the CWS upon them, which it was stored. Anybody actually even just tried to access it using AWS console, they won't be able to, but even that attempt kind of gets locked. So that kind of is the whole point of governance, as I mentioned, right, is that we have strict governance on just this one entity. So that is just this one entity that we have to worry about. Coming to tokenization, why do you secure hash algorithm? Well, the one thing is it's irreversible.  We use secure hash because it makes methods like rainbow using rainbow tables or some kind of other brute force mechanisms extremely difficult or almost impossible to use. But the great thing is that secure hash always works one way it's not entirely randomized. And so your lookups and integrity constraints work really well. So if I store a phone number and then I say, you know, hey, give me the user who has this phone number. In both of the cases, the phone number is going to be tokenized to the exact same token. So, you know, your database lookup, if you had a select star where phone number equal to some string, those kinds of lookups will still work. Your integrity constraints will still work with the tables. If you say, you know, this field and this field needs to be the same. It will be the same as long as the original data being tokenized was the same. So the developer experience, because of course, it's a very, very critical part of us. We have to roll it across so many microservices, so many service owners. We have built it as a drop-in library and you just like put that library in your creative dependencies and you add in a bean modifier which will configure Spring for you and then think you have interceptor. So then if you are building K plane clients, that is where, you know, you just add in this interceptor and it will essentially do all the magic for you. So to reiterate the principle is simple. On ingress, we tokenize sensitive fields and on egress, we do tokenize sensitive fields. So I'll run you for an example. To set the stage, let's say I have two services an external data service and a user registration service. And what you can see here is the request and response objects essentially for these services. And on the registration request, there is a field called pan and what I've basically done is declared it as a PII control data element. So I've assigned it a data element identifier which in this case is data element dot pan. And now there is a request in the other service which is the external data service. There also you have a field called pan. The field name could be different. But what's interesting is that both of them actually have the same data element identifier. So using that, the library or the entire PI redaction piece and our figures out, you know, like these are two equal elements. So, you know, if even if they were lying on two different sides of ingress versus egress, it could know what to de-tokenize and where to de-tokenize from. That's kind of the basic idea. So using this example, I'll run you through a case. So let's say this is our setup, right? You've got a user registration service. You've got an external data service. The user registration service has got a database where, you know, it stores the user data pan name, et cetera. The external data services, let's say it's our gateway to all of our other external data providers of which NSDL data provider is one. And on the top right corner, we have the folio which basically stores tokens versus whatever data they actually tokenize. So a request comes in to the user registration services like, hey, register this user. It's got a pan, it's got a phone number. When it moves into the application logic, just before it moves in, the reduction layer does its magic. It adds those two tokenized entries into the folio and it morphs the registration request. So as long as it's in the application logic, this is what it's seeing, which is why when it generates the next request to the next service, you know, the pan values are still essentially the tokenized forms. And then when egress happens, the feint line, the interceptor has the redaction piece also there, which basically does a lookup from the folio, puts in the actual pan. When it is coming down to the external data service after the controller layer, before it hits the application rail, the data is again redacted. And that is what you see. It goes down to the feint line to make that external call. And while it's making that external call, again on egress, the data basically gets de-tokenized to what the external data provider is actually expecting. Coming back, you get a response with the name. Again, it's also redacted because, you know, I would have annotated it with a PII control annotation. And let's say at this point in time, the developer has, you know, some kind of a debug log, something that's been left behind which says, hey, we got user's name. But because in the application rail, the actual name was never present, what you see getting printed is again this garbage tokenized value which, you know, is kind of, it's intelligible. You can't make sense of it. It doesn't matter that it's there. This response goes back all the way. Again, on any form of egress, we always de-tokenize and within the application rail, it's always tokenized again. And at this point of time, let's say, you know, this application logic is that, you know, if the registration piece works well, we store these details in the database. So I've stored the phone number and I've stored the name, but because it's happening within the application realm itself, as you can see, the phone number and name are all tokenized formats, right? Like nothing sensitive or nothing PII about this particular piece of information. And then we get the response back and when you get the response back, the name is still redacted because we're still in the application type. Just before the controller is about to send this data out, this final tokenization happens again and then this is something that the user would see. So coming down to it, at the end of all of this, like we saw across two microservices, there was an external data provider. Now the external data provider doesn't understand our tokens at all, but we were still able to give the external data provider exactly the data that we needed. And within this applications, both of these, so within the organizational realm, whatever happened, whatever state changes were made, whether it was by making an error in your slot, or maybe the lock would have been engaged as well. Doesn't matter. Or any database entries that were being made, all of them still have the tokenized information. Again, not something that you could get back to the actual user's data. Now, if we had another API which said get user by phone number, you know, the integrations into Springwood, again, you know, do with stuff, change the phone number to something and because it's a secure hash algorithm, we would get the exact same phone number that you're seeing, right? The one starting with JEE and your database lookups would still work. So our primary key lookups, our basic integrity constraints are kind of maintained this way. It doesn't matter which part of the application it is in, as long as the input data was the same, the token would be the same as that. And of course, you know, like this solution does not come without its caveats. So one of the things that you might have noticed is that data on-wire is kind of unredacted because on egress, we always de-tokenize it. So while we've got these two services when they're talking to each other, the data does get de-tokenized. So kind of on-wire, the data remains unredacted. How we currently alleviate this is that our all internal networking is provisioned via strictly controlled and standardized N-Y configurations. The logs again have very strict CM monitoring. So nobody has access to our N-Y and proxy logs within the network. So that is one of the ways we still save card our customer data entirely, but you know, it would just make you put be a great way ahead if we remove that part as well. One other major problem that we face from time to time that library integration is kind of required. So, you know, it imposes limitations on tech choices. You've got to stick with spring unless there's a library integration for something else. The serialization, de-serialization is kind of very strict, right? So it's got to be Jackson because it's a little deeply integrated with it. The other issue is what if you were using things that do not rely on serialization, de-serialization. For example, if you were just writing to a file, we would highly discuss writing to a file, but you get the idea it unless de-serialization and serialization is part of the data transfer process. And again, using only the libraries that we support, we cannot implement PR at action over that. So, you know, that brings in a hard dependency on using these things. Validations and transformations are of course tricky, right? So transformations like sometimes very simple transformations like phone numbers, for example. Sometimes you want the request to contain a phone number without the international code and with and before calling an external partner if you want to pre-pend 911 do it or the country code do it. So those kind of transformations are really tricky. Validations are of course tricky because you can't do validations of the form. Length should be between 10 to 50 characters, for example. So because all you have within the application realm is just a token. So that is kind of a little difficult to do. The way we have gotten around it is we have kind of a validation and a transformations VM. So what it does is you can, on a given PIA control field, you build like a set of instructions that have to be performed. Now, validation instructions are basically check kind of instructions for transformations are, you know, string to string kind of transformations which could do it for you. And all of this processing actually happens in the redaction layer itself. So what's happening, it's not a lambda that you're passing with which the developer could kind of get access to this data. But it's happening within the redaction layer itself and the developer is just saying, you know, like do A, B and C, not giving like actual functions to it. The only problem with that being as we come up with more validations, as we come up with trickier transformations, there's a slight maintenance over it. One of the most interesting things that we have experimented with and we want to do next is to implement this other service mesh layer itself. Now, we use NY and STO and NY has this cool feature called filters which they write using Wasm and Rust is a target, is something that compiles to Wasm as a target. And we explored that. Currently, NY filters are in beta and also the crypto support for Rust, Wasm is in print. So that kind of really limits us into how much we can do. But if you are able to do that then that could be like zero dependency on the service itself. And the reason that is great is because even today, the redaction layer kind of still resides within the application trend itself. So it's kind of still the same memory space and now this would just completely remove any dependency on the service owner or the maintainer itself to keep the PI redaction piece afloat. So this is one of the most exciting things that we've been experimenting with and we definitely think is the way ahead. The second thing that we want to do is what if our filters could understand an organizational realm rather than an application realm. And that would be like very simple because what we could do is within, when two services are talking to each other, we don't have to do the de-tokenize at egress and tokenize at ingress again because it's still within the same organizational realm. So when these two services are talking and the organizational realm is an alive concept, then the data could flow within microservices completely tokenized itself. And that could just contribute a lot more to correctness. And then thank you so much for your time. That was pretty much what I had. Would love to hear your questions. Hey, thanks Rohan for such a great talk. A lot of technical details in the talk on how to actually start redacting the data. And thank you for joining us right now. So first I will go on to the audience and check if there are any questions that the audience wants to ask right now. Let me check the Q&A tab. Okay, so I don't think we have had any questions from the audience on Zoom as of now. And no questions on YouTube live stream either. Okay, so I think we have one question coming up. Lava Kumar Kuppan asks that, do you have any variable naming convention so that by reading the code, you know, which variables have redacted PII and which ones have unredacted PII? Or do you use some other means to achieve this? Rohan? Yeah, so great question. We don't have conventions necessarily on the variable names because that would again, involve deeply ingrained process and an alternate process where some convention is to be followed and with these processes, the biggest question is who is going to enforce it, who is going to be the checker and maker of this, right? So as we had shown in one of the slides, right? Like the idea is that we use annotations and data element identifiers are kind of global. So we maintain a list of common data element identifiers and you know that these are essentially like strings or URNs or you could use any form of naming convention that we can say that, hey, this is a redacted information for me, right? And then transformations happen within that. But an interesting corollary of this question that I would like to answer is how do you figure out where you might have PII data itself, right? So Macy is kind of like an amazing tool and there are kind of like other amazing tools also, right? Which basically monitor the things that you're doing within your login frameworks, the things that, you basically send your logs to a centralized place, you know via cloud watch twice three would be one example. But what's fantastic is that the more data sources you put in, Macy would give you alerts if you're accidentally putting in data that could be PII. And the greatest thing is you should be confident in logging everything and anything because if you do have PII coverage, then you should absolutely have no issues with logging everything, right? And then Macy does like a good job of telling you and are there still elements that you have skipped? So, you know, that is something that we monitor to continually improve our process. Makes sense. So yeah, I think Lavakumar has a follow-up question. They're saying that, can you talk about how you are handling PII data sent to third-party clients from the client side? Like third-party data migration, like data forwarding from web app, JavaScript or mobile applications? So we absolutely don't do that. So it's almost everything is proxy by, and as a fintech partner that comes in as a limitation as well, and also like a great thing because then, you know, we can enforce our practices and governance as well in protecting our consumers data. So directly, if we were to send analytics, I mean, like the analytics at least that we sent are completely anonymized and, you know, what they eventually tag to and which user they tag to kind of happens as a background and a backend reconciliation process, but we do not send any PII data. We actually send no user data directly from the client apps to third-party services. Which is I think a good practice that a lot of organizations do need to follow. Third-party data forwarding should always be curated and an explicit choice in a structured format rather than just deciding something to send, right? Another question asked by someone anonymous is that is data reduction a CPU intensive process? If yes, how does it affect latency, cost of infra and how to optimize? PI reduction by itself is not a CPU intensive process, not generating the hash. Of course, takes up a couple of your CPU cycles, but if you actually sequence the things that you have to do to, you know, get to a tokenized form, I think the bigger penalty is what you pay on network IO for writing and reading from the DynamoDB tables, right? And that is significantly more than the CPU cycles you would take, right? Like SHA 256, the secure hash, they have optimized it for decades. So it's like super fast. It's a standard algorithm. So yeah, we don't need to wanna put the performance penalties on it. Absolutely, and you know, like so much of research has gone into like making it fast that it's absolutely like super fast. But if you talk about a penalty that we do pay for DynamoDB network IO calls, I mean, that is kind of absolutely there, right? And honestly, how you can optimize it is one of the ways we had done is initially we used to optimize at a field by field level. And now what we do is we batch the entire object together, which, you know, kind of like required us to go deeper into our serialization framework so that we can like batch our calls. But beyond that, our avenues are kind of limited and this is kind of just a penalty you have to accept, right? Because caching also is for example, not an option. If you start caching it, then the whole purpose is lost. And then the PII is maintained in the... Exactly, exactly. Somebody put to an LRU attack or something on the cache. Yeah, a lot of things could be done on top, right? Absolutely. I think this is a necessary cost to pay for privacy and especially in regulated spaces. And most of the organizations are okay paying that slight time cost in latency. Absolutely. Sudip, the Chakravarthy asks, how do you take care of reducing the attack surface from attacks such as SQL injection? Sure. So I'm not sure if it specifically applies to PII reduction, but I mean, you know, because I don't think that the PII reduction layer adds. I would say this question is not very relevant to the talk that you just gave for us. SQL injection prevention is much more like a framework driven thing these days. Absolutely. If you are walking with a good ORM framework, a good MVC framework. And if you're taking care of application security at the top layer in the JavaScript layer itself, you know, a lot of those things will get taken care of as long as you're following coding practices that do not allow for SQL injection, right? It's not pertaining to the current talk at hand. We can, of course, Sudip, you can ask this question separately on our comment section. And I think we can pick this up at a later point of time. Yeah. Actually, just an interesting thought I had, right? I mean, if you think about it, having the tokenization actually reduces the chances of a SQL injection attack because if you added something like an escaping code and said where one or something like that, finally it just comes to a tokenized format. Not that I would make that an argument of why you should have this, but just the thought I had. Yeah, absolutely. You know, you don't have any identifiers, directly obtainable identifiers anymore that you can do an injection and sort of iterate over that, right? Absolutely. Okay. I think folks are happy with the answers. And thank you so much, Rohan, for the talk. I do not see any more new questions coming up. Let me just quickly check any, if anyone has asked anything on, yes, there is one question on YouTube. Aravind Padmanavan asks, could you talk a little about the performance side of this extra? Oh, I think we have already covered that, right? So yeah. I hope Aravind, like, you know, what Rohan has already answered as an acceptable cost of tokenization and encryption. I think that answers your query. So yeah. Thank you, Rohan, for this great, amazing talk. And I really loved how you delve deep into the actual implementation details of this. Personally, I really loved it. Awesome. Thank you so much for having me. Yeah, we're all happy to have you here. Great.


So what's the deal with COVID-19 vaccines and blood clots? So we've been hearing a lot about this in the media recently, and today the FDA put a hold on administration of the Johnson & Johnson COVID vaccine because of potential concerns of a specific type of blood clots, particularly something known as cerebrovenous sinus thrombosis. So the identified six patients, all young females, out of the 6.8 million folks who've gotten the Johnson & Johnson vaccine who've had this disorder. Now this is the same phenomenon that was noted out of Europe not too long ago with the AstraZeneca vaccine where about 30 folks had this diagnosis out of 20 million vaccines delivered. Now if you look at the statistics it's about one in a million chance that this is occurring with the vaccines. Now both the Johnson & Johnson and the AstraZeneca vaccine are of similar mechanism. They're both adenovirus vector vaccine. So when we're seeing this signal there is a concern that there could be something that's associated with the vaccines themselves. Now a little bit of background information. This CVST is different than traditional blood clots, right? So when you think of traditional blood clots like DVTs and PEs which are occurring in you know six to nine hundred thousand folks a year in the United States that's a more common scenario. This particular disorder is less common. It's thought to occur somewhere between five and 15 cases per million. So again it's pretty rare and we're actually seeing it even less frequently in these vaccinated population. Now the difference is the folks who are getting this after the vaccination also have something called thromocytopenia or low platelets and the coexistence of thromocytopenia and CVST is not common. So that's why they put a pause on this to try to figure out what's going on to see if this is a potential positive relationship. Now if anything this should give us confidence that the safety monitoring is robust and we're doing everything we should be doing to watch carefully for the safety of these vaccines. So I would applaud the FDA and CDC for you know implementing this and taking a deep dive into this even though it was a very very uncommon diagnosis that occurred. Now there could be some immune related mechanism that's occurring as a result of these particular vaccines where a very tiny percentage of folks are getting some sort of immune response against their platelets that's perhaps causing consumption activation of platelets that's leading to these problems. But again that's something that we're going to have to investigate and maybe hard to prove causality but we always want to practice precautionary principle. Now to note there are risk factors for getting CVST even independent of the vaccine. For example oral contraception this can increase the risk of this particular type of blood clot from 10 to 50 fold. Pregnancy itself can increase the risk other medical disorders such as cancer and even having COVID-19. In fact all those disorders I mentioned have a higher incidence of developing cerebral venous sinus thrombosis than this one in a million that we're seeing thus far with these particular vaccines. So what's going to happen now is the FDA is going to take a very close look at these particular cases these six cases and try to figure out was there a potential causative effect meaning could the vaccines have led to this and if so are there particular risk factors and concerns we should be aware of. So maybe we're going to be limiting the vaccine in a particular subset of patients a particular age group etc. This again is occurring in younger patients and mostly females which again fits the natural history of this disorder in that general population. But again the big picture is you're more likely to get COVID and die from COVID even if you're young rather than getting this particular disorder from the vaccines. Number two you're more likely to get stuck by lightning than you are to get this particular disorder from the vaccines and finally the overall safety profile of these vaccines and all the other vaccines is very good. Now for anybody who's already gotten the Johnson Johnson vaccine again this is all occurring within the first one to two weeks if you're more than two weeks out if you're probably safe and even if you're not remember the chances of this happening are uber low. Of note this has not been described in the mRNA vaccine that's the maternal advisor and we haven't seen those signals there. So at the end of the day just remember everything we do in medicine has a risk and a benefit. There is no medication no procedure that does not carry significant risk. I just explained to you before oral contraception has a much higher risk of developing free-moving sinus thrombosis than the COVID vaccine however you're not seeing a big uproar in the media about that. So we have to take it all in context. Let's see what the FDA has to say. If anything this should give us some comfort that the safety monitoring is robust it's doing what it's supposed to do. So in the end we're going to have to see what happens. My impression is that it is certainly possible if not probable there is a tiny risk of this problem happening with these vaccines and perhaps that's happening in a particular subset of patients. I do think we're going to be resuming these vaccines sooner than later but there may be some sort of restrictions or increased monitoring in certain populations. That's all for now and thanks for watching.