Thank you so much, everyone, for being here. I am Garima, and we'll be playing a real-life Dungeons & Dragon today. Fun fact, I've never played the original game, so good luck. Let's start with something about me. These are ways to reach out to me, because there was literally nothing else that I wanted to talk about me, except the fact that I work for this company called Espivital. It's great. I love working over there. I work in the R&D section, which builds the Cloud Foundry-ish things. Currently, I'm on a team which builds Bosch, Windows, stem cells, if you know about it. So as you all know, this is going to be a highly interactive session. I'm going to ask you a lot of questions, and in turn, I promise you will get a chance to ask me a lot of questions as well. So for now, let's start with how many of you go on call? Raise your hands. DevOps days, what should I expect, right? How many of you have enough team members? How many of you think that you have enough team members to go on call? Yeah, I see the number of hands are going down. How many of you think going on call is stressful? This talk is for you. On-boarding newbies is very stressful to go on call. How many of you feel that? Kind of. Awesome. And last question for this slide. How many of you know the original game? Yeah, everybody except me. OK, so I will be referencing a lot of things from this book called Site Reliability Engineering. It is also called SRE. You know it now, right? The SRE is now a position in companies. So we all know it. I'm going to call it SRE. OK, so let's talk about why and in what scenarios going on call can be stressful. So as per the SRE book, for one person, it is healthy to go on call once a week in a month. If you are going on call for more than that, if you notice the face is now sad, so if you go more than that, it is definitely going to be stressful. So let's do the maths. So one primary person to go on call, one secondary, followed by four weeks in a month. So all those who said that you have enough team members, do you have eight highly qualified, highly onboarded engineers to go on call? Not really. Awesome. OK, so you need eight people. If you count those numbers, it is eight. OK, next is incidents per day. So even if you have eight engineers and if you have 500 incidents happening per day, it's not a healthy team, and engineers will burn out, so the healthy count to go on call for maximum two incidents per day is good. Then how do we onboard people? We onboard people by drawing some boxes and lines about how the infrastructure is, which is great. And the person who is doing it, for that person, it can become really stressful. So if you did not notice, I'll do it again, the presenter in this case was smiling and now is not smiling because it can get overwhelming to do it every now and then. So what's the next way to onboard people? This is my favorite. I like fiddles. Fiddles are great. Once you're done with the fiddles, the happiness that comes that, I think something is so good, but it onboards only one person on the team who is doing it, not everybody is doing the fiddles. So that leaves us to not many ways to onboard new people on your project, on your product. That's why we have DND. So what is DND? You know the original one, but in this case, there is a dungeon master who is a person who was on call when some incident happened and the person went through all the steps to fix that incident.  And now instead of the person solving the problem, the rest of the team would solve the problem. We will see a demo of this, but before that, I'm just going to put a snippet to increase the number of slides in my presentation from a SRE book. So SRE book talks about it in these four or five lines where it says, blah, blah, blah, selection mechanism for picking a disaster, followed by a role playing in which one person plays a part of a dungeon master. In this case, the system. And the other person plays a part of the on-call engineer, what the on-call engineer said to do, and we compare this against what they actually should have done. These are the five lines in there. Good luck trying to understand it. But at Pivotal, we came up with a way to do this role playing exercise, calling it vengeance and dragons. So let's back this up by science. When you're in a stressful situation, there's this hormone called cortisol which gets released in your system. It is not a very great hormone. It just is useful to build your memory. And that's what we are going to do today. I'm going to put you all in a very stressful situation. Good luck. And this will build your memory. All right, what would you gain after this talk? You would gain that D&D is a non-destructive, fun, and easy game to play, and then stress creates memory and it builds your confidence and knowledge on your system. So how are we going to do this? We are going to do this on my infrastructure setup. How many of you know already about it? No one, like literally if you knew it, I would be scared. But so by the end of this talk, you would know more about my system. And in 45-minute talk, if you would know more about my infrastructure setup, imagine how much would you know about your infrastructure setup when you play this game in your team. So how to play? There's a dungeon master. Today, I am going to be proudly presenting as your dungeon master. And then there is the rest of the team. Let's do a quick demo. The dungeon master says, we have a problem. We have 114 incidents raised by our customers that they cannot access the first page of our website since today morning. Oh, I see. Did we deploy something new in the system? Yes, the dev team did a deployment overnight for their customer X feature. Are we receiving any logs for these requests? No, there are no logs in our load balancer. In fact, the number of requests are less than yesterday. Interesting. What is the URL that they are trying? It's dungeonsareal.com. What is the IP that we do when we do dig dungeonsareal.com? We get <IP_ADDRESS>. This looks like an internal IP. What should be the IP of our load balancer? It should be <IP_ADDRESS>.23. Let's change the domain setting to first point to this IP. Then we can investigate on what happened. So this is just a quick demo. I'm sure you could understand everything from my voice modulation. So we are gonna do similar stress situation and we are gonna solve something together, but there are some assumptions. This is not production environment for two reasons. I will not be presenting my production environment in front of such amazing DevOps engineers because it's risky. My company should not even allow it. And second, currently in my team, I don't have a production environment, so. So these are the assumptions. Assume that when you see one request in the setup, consider that as a 10,000 request because we want to consider it as a production environment. And it is a startup, kind of a setup. So everything is fragile, things are just built on top of everything. So yeah, that's all the assumptions for today. And I am done asking you questions and now it's your turn to start asking questions. Are we all ready? Okay. Welcome to World of Dragons. While, so we are a startup, while doing some digging somewhere in Earth, we can't tell you the actual location, we found a lot of dragons, like literally real dragons, not joking. And so now we're a startup who sells dragons. We have this amazing API for our vendors, like Amazon and other Best Buy kind of vendors who can just like pull our API to get dragons and we will ship them to dragons and everything was so good, our life was great, we were building a lot of money until we decided to do a field trip for our software engineers to the dragon place. It's not good, it's not good. I'll tell you because we are a startup, right? So we had one DevOps engineer, I should say one SRE that we had that got eaten up by a dragon. Life is not that good after that. So we have a lot of problems and we don't know solutions to that. So we hired all of you to help us fix these problems and let's start to work. Let's start with our first problem, which is mystery of certificates. We are getting a not secure connection on our website and all our clients who use our APIs are also complaining the same. How would you fix this issue? Let me show you the issue quickly. Yeah, so this is our great HTML CSS and everything included in this website, as you can see. And there is this, things look normal to me, there's just this red color over here, which I know what it is, but for the sake of the dungeon master, I don't know what it is. Can someone, or I'm sure many of you, help me fix this problem? Can you, yeah? If you explicitly type in HTTPS colon slash slash before the domain, does it work? That's a good question. Oh, it was. It looks like it was there already. Okay. That answered my first question. We have a question over here. If you view the certificate details, is it still valid or can you look at the validity date? Yeah, what detail, by the way? Can you explain what the certificate detail is? Hit F12. Hit F12, all right. You click view certificate. Sorry? Yeah, there it is, awesome. Please expand the details. So it says expired, that is interesting, but that still did not fix my issue. All right, we know the issue now, so that's great, that's great. Help us, like, I understand that this is a certificate, so I need to go and change certificates somewhere, but can someone help me how to go where and make these changes? Go to the machine where the web server is running? Good question, very good question. We are a startup. We have 5,000 machines. Which one do you want me to go in? Why do you have 5,000 machines? We are a startup. You take a name, we have everything.  We are hiring as well. Okay, are you in the cloud or on-prem? On both. We got everything, like, as I said, we are a startup. Okay, then if it's microservices, then go to the load balancer, machine where the load balancer is running. Yeah, which load balancer? Load balancer, the service. The service, yes. Yeah, so you would have multiple instances of this service running behind the load balancer, right? Help me get there, our DevOps engineer is dead. Help me get to that. I understand, so you go to the machine that's running the load balancer now. How do you get there? If you don't know your own infrastructure, that's a problem.  All right, we had. It was under construction. We have a map, yeah. So the question was, do we have a map of infrastructure? It was under construction when our DevOps engineer was eaten. You had, use dig or NS lookup? Use dig or NS lookup, okay. Let's do that. Dig what? Dragon's world dot. Okay, so we get something here. Do you want me to remove these things? We got something here. 35, 241, 243. Awesome. What do we do with that? Console and find that IP address. This startup is not pivotal. So you want me to find this IP address? Like, what should I do with this? Is the machine Linux or Windows? Either you SSH or RDP into it. In front of this? Okay. Before I answer your question, someone else, I'm sorry, I don't know your name yet.  Okay, what should I do next? Hit advanced, okay. And I'm saying proceed. Okay, this is what we get. Okay, so it doesn't have web interface, but I'm not sure. Can you trace throughout the IP to figure out which of your providers the load balancers is hosted on? Awesome, that's a good question. Trace through. How do we read this? We know we're at UIC, so we're starting at one and going out to the internet. It ends at googleusercontent.com, so we assume it's on GCP or Google App Engine? Oh yes, that rings a lot of bells. We have something on Google Cloud. You name it, we have it. And this is how the platform looks like. What to do here?  Tell me what to do next. We have AWS. We hired you. Okay, so we know that there is something hosted with some IP on GCP. What should we do next? GCP, so first figure out what service manages certificates for your infrastructure, for your VPC. Okay, so let's Google. What is that certificate management service called in GCP? Google Google Cloud. Yeah, I mean, again, I don't know. Maybe somebody here can tell better about GCP. Okay, so find. Where's Harvey? Harvey? Harvey? Okay. Yeah, so once you find that certificate managed, most clouds have a centralized certificate management entity for whatever you're deploying. So once we find what that service is, from that portal we can get to that and then there see what's going on. Said something. So, what he suggested is, go to the certificate provider interface on Google Cloud. We don't have that, but we have something that, some word that you used, so I'm gonna navigate over there. We do have a load balancer here. Yes, that's the certificate. And I look at it. It has expired. And thank you for getting us till here, because obviously I did not have any idea that we are hosting our services on Google Cloud using this load balancer, so this is great. Now I know where to go and change the certificate from. Thank you for solving that problem. But this is literally the beginning of the problem. Now let's get on to some real problems, right? This is just a test whether you all got DND or not. Now let's actually play DND. So, next problem. Request Sarin Dungeon. After a late night deployment of our new app by our dev team, because you know we are all agile, we don't deploy, the dev teams deploy. We have started getting errors where all our clients are randomly receiving HTTP code 429. This issue, as usual, did not happen in our staging environment. Help us fix this issue. Well, again, first question I would have for the dead guy that, is QA representative of production environment or not? You said, you tried this in staging and nothing went wrong. My question is, is staging representative of production? For the dead guy? Yes, it's all representative of production. We are all agile. Okay. Well, the dev of person is dead, right? So, then first we would, well, in this case I would go to the version control management system and see what that deployment, how much of a change was it? Is it a big impactful change? Is it a minute change? What is it, right? And at this point I would ask, there should be a rollback strategy or you spin up another staging environment or another prod environment that's representative and deploy the application there so you can test within it. But first you roll this back so you can get service up. What's the record for 29? Okay, Google this. Too many requests. Too many requests, okay. So there are too many requests. Maybe there's a new feature which is highly in demand. We had some hands over there. Yeah, Mac, get on it. How can we reproduce it? Is it random or is it some pattern we can figure out? Good question. Let's see it being produced. There it is. Some reasons. So we can reproduce it. Try scaling up. Literally the DevOps engineer is dead. If you're talking about scaling up. Walk us through it. I mean again, there's this concept in cloud called auto scaling group where it would scale automatically. You imagine that we must be using auto scaling group, right? You said you're all agile and that would imply good practices. Wait, whoa, whoa, whoa. Whoa. Whoa. Okay. Who else? Let's try this. Let's try some other folks, some other suggestions. Have you rolled back the change to see if this fixed the issue? Great question. Because that's the first thing they force anyone who does changes immediately if we get this. They force us to roll back and if it fixes it, it's our fault. Makes sense. Makes sense. We can totally do that. Our dev team is sleeping right now. So we will pass on to that for now, but we will totally do this. Was there an update to the web server which changed their max request count? Very interesting. How should we find that out? Well, see, there are VMs. There are ways to get to those VMs and find the IPs and SSH onto those VMs. You all need to just tell me what to do. To services and try to see if that helps. Great question. How do we get to those VMs for that? Do you have any ideas how to get there? It depends what you're hosted on and if you have those guys available. But like for example, if it's a web server, just restart HTTP or something, no? Yes. So we got to a load balancer in previous issue. Do you wanna add to that? So he said, do we need to figure out all the app servers which are behind the load balancer? Yes, yes. Amazing. Let's get there. Sorry? Good idea. Okay. So I see some instance groups word here. So I'm just gonna click on that. And these look like VMs to me because they have external IP, but happy to be challenged over there. Does this look good? Should I get on to one of these? I would have asked you how to SSH, but that is too basic. Okay, we are in here. You want me to do a top? Should I sort this by memory? Or load average or whatever? Okay, I'm doing it. What's your thought process? Do you wanna just like talk about it? He's looking for resource contention. Nice idea. We have our hand raising over there. Are the instances healthy in the load balancer? Let's check that. Since I use Google Cloud, this means three instances are healthy. So the instances are healthy. We have our hand over there. Can, are all the 429s happening on all three instances? And if it's isolated, can that be not destroyed but removed from the load balancer so you can interrogate it while it's still alive in situ and still maintain actual 200 connections? Amazing. How do we check that? How do we check that? How do we check that? Is it happening on all three of them? So earlier in the scenario, you mentioned that you were getting 429 errors. Yes. So was it through whatever, if it was through a ticket system or people were calling, perhaps if they had a direct IP, if not then we should have a log, do we have a logging system that through there, a log where we can check and try and isolate where, if it's one particular or if it's across the board? Nice, nice. Can we try to hit each of those IPs and that's behind the load balancer? Okay, does that answer the question that you have been asking? Right, how to get one of these? Okay, let's go back. I'm not sure if I had put it in a VPC or not, so it's behind, it's behind a firewall. See, it is not reachable. But why am I giving answers? So this is not reachable. What should we do next? Can you say, hit it locally from the box. Good, good. Okay, so we are already in the box. So I'll do something like local IP address slash, let's just do this, right? Are we getting any answer? Not really. Should I try the private IP of this? He said, what interface or IP is it listening on? You have to check the config and my question is, how do we check the config? Well, we also know. On the web server, you're running the browser, could tell you that now you can hide it. Yeah, okay. So yes, check the config. Probably HTTPD would be the, yeah Apache would be the most common. We also know we need to hit on HTTPS and it's insecure right now. You had something to add. Go back to the Google interface, I should tell you. Okay. No GCP at all. If this was AWS, I can tell you exactly how to do it. He's saying if this was AWS, he could tell us exactly what to do. And there's another hand over there. Do you wanna just like move in a little bit so that Matt doesn't have to walk too much, even if I like that? Can we try running that curl command on the box again? To hit HTTPS and to hit with the insecure flag? But not on the internal network, on the external network. Okay, on it. With dash, dash and secure. We have a handle.  So I don't know Google Cloud, but there was a tab above that said monitoring. I wonder what happens. Hmm, interesting. This tab, did you mean? Okay, there it is. You had high expectations, didn't you? Why was there a spike early in the morning? Was there any activity? Because I was testing the presentation. Let's do it six hours. Maybe I can't test it as much. Are you able to ping the gateway or the other nodes behind the load balancer as well? You mean other instances in the load balancer from browser or from within? From within, if you're able to ping the other nodes and if you're able to connect to the load balancer from there. And within the load balancer itself, you might want to check the configs to see if those nodes are actually included in the config file on the load balancer itself because it might not be configured properly maybe. I missed it, can you say it again? Oh sure, the config file on the load balancer itself to see if it includes those three nodes. Okay, so you want to see the config of load balancer itself? Yes, in addition to the pinging of the gateway and the other nodes. Okay, so I did try a call here. It didn't work, do you want me to ping it? Sure, all right. Okay, it looks like it's timing out. Try the internal address is what I heard, so let's do that as well. Okay, so let's take a step back after this and let's see what are we trying to get to? Like, can somebody talk about the thought process? What are we trying to get here? So he said, we want to know what process is running on what port in the VM so that we would know what to do with it, something like that? Where to keep looking. Where to keep looking, all right, okay. So, and I understand that we are pinging different machines in the system, but on the machine itself, we haven't done much. We want to figure out if the problem is with the load balancer or with the apps running on the app servers. Awesome, how do we do that? Do you have application logs for the load balancer or do you know what the application logs are on the app servers? I don't have logs from load balancer, but I do have logs on the machine from the process itself. However, you will have to help us get to those logs. I don't know what your question is. I mean, this is about, this whole session is about, right? We are learning some new tools, new techniques from each other, how do they debug, and that's what this whole thing is about. So I tried it from two other devices and it comes through just fine. You got to keep trying, you would see it. All right, so now that randomly I started seeing the count work on your laptop too, tells me what Jeremy was saying there. Maybe it's an issue with one of the machines because the load balancer keeps sending us to the bad one first, then it probably did a round robin, had enough, and then sent us to the one that's working correctly. Fair enough. Okay, so now I would say you remove them from the group and then attach them to an external facing network to the machine and try each machine individually to see what the issue is there. All right, is there a better way of doing this because this will bring down the site, right? If I remove them all from load balancer. Well, can we look at the web server logs and see which one is throwing 429s? There was an idea, but it got cross-questioned with how to find the logs. You have something. Can you go to the details tab on the load balancer? Can we go to the details tab on the load balancer? What's your thought process? Okay, when you want to curl those, hit them on port 8080. Oh, how do you get that? Do you want to talk about that before? Why we saw that? Because it says incoming traffic is directed to port 8080. Because of this. We missed this bit. Okay, so let's do this again on port. Not ping, curl. Try the other two. That's good, all right. So what was breaking? Ha ha, you didn't fix it yet? Okay, so now we can hit it internally. Wait, is this the IP address of one of the nodes or is this the IP address of the load? It is one of the nodes. Okay, try the next node. Next node? Okay. Please. Only because you said that, I'll do it.  Okay, let's do it for 197 this time. You got a space in there. Third one, ready for this one? That's that dash TULNP space pipe grep 8080. Is that what you want? Space pipe. Like your IP address or no? No, it's pipe. Okay. Grep 8080. Ah, before I hit enter, do you want to explain the room? What it is going to do? You're showing all listening ports and you're looking specifically for port 8080, which will tell us which web server you're running. Add a dash A. Dash. Oh, okay, pseudo. There's some. That IP address in the list is not in the batch of load balancers. Is that what you're saying? I thought it was, I see two different 35.add addresses. Third one was a 35, but that was the app server itself, not the load balancer. The load balancer IP as we saw was 35 this. I saw a hand somewhere, maybe not. Okay, so what do we get from here? Yes, this thing. Okay, before I hit enter, you want to explain people and this time I'm going to request Matt to give them my. Sure. So I just want to see what file handlers are open by that process, the PID 14704. That should at least tell us where the logs are going. Makes sense. Okay. I wasn't expecting that, honestly. Yeah. Any other way? I know. Okay, there's some. Can we get for dragon and varlog? Varlog, that for dragon? Yeah, can we act for dragon? Okay, that's because varlog is a place for all the logs in general and you have high expectation from this setup for standards and everything because I talked about agile. All right, all right. You hope and pray the defaults weren't changed. Oh, you want me to get there? Varlog, there's something like a dragon. Oh, you did have a high expectations, didn't you? You want me to tell those logs? Okay. Well, that's how production systems are. Not that straightforward, right? But it says health check is healthy. Can you say it again? Grab that file for 429. 429 and I heard we are still doing it on one node. Right, yes. So our working theory was one of the three nodes is bad but then we did a curl with slash dragon API with all the three IPs and all of those returned a yes. Can we run Bosch? So can you say it again? Once. We tried once and once it returned yes. Okay, what was the next thing? Can we try to run Bosch? Is Bosch on here? High expectations, don't you? Right, no. Don't have Bosch. Chef client or puppet, Ansible, anything? All right. I'll add it to our backlog and then we'll create stories for it. So do we know what image this VM was even created from? Like how do we know that this image was not updated along with the application? That might be causing the problem. Nothing happened to image because our DevOps engine is dead. Literally she did not make any automation around recreating the VMs yet. We are almost there. We're almost there. So we are on the VM. We know dragon API is something which is running with some process. We want to get the logs. We are almost there with the logs. You talked something about version control. Somebody else also talked about getting to version control. We just need some way to connect the dots. Anybody? Now, yes. Literally could not hear any of that. Matt, you got to pass the mic. Can we make a request while tailing the log to see if something pops up? Good question. Okay, so let me ask you such again. So we tail the logs. I should not tail the check logs, right? We are empty. And then call itself at port 8080 for getDragonCount. Sorry, one. That should have worked. Let's see address you're coming. IP right? Oh, I'm sorry. You have to do HTTPS? Let's get the right IP. Let's check the private IP. DragonCount request responded. Okay, so based on your hint, I'll go let's find where this application code is. Based on what? Based on your hint for the version control and something else. Somebody said, where's the source for this? Right, it's being served out of var, not W, what is that? Var, W, W, HTML. Let's try that directory. W, W, W. Don't have that. No, that directory doesn't exist? No. Okay, let's grep for the HTTPD process. Is this Debian or CentOS? So you want to get to the place where this code base is sitting. Anybody here can talk about how can we get it using slash proc? So we have a process ID.  I don't remember which part exactly, but if you ls dash l... I'm just trying to get the... Okay, so you want me to do ls... Oh, sorry, yeah, ls dash up and enter. And then that's fd, yes. Could you ls in the fd directory there? So the fd directory, file descriptor, right. And you need to sudo. So that's the files that the process is opening and sockets and sockets and if you see... But I guess the source code is closed. You're almost there. If I get here, and if I do ls, that's where the git repositories are sitting and the name of the repository is dragon API. Can you run ls dash al? Ls dash al. Okay, can you run git status? Where are you talking from? Why can't I see? Yes, okay. Wait one second. What do you want me to see? I'll do here, okay, git status. Okay, I don't see a dot git folder. Okay, in interest of time, we're almost there. The code base is actually here, not this one, DevOps stays here, there was a dragon API and there it was. And if we had looked for it over there, there was a throttling being added from developer themselves. And that's why it was working well in staging because not that many requests were coming up. So this case was when load balancers itself, IP was being rejected because load balancer has finite number of IPs. This is one of the problems that I faced when I was actually in one of the startups. I had few more cases of trying to solve here, but in interest of time, I'm gonna move on to recapping it. We use this technique for not just going on call and not just for incidents, now we use this techniques for any kind of debugging that we do in our teams like pipeline investigation, why something is not working and if one person or a payer solved the problem, they would present it to the rest of the team and that's how we increase knowledge and context on our team. That's all, thank you for having me here.


In the name of Allah, the most universally merciful and the most particularly merciful. Allah is manifesting mercy on us by gathering us to gather for Juma'a, to gather for a Jama'a, a congregation in which the hearts are congregated towards Allah Azza wa Jalla alone and where the hearts are not focused on anything except Allah and what our rights are on to Allah and what our rights are to fulfill for Allah. This Deen can be summarized, the actions of this Deen as Imam al-Hanifa says, it can be summarized as what you are responsible for and what the responsibilities are towards you. And so this Deen is an understanding of the transaction or the conduct that you have between firstly yourself and your Lord Azza wa Jalla and then the responsibilities you have to yourself and then the responsibilities you have to others. And from the mercy of Allah is that he has not how this accountable for anyone except ourselves. Allah says that you for you is what you earned and for them is what they earned.  When we're raised on the Day of Judgment we're not asked of what others did, we're asked of what we did. And so in that sense the people that we live with, the people that we are surrounded by, the people that we interact with in our daily lives, those are also people Ummatun Qadhalat in that sense and they're also people who are passed and on the Day of Judgment when we are risen, when we're resurrected, they are as good as from that vantage they're as good as gone. And in this regard Imam Abu Hassan Al-Shathali, one of the great spiritual masters of this tradition said, Don't demand others to honor you, that you're not charged except with yourself. The only responsibility that Allah has given us is to ourselves. And so when we spend time thinking about what so and so is doing or how so and so is doing their acts or what they're busy with or what they're doing to us or how they made us feel this distracts us from our responsibility on this earth that Allah says that we have made you in this earth a vice-regent, a representative of Allah on this earth, that He has made us a representative and not made us responsible for someone else. That when we see that our role on this earth is to properly execute the laws and the morals that Allah has sent us with, us with and we see that that khitab that addresses direct towards us and not seeing as anyone else being responsible when we take responsibility for the rights that we have towards Allah and then the rights that we have towards ourselves so that we can properly enact the rights towards Allah and His creation and that is when we are focused with what Allah has charged us with. That is when we are focused with the fact that Allah has made us responsible for us and not for someone else so that when we busy ourselves and preoccupy ourselves with what so and so is doing and what is the fault of so and so that it distracts us from our own faults, it distracts us from what we need to fix within ourselves, it distracts us from the fact that perhaps when we see that someone else we might be standing next to someone in prayer and we're so focused on how fast they're praying and we feel like their prayer is not as good or that we feel that somebody else's will do when we see them as not with so much presence or someone else's treatment of someone else is not so well when we're concerned with that we forget about what we are to do. We forget that we are first asked to take to take care of ourselves that our own houses that we're supposed to put those in order before looking towards someone else, that before able to deal with someone else's problems we're supposed to be able to deal with our own so that when we are distracted from what our goal is we're distracted from our own questioning that when we're raised on the day of judgment we're not going to be asked what it's so and so doing what did the person next to you praying do or what did your friend do or what did your co-worker do or what did this person or that person do in their life that's not what we're asked for. We're asked for what did you do and so when we preoccupy ourselves with what other people want I'm sorry with what other people are doing then this is Allah taking us away removing us from actualizing our actual and from recognizing our actual job on this earth. On the similar line Imam Hussana Shadr the same the same Aalim says that if Allah wants the humiliation of a particular person he veils from himself his own faults and if he wants his honor if Allah wants this person's honor then basarahu biha then he shows him his own faults why liyatuba minha so that he may he may make tauba he may repent from them he may turn back to Allah and recognize what is wrong with him before recognizing what's wrong with someone else because if we want to approach Allah on the day of judgment with qalbin salim with the sound heart then focusing on what others is doing is not going to do much benefit to us focusing on what we are doing is what is going to benefit us on that day so Allah is not going to accept the person the day judgment unless they have a sound heart and how can we rectify our hearts when we're worried about the hearts of others or how can we expect to do some sort of good to others if we haven't fixed what's in ourselves we haven't rectified our own fate our own affair how can we expect to cause some positive change in the world how can we expect to remove oppression when we haven't removed the oppression that's in our own hearts how can we expect to do something about the greater environment when the environment of our own hearts is still sullied and still polluted how can we expect to do something about others if we haven't recognized that Allah is the one that does something for us and for others if we're so focused on what can I do to fix this person or that person and we don't recognize that Allah is on the fixes me and that person how do we expect to have some sort of change for the better for not for not only others but for ourselves how do we expect to reach Allah and to reach the end of this this journey that Allah has set us forth on how do we how do we expect to reach the end of Sirat al Mustaqeem the straight path if we're wavering left and right to what others are doing and the first degree of finding that Hidayah that Huda that guidance is asking the first degree of finding our way to Irshad to Rashad to guidance is to ask and not to ask others but to ask Allah before we ask others this is not to deny that we should take our means and enact the means that Allah has given us but as is in the in the famous the famous hadith in the abayn of imam no we the Prophet says if you want to ask and ask Allah if you seek help and seek help from Allah meaning seek ask Allah first before you ask someone else so before we ask someone else to change what is in that what is in them or before we even ask ourselves to change what is in ourselves the first step in this guy into fighting this guidance is to ask and to ask only Allah and to supplicate the Allah I find this problem in myself Allah fix it for me Allah I find that I think I'm better than others or Allah I find that I'm lazy in my worship or Allah I find that I am rude to people I or Allah I find that I'm harsh with my family or Allah I find that this is my issue and that's my issue then before we take those steps and we have to take those steps we can't deny that we have to take those steps to fix those issues similarly if we find if we have the ability to help someone else we take those steps but first ask Allah that before we find ourselves criticizing someone we should ask ourselves have I even woke up and have I woke up and no one else is awake and ask oh Allah this issue that you made me see in someone else first of all protect me from that issue and then if that still is if that if I don't find that in myself Allah then fix then fix that and rectify that matter in them as well and that's the first step to guidance that's the first step to completing surat al mustaqeem completing the straight path and none of us are free from any defect or fault the first step of a person being in fault one of the greatest signs of a person being in fault is that he sees no fault from in himself if he sees no fault in himself Allah has veiled him veiled him from his own reality and then he thinks I don't have to do toba I don't have to seek a stifar I'm fine everyone else is the problem we think everyone else is the problem that's usually a sign that you're the problem when the Prophet sallallahu alayhi wa sallam sought forgiveness for Allah 70 some 70 odd times and some narrations 100 times a day and he was the perfect man sallallahu alayhi wa sallam that he was the best representative of Allah on this earth but there was no better Khalifa of Allah on this earth than him sallallahu alayhi wa sallam and he sought forgiveness from Allah 70 to 100 times a day and and more right in Arabic we say 70 or 100 it means many times it doesn't necessarily just mean 70 or 100 if he was to ask 70 hundred hundred times a day who are we to see ourselves as safe and secure from the faults that Allah has placed in us Allah has placed faults in us and that's a reality but Allah has also made us his Khalifa on earth Allah's also made us his representative and part of us enacting and taking taking on the characteristic of a tawab Allah's a tawab the repentant and there's a saying some say it's a hadith some say it's not but the meaning is sound to khalaqu bi akhlaq illa that take on the characteristics of Allah that Allah's a tawab so our job our ability to take on the name a tawab there the pen that the one who turns it's for us to turn to Allah it's for us to see our our faults and to turn to Allah and say Allah even with this deficiency that I have your mercy is greater and you are more accepting that you are a tawab and a sign that Allah is accepting the tawab of a person is that the fact that they made that tawba because the sign that they have been accepted is that they've turned to Allah and Allah wanted them to turn to him the sign that a dua is answered is that the fact that the dua is made the fact that Allah hears you asking is a sign that Allah is ready to accept that dua and all is all that is demanded from us is that we ask if you came late and you have no idea what we're talking about this the responsibility of a human is three things fulfilling the rights of Allah fulfilling the rights of himself and fulfilling the rights of others if one is distracted with what is happening with the other and not worried about what is happening in terms of their relation to Allah this is an impediment and an obstacle in the in the course in the journey that is to our end to Allah that our journey when it when we and our ultimate end is with Allah right if if we are promised Jannah the greatest pleasure of Jannah is not the many descriptions of Jannah it's the fact that we are in the highest presence with Allah that is the end of that journey and if we are so focused on what is going on left and right and we are distracted from our true orientation our true objective and our true goal we'd find it difficult to fulfill our responsibilities so we ask a lot to be people who are cognizant of our own responsibilities we are we ask all that we are people who are cognizant of our responsibilities to Allah we ask a lot that we are people that are cognizant to our responsibilities to ourselves we ask all that we are people who are cognizant to the responsibility of others of our families and of our and of our of our workplace and of our co-workers and our peers and our friends and our children and our parents and and whoever it may be we ask all that we are people who are responsible we ask all that he makes us people who are cognizant of that responsibility we ask all that he makes us cognizant of the fact that we are the representative of Allah on this earth and that we not fail, that task that Allah has charged us with. We ask Allah that He does not make us people who are heedless of the fact that He has charged us with ourselves and not with others. We ask Allah that we recognize our own faults so that we may seek forgiveness from them and that we may rectify them. We ask Allah that we are people who are lessons for others that when we rectify ourselves the issues that we see in others are also rectified. We ask Allah that He makes us a people who are sincere in turning towards Him and sincere in seeking to reach our end with Him. We ask Allah that we are people who are sincere in our journey towards Allah. We ask Allah that we are people who are completely oriented towards Him and not turning left or right in this journey towards Him. We ask Allah that He rectify us and He rectify others and He rectify our families and He rectify our loved ones and He rectify anyone who has some sort of issue with them so that they meet Allah and Allah is happy with them. We ask Allah that we be people who have no source of peace other than with Him. So we say, Allahumma anta salamu minka salam, tabarata yadil jalali wal ikram hayyina wa adkhilna darul salam Rabbana aatina fiddunya hasnah wa fil aakhid ya hasnah wa khina aadab al-naar. Rabbana hablana min azwajina wa dhurriyatina qurra ta'ayuni wa ja'nna limuttaqina imama. We ask Allah that He have mercy on our teachers, particularly our first teachers or parents. We ask Allah to recognize that Allah and His angels send their salutations and medications on His beloved, peace be upon him, and they were asked to do the same, inna allah wa malaiqatahu yusalluna al-nabi.  Inna allah ya amur bed-adil wal ihsani wa ita'idil qurba wa yana ha'in fahshai wal munqiri wal baghi. Ya'idukum la'allakum tathakkarun. Uthkur Allah, yathkurkum, remember Allah and He'll remember you. Wadu'uhu, yastajibilakum. Qa'an him and He will answer you. Waladhikr Allahi akbar. Wa allahu ya'alum ma tassna'una wa aqim as-salah.


In this presentation we will enter a transaction for a sales return. We have our data up here on the left side. We're going to enter this information into the general journal and then post it to our worksheet here. Not posting it to the general ledger but posting it to a simple worksheet that will show us immediately the transactions that will go on a worksheet that is assigned well for a small number of transactions where we will have a beginning the entries that we will enter and then the ending balance to showing us the result of the transaction worksheet is formatted in terms of the assets and green liabilities in orange equity in light blue and revenue and expenses in dark blue. We have both debit and credits represented in terms of a debit and credit column in the general journal as well as credits will be represented with bracketed or negative numbers however in the trial balance we're going to represent the debits and credits in order to save time and simplify the formulas to have debits as positive numbers and credits as the negative numbers the check figure to see that we are in balance is the zero adding up debits and credits debits minus credits equaling zero. We then have the net income calculated here we got the sales 107 the cost of goods sold we are limiting the accounts to just those that we will be working with here so that we can see the results in terms of net income and account types here is our information on the left side we're talking about a sales return and we're going to have this is the sales price and this is the cost that we originally had so what we're doing here is we're saying that there was a sale that happened and the inventory was returned and therefore what are we going to do how are we going to deal with this return of the inventory now to do this we can start thinking through what type of accounts will be affected if the inventory gets returned well we know that we're going to increase inventory we got it back unless it's damaged of course and then you know we could start thinking through it in that format but since this is a more unusual transaction and it's closely related to the sales transaction probably easiest to think about the sale and then reverse it so even if we didn't have the numbers like we have here you want to basically go through or I suggest the process of going through first thinking about what did the sale look like and then reversing it so remember when we sell inventory there's two transactions that we can think of it as meaning we can have the transaction related to the sale similar to a service company similar to a company that doesn't have inventory but recording the sale for work done and then we have the second component dealing with inventory and then related expense of using that inventory to generate revenue cost of goods sold so if we think about those we're going to put that into our worksheet down here and just write that down and then we can easily look at it and see what we need to do in order to do the reversing of that in order to deal with the return that we have so first of all when we make the sale if we made it on account we didn't get cash we're going to say it went into accounts receivable accounts receivable has a debit balance we're going to make it to go up by doing the same thing in our worksheet we're not going to make it go up here for the return i'm just saying what happened when we made the original uh the original sale well we know accounts receivable went up so i'm going to put that in our worksheet it's down here we're going to say and so i'm just copying this i'm just right clicking copy we're going to put that in the worksheet right click and paste we're going to make a mock journal entry just to uh then reverse it for the seven or nine hundred and eighty so put nine eighty then we'll credit something for nine hundred and eighty we could put a negative negative nine eighty here or i'm going to use a formula instead of equals a negative of this number take that number and flip the sign so there's our nine eighty credit and then the other side of it is going to be what went to sales our revenue account so if it was a service company it'd be fees earned if it's a merchandising company typically called sales has a credit balance it only goes up in the credit direction we will then credit it so i'm going to copy that i'm going to paste that down here right click and paste it 123 going to we could indent it by going to the home tab alignment increase indent or just double clicking and hitting the spacebar three times next we're going to have the fact the inventory must have going down when we made the original sale that's merchandise inventory this must have gone down so it has a debit balance to make it go down we do the opposite thing to it a credit so i'm going to copy merchandise inventory right click and copy scroll back down we'll skip a line skip along the line to put it on the bottom we're going to put it here i'm in g 23 and sell g 23 right click and paste 123 and then we're going to put that in the credit column the amount 700 so in the credit column i'm going to put a negative 700 that'll put the brackets around it i'm going to then indent the sale go into the home tab alignment increase indenting then we know we're going to have some type of debit for 700 we can type in just 700 or use a formula of negative of that number take that number flip the sign it gives us a positive 700 then we just need to know what that account will be and it will be the expense it's going to be the cost of goods sold so we're going to say cost of goods sold as an expense it goes up in the debit direction we would have expensed it when we recorded this because we use the inventory in order to help generate revenue so we'll copy that we're going to put that down here in a g 22 right click and paste 123 so there's our journal entry that we would have so these are our two journal entries and in essence if they return to the merchandise we need to reverse this so we can do this step by step now note it's a little confusing when we reverse it because if we want to be in alignment with having the debits on top and the credits on bottom we're going to reverse the order in other words i could go up here and put a counts receivable and then have it in the credit side to mirror what we have here but we would rather put a counts receivable on the bottom so if i was going to copy this first one for example i'm going to say counts receivable copy that we're going to reverse it by putting it on the bottom right click and paste 123 we then indent it home tab alignment increase indenting and it's going to be for the sales price i'm going to put a negative of 980 now that of course makes sense because if they returned the merchandise if they had paid for it they would expect money back but they didn't pay for it they we still had it on a credit that's going to be typically the situation in most type of book problems so we're not going to give cash back what we're going to do instead it's say i will reduce the amount that we owe you will credit your account see we're going to credit the account there and that'll reduce the amount we're going to debit something of course and then that debit if we look here would be a reversal of sales meaning sales would be debited to make it to go down now this is the minor exception we have with this this is the thing we have to remember when we go through this process and basically i recommend remembering it like this sales doesn't go down it only goes up in the credit direction almost never goes down until we close it out at the end of the time period what we do instead is we add a contra account and the contra account is not it acts like an expense meaning it goes up in the debit direction and it brings down a net income however it's not really an expense it's a contra sales account it's an account that is going to be recorded in the revenue section of the income statement and to record net sales not net income net sales so instead of bringing this down we're going to put the debit here so we're going to increase this account a contra sales account which is in essence bringing down sales or net sales in any case so we're going to copy this right click and copy this and paste it up top right click and paste 123 so that's the one exception we'll have there then we can just reverse the second piece inventory is going to be reversed i would think of it first because it often makes most sense when we think of returning of inventory we got more inventory back if it was returned therefore the inventory should go up it has a debit balance we're going to make it go up by doing the same thing to it another debit so we'll copy the merchandise inventory i'm going to put it up top in c8 right click and paste 123 and the amount will be for 700 the cost so it's the cost then we're going to have a credit of 700 i'm going to do that with a negative of this number and enter and then we just need to know what that account will be and of course it is cost of goods sold this is really the most confusing component and that's why i really recommend doing this first writing down the sales transaction and then reversing it it's confusing for a few different reasons one is the cost of goods sold i'm going to copy this and put it on the bottom right click paste 123 in c9 then go to the home tab alignment increase in denting now cost of goods sold oftentimes is is just a confusing account for a lot of students it's an expense but it doesn't deal with spending cash which often a lot of expenses deal with and therefore it doesn't seem possibly like a normal expense maybe but it is an expense because we're consuming something merchandise in order to help us generate revenue it's also unusual because we're decreasing the expense we're crediting an expense which almost never happens so the rules basically expenses as well as sales only go up sales go up with a credit expenses go up with a debit this is the exception and exception to this rule and that's why it should be a bit confusing we we make a workaround in the sales side note by making this contra account but we don't make a contra cost of goods sold account we just reduce the cost of goods sold account so keep that in mind if we post this out then we're going to go straight down we're going to say sales returns and allowances that's going to be right here we are in i11 i'm going to say equals point to the sales returns and allowances it'll go from zero up in the debit direction to 980 and that of course goes up here it's a contra revenue account and that puts us out of balance brings net income down net income started here we brought it down to here then we're going to post the other side of accounts receivable so that's going to be here in i6 we're going to say that equals and point to that 980 this is a debit that's a credit it's going to bring this 10,200 down by 982 9,220 then we're going to post the second half here's the merchandise inventory here's the merchandise inventory here it's going up we're going to be in i7 i7 we'll say equals point to that 700 this 21,000 uh 27,000 debit will go up in the debit direction 702 27,700 putting us out of balance no effect on that income yet until we go to the cost of goods sold cost of goods sold here cost of goods sold here we're going to be in i13 within i13 we will say equals pointing to that 700 credit this is a debit of 5,000 it's going to go by down by 700 in the credit to 4,300 that puts us back in balance and that affects the net income so the net effect on net income is is this is going to be decreasing net income the contra account it's kind of acting like an expense but it's really a contra sales account it's going up and then the cost of goods sold an expense account is going down again spend some time with this transaction because these should look funny and it's something that's unusual the returns not a transaction we see all the time and therefore it's it's one that you want to spend some time with and I think the best way to do that is to record this transaction a few times and then reverse it and note that this transaction even after a few times of recording it is still confusing in and of itself because there's two components to it because there's a sale component and there's a merchandise or cost a good sold component


I looked at a number of programs and I decided on San Jose State because of its connection to Silicon Valley and its Tech Forward information science program. I am currently working in the field of website management, content management, content strategy. And so I am taking the information science, information architecture track of the program. I happen to work in retail at the moment. It's incredibly important for a retail company to understand their data and have strong systems for managing their data. And so that part of it, the structured content side of things is really exciting to me. The work I'm doing at San Jose State is entirely applicable to my day to day job. What I always tell people that I love so much about this program is how the professors tend to be people who are working in the field. So I have loved learning from people who are working at companies like mine or working in libraries and applying the concepts in their daily work. I think the advice I would give to students looking at programs is to not just think about the jobs that they know about, but to do a little bit of research into the jobs that they might not even know are possible. There are a lot of amazing jobs in the information science side that this degree really prepares people for that I don't even think people know exist.


I'm Uma, and this is John, and we're going to be talking about succinct verification of consensus, and we're part of succinct labs. Okay, so let's just start with the super high overview of the multi-chain and cross-chain landscape. So in the past few years, a bunch of different L1s and L2s and app chains have come online. They all lie on very different points of the trade-off curve in terms of decentralization, security, transaction, cost, and throughput. But basically, as the number of applications on these different L1s and L2s has increased, it's important that users are able to interact between these different applications. And in this future, in the multi-chain future, bridges have become critical infrastructure to make blockchains interoperable. It's really important that users' funds and assets are not siloed in one ecosystem, and they can interact with all the applications that they want to interact with in a seamless manner for the best user experience. So bridges are really important, and so let's talk a little bit about what bridges look like today. So today, most L1 bridges are built with multi-sigs or off-chain oracles. And so the high-level design is you have a multi-sig run by a centralized entity that watches for deposits on one chain, and then will sign off on withdrawals on another chain. And these multi-sig designs are generally pretty bad for a lot of reasons. So they're sensorable, they're not permissionless, they're very centralized, and they're actually, you know, empirically have been very insecure, and there's been many such bridge hacks. And I would argue that these bridge hacks are not only bad for the users who've lost billions of dollars of funds, but they're actually broadly bad for the entire space. It reduces the credibility of the entire space and then leads to severe downstream consequences like regulation. Okay, so that's kind of like an overview of the current problem, and so let's start talking about solutions. So what does a maximally secure and trust-minimize bridge look like between L1s? Well, we already have a mechanism for a decentralized group of people to come to agreement on the state of a chain that's called the consensus protocol. And so the key idea here very simply is that bridge security should be based on the same mechanisms that validators already use to agree on state, which is verification of consensus. So if you're able to verify the consensus of a source chain in the execution layer of a target chain, then in a trust-minimized way, you can know the state of the source chain without having a centralized intermediary like a multi-sig that has to sign off on what the state of a source chain looks like. So now that Uma has painted a picture of why these proof-based, proof-of-census-based bridges make a lot of sense, let's think from a high-level overview what implementing the system end-to-end might look like. So it means that we have some blockchain, like Ethereum, and over the course of its lifetime it's naturally producing information about its consensus, such as block headers, validator signatures, attestations, and other important metadata. And normally the peer-to-peer network of Ethereum would broadcast and gossip this information to all the other honest validators, and these validators will also verify the consensus algorithm. But what if instead of just broadcasting this information to other honest validators, we broadcast this information to a smart contract on the execution layer of another chain, which will re-implement the honest validator's logic? The reason why this is so powerful is that if we can verify the consensus algorithm in a smart contract, that means we can essentially run a light client on chain. And this means that essentially we can trustlessly access any state from the source chain by simply providing a Merkle-proof, proves the inclusion of some sort of data. And inside these block headers we have access to basically commitments to the entire state of the source chain, such as how much eth I have in my wallet, what transactions were sent in the past, and what events were emitted in contracts. And essentially we can access all this state with just a simple Merkle-proof if we have this light client running on chain. So the question is, why hasn't anyone done this before? And the big problem essentially with these proof-of-stake-based block chains is that verifying the consensus algorithm is really expensive. And in particular with Ethereum, the challenge is that the validator set is so large, there's over 400,000 validators. And to run this verification of consensus, you have to keep track of all 400,000 validators, their public keys, how much they've staked, whether some new validators have came in, whether old validators have unboughtened their stake. And this is quite difficult to implement a contract in a gas-efficient way. The second problem is that the signature scheme used by Ethereum's two beacon chain is the signature scheme that is BLS. And unfortunately, the elliptic curve needed to compute the signature is not currently supported by the execution layer on many EVM block chains. So even outside the context where we want to do this in a smart contract, basically running a light client for a proof-of-stake block chain like Ethereum is just ridiculously expensive even on consumer hardware like iPhones or laptops. Which is why the consensus folks essentially implemented a specific consensus algorithm for light clients. And this protocol is known as the SYNC committee. So essentially, instead of verifying consensus against 400,000 validators, the SYNC committee reduces this problem down to listening to the signatures of 512 validators, which are randomly chosen every 27 hours. And it works exactly as how you'd imagine. Essentially, these validators will sign every block. And if enough validators sign up from the block, the block is justified. And after some finalization rules, the block is also considered finalized. And obviously, at the cost of being much cheaper to verify, the SYNC committee provides much weaker security guarantees and it requires a two-thirds honesty assumption. However, we originally explored the possibility of verifying this light client protocol on chain. And what we found is that even in this scenario where there's only 512 validators, actually verifying this consensus on chain is still too expensive. So as to concrete reasons why it's too expensive, it has to do with the fact that you still have to store 512 of these public keys on chain. And you have to rotate them every 27 hours. And storing a state-on-chain is still very expensive. Furthermore, like I said before, the current EVM doesn't support a pre-compiles for the specific elliptic curve needed to verify these signature schemes, which means that we have to implement this elliptic curve natively in solidity. And in terms of gas costs, this is also prohibitively expensive. So our key insight basically is to implement the honest validator's logic on chain is obviously a super-computation-expensive task. But what we have available to us is the power of Zernald's proofs, which have this magical property of succinctness, which basically means that for any arbitrary long computation, we can generate a succinct proof which can be verified on chain. So essentially, the code in solidity is going to look something like this, where we have some function that validates a new block. And obviously, you have some pre-processing step. You have some step to verify that the current validator set is valid and that we've rotated the stakes and whatnot. And then finally, of course, we have to verify these BLS signatures. And as I mentioned in these previous slides, I've talked a lot about how verifying the validator set and verifying these BLS signatures is quite expensive. But what we can do is we can just compute a ZK proof that does these two expensive operations off chain. And instead on chain, we just implement the same function, but we swap out these two expensive parts with just the verification of this NARC, which will verifiably prove that we computed those things correctly. And I think this is really exciting because this is a framework that generalizes outside of this Ethereum Sync Committee and can be generalized to other block chains to basically make it very easy to verify the consensus algorithm in the state of other block chains in any execution layer that supports the op code for verifying ZK proofs. And I think in the same way that ZK is being used to scale the throughput of block chains, this is really exciting because it's showing that we can now scale the verification of these consensus algorithms. And yeah, for these reasons, we're trying to coin this term called proof of consensus. We're essentially trying to build these bridges, which use ZK SNARKS to generate a validity proof of the state of some block chain. And yeah, we believe that these things, these things called succinct line clients will be sort of the end game for cross chain interoperability between many different ecosystems. Okay, so John gave a really great overview of, you know, how we're going to use SNARKS to make these succinct line clients. So now I'll talk more about the details of how we did this for the Ethereum Sync Committee. So as John mentioned, the Sync Committee does two things. One is for every single block header, the 512 validators will sign that header, and they'll produce an aggregate BLS signature of the particular block header. And then the other thing they'll do is they have to sign off on the new Sync Committee that gets rotated every 27 hours. And it's really important that validators in the Sync Committee get rotated every so often for like security reasons and like making making sure that set is decentralized, etc. So we actually have two different SNARKS. So one SNARK verifies an aggregate BLS signature of a particular block header and make sure that like the signature is coming from the set of validators in the Sync Committee. So every block header that we want to have accessible in the execution layer of the other chain, we need to generate a proof that this BLS signature was like actually verified for that header, and we have to send that to the like client contract on the different chain. And then once every 27 hours, we have to generate another proof that will update the Sync Committee validators and set the new validator set that we're going to verify against. So we have two different SNARKS. So without going into too much detail, but to kind of cover some of the primitives that we had to build to produce those SNARKS that I mentioned on the previous slide, we used called GROSS16, and we use the programming language SIRCOM, invented by Jordy and his team at IDEN3. And so our suite of circuits was pretty complicated and resulted in over 70 million constraints, which is one of the largest circuits that we've heard of being used, at least in SIRCOM. And so some of the primitives we have to build were a public key addition and aggregate verification. So you basically have to add up all the validator public keys to produce an aggregate public key. We also had to implement verification of the block header. So this involved implementing pairing in a SNARK and then having the pairing check the BLS signature, which we, other collaborators at Xerox PARC Yi and Jonathan and Vincent worked on. And then we also had to implement serialization methods to basically check that these public keys are actually being, are actually like the correct public keys. So we had to implement the SSZ serialization that two uses, which is a SHA hash function, basically. And then we also had to implement Poseidon commitments to a vector of public keys, which is a SNARK-friendly hash function. And there's one trick that we use that actually helped us significantly save on gas costs of storing public keys. So I'll go into that a little bit, because I think it's pretty interesting. So at a high level, SNARKs have public and private inputs. So the public inputs are in red because they're bad, because if we have to, if we have to verify a proof on chain, then the public inputs means we also have to put that data on chain, which is expensive. Private inputs are transparent, like we don't need to put them on chain when we're verifying this proof. So if we were to implement the verification SNARK, we would have all the public keys as a public input, because we need to make sure the public keys correspond to the correct validator set. Unfortunately, storing public keys on chain is really expensive. So our idea was we could store the public keys instead. So we can store the public keys as a private input in the SNARK. As you can see here, it turns into green. And then the public input becomes this commitment, which is basically like a hash of the public keys. And it's like much shorter. But then the question is, how do we update the commitment when the sync committee rotates? So one idea is that when the current sync committee is signing the rotation of the new sync committee, they basically sign an SSZ, which is this ETH2 serialization they use of the new public keys. And so we could just use this SSZ serialization of the new public keys as the commitment. But the problem is SSZ is really SNARK unfriendly, because it's a bunch of shot hashes. So it's really expensive to compute in every header verification SNARK. And so what we, our idea was what we could do is we could map this SNARK unfriendly SSZ to a SNARK friendly commitment. And so our second snark that I mentioned below for the sync committee rotation, basically maps, it takes in the public keys, and then it produces the SSZ commitment and then also produces a SNARK friendly commitment called a Poseidon commitment. And then it just asserts that those two things are the same. And the bottom line is that we were able to save around 70 million constraints in our header verification SNARK, which we run every update that we want, which is like a huge savings in terms of proving time. And just for some benchmarks, so our sync committee rotation SNARK has the SSZ computation. So that's why it has 68 million constraints, which is quite a lot. Thankfully, the proving time is not that bad. It's four minutes. And that's the SNARK we run every 27 hours. And then we have another that's verifying the signed header that has around 20 million constraints, and it's proving times also around four minutes. Yeah, so everything that me and Uma have, we've described here, we have like working prototypes of it. And we actually built a two way like client bridge between Gorley and Gnosis chain, which is actually another L1, which implements Ethereum consensus exactly. So we're able to reuse many of our circuits. So we have a demo demoed us to sync to XYZ and fair warning, it's in beta and it works much better on your laptop. But here we have some screenshots of like what it can do.   And essentially, what's going on behind the scenes here is quite interesting. So when you make your deposit transaction, it basically stores some data on some contract on Gorley that indicates that you've made a deposit. And what you have to wait for is essentially for the light client on the target chain, Gnosis in this case, to be updated with a block header that can now reflect your new deposit. So you're going to wait for the finalization of an Ethereum block, which is around 12 minutes. And on top of that, you're going to have to wait for our proving time, which is right now is three minutes. And after that period, we can send a transaction to the light client to update it to the latest block header. And then to initiate your withdrawal transaction, essentially, you have to provide a Merkle proof proving that you made a deposit on Gorley. And this basically, this Merkle proof will essentially unlock funds on the target chain. And you can see, yeah, over after some time, like the top emoji becomes like green check mark. And yeah. So yeah, to zoom out a bit, I want to take some time to compare what the tradeoffs are between, you know, this proof based bridge versus the bridges that exist today, because I think they very much exist at different tradeoff points. And I think they have different use cases. So with these proof based bridges, you know, obviously, the really big benefits are that there's much higher security guarantees. You know, assuming that the ZK smart security is fine, essentially, you're able to borrow security directly from the source chain, which I think is something really powerful, and that many of the existing methods don't have at all. Another property is that because we don't have any additional trust assumptions, besides trusting the L ones validators, this this protocol can be very permissionless and censorship resistant, because theoretically, anyone can be running the operator that generates the proofs, and anyone can be updating the like clients. And yeah, for these reasons, it's much more permissions than J centralized.  But thankfully with like graph 16 and stuff, we expect that the verification cost will be around 300k gas. And obviously, another con is that you're gonna have to wait for the proof generation time. And in our case, you know, our circuits, we didn't spend that much time optimizing them, but you know, right now it's three minutes. And I think in the context of Ethereum, this is okay, because the finalization period is already so long. But for other consensus protocols, you know, you might want to have a much faster proving, proving time. And that's something you could probably fix by using your generation proving systems. And many of the techniques that a lot of the ZK VM people are exploring actually. Furthermore, another challenge with this approach compared to multi-sig approach is that for every new chain you want to adopt or a new L1, or even when a consensus algorithm on Ethereum changes, you sort of have to rebuild the circuit and you have to like kind of hand design these snarks to verify the consensus algorithm. And I think from the developer's perspective, this can mean that onboarding new chains can be quite difficult. So in terms of our roadmap, we're trying to take what we currently built for Ethereum to production. And I think our end goal is essentially to be this like trust minimized interoperable layer for Ethereum and other decentralized platforms. And we want this to all be powered by proof of consensus and adopting these values of decentralization and permissionlessness. And I think in the future, the two things we're really excited about is building these succinct ligands for other blockchain ecosystems, and also exploring these newer proving systems to decrease the proving time significantly. And yeah, finally, we wanted to give a big shout out to NoSysDAO who originally funded this work and super we're super helpful and also Xorex PARC where we worked for the summer and the community provide us a lot of support and help.