talk on CentOS infrastructure. Thank you. So we just heard a little bit from Fabian about the CentOS infrastructure and how the things are going for basically the things that come out of the Corsig. So CentOS Linux and the stuff that you would actually use every day, you download a CD, install CentOS, and yum update. That's all the infrastructure that Fabian was talking about. What I'm going to talk about a little bit today is some of the other things that we're doing in the infrastructure space and some of the public or some of the services that we provide to people in the CentOS community. But before I do that, it would probably be good to introduce myself. My name is Brian Stinson. I started working full-time for the CentOS project about a year ago. But I've been a long-time CentOS user as a sysadmin. You can find me at bstinsonmhk on Twitter if you want to tweet me or yell at me or whatever. Yeah, go ahead and find me there. So like I said, I want to talk about some of the resources that we provide and the things that you can do with them. So we have the stable base of all of those services that Fabian talked about this morning, the mirrors, all of the bug tracker, the wiki. All of that stuff is actually going really, really well. Some of the new things that we're working on is to provide services for people who are wanting to build things on top of CentOS. And that's where things like the account system that he mentioned. We'll talk a little bit about CBS and CI. But Fabian dove into a lot of the details of the hardware behind it. And I think that was a really good introduction into some of the things that we can actually do on top of that. So what we're going to look at today is a view into a day in the life of a contributor. What you could do on some of the infrastructure things that we provide. So we'll walk through a short demo of how to actually get things built into CBS. And more importantly, one of the real fun things that we're working on right now is our continuous integration setup. And I'll explain a little bit about how those work along the way and maybe lift up the covers a little bit and show you what we're working with. The next thing that I like to show, and that may be a little bit small print for those of you in the back, so I'd apologize for that. But this is actually a graphic that KB used in one of his talks when he was presenting some of his ideas to us for how people could contribute and where. And if you think of the whole thing kind of like a pipeline, what does someone who wants to actually get some software out there, they're working on a project, what do they want to do? And depending on which stage in the process they're in, they have a number of different steps. So they start over here on the left with their development process. They generate a set of sources, whatever that may be. And in the first case, we're gonna go through and talk about CVS. Right now we're talking about RPMs. So how do you get those sources actually from the source code built into RPMs, tested, and then released? And that all sort of feeds into that sort of nebulous user engagement because you don't build RPMs in a vacuum. You actually want people to go out and use the software that you packaged. So if you put yourself in sort of a mindset of someone who's building a software project, just to get a short poll of the room, how many of you have actually packaged something into an RPM before? Okay, great, we have a good set of packagers in here. That's awesome. So you'll kind of have an understanding of how this goes. So how many of you packagers have actually packaged something in Fedora? Okay, that's a good subset. How about for your own company or for your own use, just at home on your own machines? Okay, good. So those of you that have packaged for Fedora, you'll sort of be familiar with a lot of the tools that we use in CentOS. And that's sort of, that's a little bit deliberate. We wanna make it easy for contributors to use the tools that they're used to and to, you know, we do have our own separate ways of doing a few things, but we wanna make it as familiar to you as possible. And so to start off the whole process, one thing that we should mention is the special interest groups that we work with in the CentOS space. And special interest groups are basically just a group of people who are gathered around a common idea. So we have the cloud sig. I think right now that's RDO and a couple of other people in the cloud space. It's separate projects that are all working together. And it's nice because those guys can get together and basically solve the problems that they all have in packaging together. So RDO works on their packages. They have a problem. They can actually go back to their special interest group and they have access to a wide set of packages who are sort of dealing with the same thing. And so once you start grouping people together and getting them into meetings and stuff like that, once we provide those resources for people, we actually have to find a way to distribute that and to control access. And so that's where accounts.centos.org comes in. We just deployed this past year. And right now it's only connected to our CBS setup, but like Fabian said, we are planning on expanding this to other roles in the future. I can show you a little bit of what we have here. So let's go over to this tab. Nope, apologies. Yeah, here we go. So if you've contributed to Fedora before you're familiar with this interface, this is just a list of all the groups that we have. So we've got the atomic sig, the cloud sig, and a couple of other of those. Basically what we're using the accounts.centos.org system to do right now is just to divide up the build permissions in CBS. And we'll sort of go through that here in just a minute. But the process for that, generally what we do is the special interest groups have regular meetings, and that could be every week, every two weeks, it's all dependent on the group itself. But generally if you're interested in one of these spaces, you can find a page on the wiki with all of those meeting times and who to contact and all that stuff. It's best to go out and sort of introduce yourself, say hi, I am somebody working on this project and I'd like to join the sig. And usually what they'll say is they'll bring you to a special interest group meeting and they'll sort of talk you through the process of how it works and then once that happens you can actually get sponsored into the group and get build permissions. So let's talk a little bit about Koji because CBS is just a Koji instance. How many of you familiar with Koji, your packages? Yeah, yeah, you're all familiar with Koji, that's good. So CBS is just a Koji instance that we have that is specifically for stuff coming out of the special interest groups. And we sort of follow a little bit of a pattern because the idea is that we wanna scale this out to a number of sigs and so we kind of want a way to identify things in CBS pretty easily. So if you look at the front page of cbs.centos.org you'll find a lot of things in this scheme where we have the name of the sig, the centos version that they're building for. So five, six, seven, actually I think we only do six or seven at the moment. But then we do a project and a release. Now that may not seem a little bit opaque so maybe an example would help. So think of the storage sig. I'm a developer working on Gluster version 3.6. A lot of things in CBS would be tagged would be identified by this identifier right there. And so if you think of the targets what would that do? You'd have a target that you can send your RPMs into and then you have the release process that you can go through. So it sticks it into a build root, builds the RPM, sticks it into a place, the candidate repo where you can do interesting things with it like send it off to CI and then you can send it through the rest of the release process. And so if you're familiar with Koji that's basically what is going on right here. I just did a quick scratch build into that the storage seven tags just to show you that it works and this is building, so this would be building Python date util on CentOS 7 for the storage sig. And then that is also very, very tiny so I apologize but you can see that we have tasks that go through. This is the cvs.centos.org interface there. And you can see that it finished and you can get all the build logs and stuff. So we actually published these repositories just like any other Koji instance. You can find the repositories for the various tags. So I think this is, yeah, this is a cluster stuff coming out of the storage sig. And you can see that that's the candidate repos. Generally what we do is we recommend sort of a step-by-step process and that's what this is. So you start with the candidate tags, the stuff that you just built in cvs and then you run it through either CI or something like that and then you can promote it to do some developer testing. So you send it out to people that you know and love and love your software and are willing to sort of help you find some bugs. You can actually tag the stuff into the testing tags and you can actually talk to KB to get that stuff promoted on buildlogs.centos.org. That's a place where we publish a whole bunch of repos that and other things that people are going to be hitting quite a bit. So once that's promoted to your developers, you can promote it again to release and that actually goes out to once you're happy with it. It actually goes out and is signed and promoted and delivered through our mirror system. So your special interest group actually has a separate space under our mirrors that you can distribute all of your packages that you decide to release. Some things that we're looking at soon in CBS, we're working on some of the tools. So sent package, if you're familiar with Fed package in Fedora, this would be the analog to that. We're looking to do some things in diskit. So we have, if you go out to the web and type in git.centos.org, right now what that is is that's all of the sources that go into building Sentos 7. We're hoping on down the road to actually deliver some of the special interest group content through there. So you'll check in your spec files and all that stuff and then we can build it in Koji from there. Along with that, the look aside cache, this is where you post up all of your sources, your source tar balls so that we have a copy of it that we can use from CBS. And then we're working on better ways to do inter-sig dependencies. So let's say that the cloud sig wants to use a bunch of stuff from the storage sig. I think that's a pretty, it's pretty fair to say that that's a good dependency relationship there. We don't want the cloud sig to have to build all of those dependencies, all of those things that the storage sig is already doing. We just want them to rely on the stuff that comes out of there. And some of the things that we're gonna have to do in order to make that happen is work on a few things related to testing and stuff like that. So to do just a quick start for how to get started, we provide all the tools that you need to talk to CBS in a package called CentOS Packager. That's shipped in CentOS Extras so no need to enable any other repos or anything like that. You just yum install CentOS Packager. Or if you're on a Fedora system, you can use this coper that we've built and we plan to keep that up to date. To talk about the actual tools themselves, we have user bin CBS. Basically what that is is it's just a Koji client that uses our profile. So we set that up for you, we ship that in CentOS Packager. So you'll always be sure that you're talking to the right instance of Koji. And the other thing is that we ship is a tool called CentOS cert. When you sign up for your account in accounts.centos.org and you apply for a group or you apply for membership in a group, one thing that you'll need to do is we need a way for Koji to identify you as you. And that's what this tool does. It downloads a certificate that you present as a token up to CBS. So once you have your account, all you have to do is just type in CentOS cert and then give it your username and tell it that you want to generate a new certificate, type in your password and you can see that it drops it in your home directory right there. And then you can talk to CBS. So that's what I did right there. I just used my new certificate to contact the Koji hub. So that was sort of a lightning walkthrough of CBS itself. It's, there's quite a bit of activity going in there right now. So if you saw one of those SIGs on the page up there that you're interested in, if you're working in an upstream project that you think would fit somewhere, definitely do reach out to those SIGs and get involved there. Or come find one of us where we're always happy to talk about new groups of people who are working in that space. So next I'd like to talk a little bit about ci.centos.org. Cause this is one of the other really big sort of public infrastructure things that we're sorting to provide to people. And it's something that I've been working on during my day job for the past little bit.  We'll give you a space in CI, you can do whatever tests it is that you like, take the packages that you build, stick them in there, do your, I don't know, unit tests, you can test the repos themselves, test the dependencies, really whatever it is that you need to do. But we're also looking at infrastructure projects that wanna verify on Centos. So if you think about what that means, one of the big projects that we're sort of happy to have in ci.centos.org is RDO. What they do is they take all of the packages that they produce out of CBS, and they actually have an entire long test suite of things that they do. They spin up and open stack cloud and do various things to verify that those packages have worked, but also that the set of packages that they have work well together. And the idea is to do this in an environment that looks more or less like what you would get if you were to hand this off to a sysadmin and say, okay, go deploy RDO. So like I said, they actually spin up an open stack cluster and run all these tests. And you might think that that's a little bit daunting or a little bit slow. But it all comes back to the way that we do things in ci. We have bare metal machines that are provisioned more or less on demand with Centos 5, 6, or 7. We have Jenkins to actually orchestrate the jobs. And finally we have a place that you can actually store off all of the stuff that is generated by your test run. So some people actually do nightly builds or something like that in their ci runs and they want a way to actually store off all of those packages, images, whatever it is. So we need a place for them to store the artifacts and their logs and stuff. Now these bare metal machines, I think right now, the cool thing is it's all behind a set of chassis. We have four of them and it's all API driven. So if we want four network interfaces and two disks or something like that, the fabric itself can support that sort of thing. In general, we provision with a single disk on there and then we have the network interfaces that are up and available for people to configure. And that's all done by a tool called Duffy. So each project is assigned its own API key and that API key is used to request those nodes in whichever configurations we support. So if you want CentOS 5, 32 bit, you just ask Duffy for it and it'll give you a machine. Duffy contextualizes that machine and basically takes your SSH key and injects it into the root account.  And obviously you do other things afterwards.  Then you actually return that node back to us and we'll reinstall it and give it away to someone else freshly installed. So this graphic sort of shows a little bit of about the architecture. If you're coming in from the internet, you can, everything is done on the project side. So if you're a project member or something like that, you can get through our jump host and onto a jink and slave. Now this part is, if you've done continuous integration before, this part is a little bit, I don't know, kind of freaky and a little bit weird to some people because it's not immediately obvious what's happening here. So you actually landed on the slave, that's slave01.ci.centos.org. You have a workspace there but that's not actually where your tests are running. We actually want you to work in a pattern where the slave, so your workspace that you land in, that's the place where you just request those nodes. You request that bare metal hardware and then remotely from your workspace there, talk to those machines and do your tests. And that can happen over SSH or whatever it is that you like. But Duffy itself is just an HTTP API. And I'll give you some links to a couple of clients and some scripts that you can use to actually make things a little bit easier on yourself when you're working with jobs. But again, we have the Jenkins master, that actually talks to your workspace there to trigger out those jobs and deploy those things.  And then we start the process all over again. We just set up that machine over and over and over and over. And I think Fabian mentioned in his talk last time that we're doing about a hundred deploys a day, something like that. And we're looking to get a little bit faster with that even. So there are a couple of links to some Duffy clients. If you want a simple one that you can just copy and paste into a Jenkins job, look at that top link right there. That's a pretty good quick start to get you going and started with that. And then we actually had some people in the RDO community who are using our infrastructure quite extensively and they wanted a little bit better interface for them to hook into their tools. And so they wrote a Python client library that you can use. And I do know that a couple of other projects are looking into that. But so that provides not only a Python client library but also a command line tool that you can request and replace machines in Duffy. So what's coming up next? One of the features that we're working on in ci.centos.org is to take those machines and actually group them together so that you have the same network across all the four, five, six machines that you have. Right now they all have interfaces on them into our CI network so they can get to the mirrors, they can get to the internet, all of that stuff. But some projects have asked for a separate network that's maybe a little bit isolated so they wanna test out a storage configuration. They have a separate storage network or they have a separate virtual machine network for their OpenStack deploy. That's one thing that we're looking to do to sort of tie all those things down. We're working closely with Jenkins, some people using Jenkins Job Builder because once you have a whole bunch of things to test it gets kind of tedious to go through the Jenkins web interface and create, you know, I wanna test this thing with this thing and this thing with this thing. It gets really, really, really tedious to go through and create all those jobs by hand so Jenkins Job Builder can automate a whole lot of that. We're hoping to spin up an OpenStack instance soon so we have the bare metal that is run on CentOS. You know, you get those machines there but OpenStack would allow people to, number one, consume that service and test against the cloud paradigm there but then also for them to bring their own images. So we have a couple of groups that are looking to verify on CentOS but maybe do a lot of other unit testing on some other operating systems. We wanna make it easy for them to bring along their own images. I don't particularly want to be running a whole bunch of Ubuntu stuff but they're welcome to bring their images that they have and verify those unit tests too because that provides some value to the project. Let me show off Jenkins just a little bit. If I can get over here. Here we go. Yeah, so if you go to ci.centos.org you'll actually see over on the left here we provision the workspace for every project and then you can see the whole set of jobs that we have. So this, these here are some of the core jobs, the people who put out CentOS Linux actually run through functional tests every night and then also on updates. So if there's ever an update that goes through we do some pre-update checks that all run on ci.centos.org and you can see them up there. So let's look at a quick job run here. So I mentioned that I was working on a tool called CentPackage and I want to run my unit tests. That's something I want to verify on CentOS and of course live demos. Here we go. What Jenkins does, when I actually go out and call to Duffy, get a machine and SSH into there I can, all of these logs are actually piped back into the Jenkins interface. So everything that I do is here and you can see that if we were to go back up you could see that this test passed but everything that I did during my whole entire test run is saved off in this interface and consumable by you guys. So if you're interested in CentPackage or you're interested in RDO or you're interested in CentOS 7 itself you can go back and see what tests were run, what tests were, if it's a failed run, what caused the problem, anything like that. And if you see something we're always happy to be notified about that but we get emails and pages and all that stuff. So we're sort of in the beginning process of standing up this service and working with a bunch of those infrastructure projects. So if you are one and you want to sort of partner with us on that verification, do let us know because one thing that we're working with people is one thing we've noticed is that everyone sort of has a different test setup and we're happy to help sort of accommodate that and add features as projects need. So yeah, definitely do come talk to us. Here are some links that talk a little bit about the community build system, about our special interest group process and about CI and a little bit of contact information. Usually we hang out on CentOS Devel for a mailing list or in FreeNode and there's my email address and my Twitter if you're interested in getting involved in this process. So that is all I have for you guys today. Do you have any questions for me? Yep. No, so we're doing a copper because the velocity right now is gonna be pretty quick. The plan is to get it in Fedora proper and get it reviewed once we sort of stabilize a little bit. It's, I would probably call that sort of in beta status right now in terms of the client tools, so. Yeah, I definitely will. Yep, yeah, the disk it. Yeah, so the, yeah, sure. The question is, is there going to be a policy on the disk it branches to sort of help facilitate people working on the same package? And I think that part might be a little bit up in the air. One of the things that we need to work on first is getting the integration done with the account system and the actual policies themselves might come out soon after that. But that's, I think that's something that we should maybe talk about. Okay, yeah, so the question is, is there any sort of bit for bit reproducible builds effort going on? And I don't know of one, basically the, so the distribution itself isn't actually built in CBS. That's still the same process that it's always been. We take the sources and build them. But yeah, I don't know of any sort of effort in the project that's working on that. Other questions? Yeah. Yeah, so let's actually take a look at that. And we'll hope for good conference Wi-Fi. So let's actually take a look at, yeah, we've got plenty of time. So under git.centos.org, you'll actually find this project that says sources for RPMs shipped. And if you click on that, and this is gonna take a little bit because we have a ton of RPMs in there. But this is, so this is the everything that goes into Centos 7. So if you see something that comes out through an update or something like that, that's all gonna be in here. And of course, we're waiting for, yeah, here we go. So let's look at bind. Oops. Sorry, I'm trying to read this on the, here we go. Yep, so if you come to the C7 branch, that's gonna be all of the sources that we have. And if you look, basically what we've done is taken the source RPMs and sort of exploded it into this directory. So you have the spec file under specs, and then this metadata file in there that actually pulls from our look aside that we have set up separately for the sources coming into the distribution. Does that answer your question? Okay, other questions? Yeah. Yes, they do. And that's all set up through, so initially what happens is the SIG is sort of onboarded into the build process. And we have people go through a few builds and publish those to the build logs section. That stuff on build logs is unsigned. So we sort of have people point their developers at it, people who are early adopters, but that's not really for production use. Once the SIG is ready to actually start doing the release process, the full-fledged release process, they work with one of the people in the core SIG to basically say, okay, these are the packages that we're gonna promote to mirror.centos.org, and those get signed. Yeah. Yeah, so the question is the diskit layout that we're showing here is different from the one in Fedora, and the question is, are we going to do that for the special interest groups? And really the answer to that is no. This is the layout that we're working with, and the idea is once we have people actually pushing up to these repositories, they're actually going to have a separate branch for the special interest group. So if someone wanted to come and rebuild bind in a SIG, they would have a specific branch for that, and so we need to sort of maintain this layout for that. Other questions? All right, thank you very much. I have stickers up here if you want community build service stickers. Otherwise find me around the conference. Thanks guys. Yeah, so you can come in for that. Okay. Right here, I have two. Redactions. Okay, I think so, HDMI, okay, maybe. It's the reservation for, I'm making a chart. All the way up here. Yeah, I've got a meeting. I don't know which one. I don't see it. I don't know the signal there, so I mean. So I have a second resolution. I have a second resolution. But I want to say it. Oh, wait. I have a great, great, great resolution. And I just want to give you an answer. So just in case this happens again, so. 800 by 60. 800 by 60 is not a report. There were probably another session chat introducing it, whereas in Iraqi, the first one was for you. that you have.


You're watching the Daily Decrypt. Welcome to currency competition, bitches. I am your host, Amanda, and today's episode is brought to you by BitShares. Today I've spoken with Vlad Zamfir, who is the designer of software called Casper, which is intended to take the Ethereum network from one that is mined in proof of work to one that is mined via proof of stake. Many of you chimed in via Reddit with the questions you wanted me to ask Vlad and your wish is my command. Tell us a bit about your background and how you came to be a developer at Ethereum. Well, firstly, I'm not really a developer. I'm more of a researcher. Okay. And I kind of like more like design software than actually implement it. Okay. My background is in mathematical statistics. Well, most of my background, I guess, now after having been in the blockchain space, working progress full-time for two years, I guess I have more background there. Before the blockchain space, I was doing inference and mathematical statistics. Okay. With a focus on small sample sizes and avoiding overfit. Okay. And how did you come to be connected to Ethereum's core developers? Well, so I met Vitalik in Toronto back in March of 2014. And then April, I mean, Ethan, who is now working at Aeroson, moving to Tenor Mint, won a hackathon where, by making their EtherDot, like basically it was called CryptoShorts. And from then on, I came much more involved with the community, you know, the founding and broader community. And so for, I know that even before the Ethereum network launched in its initial stages, they had wanted to switch from proof of work to proof of stake. And so were you approached for, to create a solution for that? Or did you approach the core team with your idea of a solution? So, I mean, at first when I said we'll start working for Ethereum, I was doing like reputation systems and kind of more infrastructure stuff, like a layer above the consensus protocol. And now we're spending a bit of time working on blockchain scaling. But at this point, I didn't really believe in proof of stake at all. I didn't think it worked. I thought I had like fundamental problems that couldn't really be solved. And then in September 2014, I kind of realized that proof of stake actually can be fundamentally secure. Mind sharing your thought process with us. What were your initial doubts?  Sure. So initially I had basically like the classic problems, right? The long-range TAC problem, the costless simulation or stake grinding problems or the nothing at stake problem. Just kind of like the, you know, the fundamental problems of proof of stake that like people in the proof of work community kind of still generally believe in. And then what I kind of realized was that by using security deposits, we could kind of ensure that a signature is meaningful for some duration of time. And additionally, we could rely on it only during that time when the deposit is placed behind that key. And so we can basically, I kind of realized that, you know, it would be possible to do the economics using kind of only signing and security deposits. And I'm already quite satisfied that the reason that Bitcoin works is because Bitcoin has a price. So Bitcoin has a price, which makes the hash rate be pretty high, which means that it's secure, which means that it can continue having a price. So the kind of self-referential, self-bootstrapping thing where you kind of need to assume that you start off with consensus and with these kind of these units having a price in order to prove that the consensus can be maintained. That's something I was already comfortable with because in my view, proof of work does that. And so I was just able to convince myself that with security deposits and signatures, basically I could accomplish a lot with proof of stake. The reason that you can actually do a lot more with proof of stake than proof of work is because you have direct control on the anti-civil mechanism. It's like in the protocol and you can like, you know, take it over. Will you briefly tell us what a civil attack is so that we can visualize what an anti-civil mechanism would look like? Sure. So a civil attack is basically when an adversary comes onto a public network and introduces a lot of nodes, you know, perhaps behaving correctly until it has a majority or a very large super majority of the nodes and then it would switch to bad behavior in order to kind of sabotage the network. So essentially the civil attack problem was solved by Bitcoin by using this scarce resource, which is like difficult for civil to get because it's scarce. Civil can't just like multiply it by running up new nodes. And that resource was the hashing power, right? And in, but in general, the idea of using economics as a civil resistant force makes a lot of sense because if something is scarce, it can't be easily copied by a civil attacker. And civil attacker relies on the fact that it's like basically cheap to spin up lots of nodes. You know, the fact that we have the asset which acts as an anti-civil mechanism in the protocol means that we can do things like specify, you know, policies about when which players lose how much money for, you know, different outcomes. Okay. So this takes us directly into Casper. So I would just like to start with a fresh question, which you were already going into, which is give us a basic overview of Casper in I've had specific requests to keep it in E-L-I-5 explained like I'm five terms. So in an explained like I'm five way, not that any of this could actually be explained to a five year old, but do your best with just a general overview of Casper and what it's intended to do. First let's start off by talking about like the intention. So the thing that Casper is trying to do is essentially act as like, essentially a consumer protection policy for Ethereum for the users and developers against and like defending them and providing them assurances that the people who are maintaining consensus can't undermine their guarantees. And when they do, that it's expensive for them.  The thing that I do additionally on top of that analysis in Casper is to ask, you know, say they do have enough resources to conduct an attack, then how much money would they be losing or profiting from conducting that attack? So in Bitcoin, for example, you know, if two thirds of the miners censored one third of the miners, each one of those miners that are in the censoring kind of coalition has its profit go up by 50%, well, I mean, because revenues go up by 50% and they don't need to necessarily hash any harder because the difficulty would just adjust to exclude the third. So in Bitcoin, you know, if an attack is successful, it's highly profitable, right? If you double spend, if you censor, you can make a lot of money from doing that. One of the main design criteria in Casper is that it should be expensive for an adversary to conduct even a successful attack. So the main thing that we do is basically have adversaries, have the validators pledge their deposits on the fact that- And a validator is the equivalent of a node or is a node, is that correct? So it's actually, it's like the equivalent of a miner. So it's not just, it's really like a node that can validate transactions and which has a security deposit in order to, which gives it the right to basically like produce blocks. Okay. So when you use the term validator, you're talking about a, in proof of stake, a miner and a node are one and the same. So we're talking about somebody who's running a node is acting as a miner and has a security deposit in place. That is a validator. So really what we usually call them bonded validators. Bonded validators. It's a little bit more clear that, you know, maybe you can validate without being bonded. Got it. But, you know, bonded validators, there's a lot of syllables in it. So we often just say validators. Okay. And basically the story is that like, to produce a block in Casper, you have to place a security deposit and then you have to engage in this process of essentially placing bets on whether blocks will be included in the future histories of the chain.  Right. So you'll lose money because you were previously to build the canonical version of history.  And then when you produce your other fork in order to say revert history, you would be betting that some other thing would be the canonical versions of events, but both can't be true. And so you're gonna lose money regardless. Okay. And I have some more detailed questions about what you've just mentioned. Before we go further, tell me so like the, say the moment that Casper is deployed into the Ethereum protocol, the actual switch from proof of work mining to proof of stake mining. Do you, is this going to be like instantaneous? Like, is there going to be a moment in time at which GPUs and CPUs mining Ethereum simply stop or what does that switch look like? Basically what's gonna happen is, you know, every client, every Ethereum client will download this upgrade, which will say that at some feature height, the like consensus slash authentication mechanism changes from this proof of work to the proof of stake version.  Okay. And, you know, do we definitely need to make, to put a bunch of thought into how to do it as soon as possible. But, you know, we generally understand how to do these kinds of hard forks because it's basically a coordination problem that we can solve provided that people will install the software and that we have like, and that we have a consensus protocol to begin with. Okay. The only concern is that the proof of work chain might fork and so if you might disagree about which block is the last block, but basically what will happen is, you know, eventually one of them will win and then everyone will agree on what the last block was. And then the proof of stake validators would finalize it. Okay. Well, if you don't mind, I have a few community submitted questions here that I think will act as a nice roadmap for us going forward. So let me go ahead and pull those up. So let's talk about how many Ether will be, I'll read off the names because I think it will be fun, but Diebendorf wants to know how many Ether will be required to stake, how many as the security deposit? Right. So there will be a minimum. Now, what that minimum is precisely, I can't say at the moment, most likely it'll be dynamically adjusted over time. Basically, the minimum has to be, we need to have a minimum because if security, if people get in without having many deposits, then the cost of screwing up the system will be low, but if the minimum is too high, then we won't have enough people who are able to bond. And so really what we're gonna do is we're gonna like target some amount of extra bonding attempts. So there might be like some number of validators that we wanna have, and then we'll have like that plus like 30% to be the amount that we target using the minimum bond amount. And so if people stop bonding, we might lower that amount. And, but there will almost certainly will be a minimum. And is there currently a number, a minimum of minimum validators you'd like to have? No, I mean, it's best not to have numbers to argue about until it becomes like the last argument to have, right? After we have all the other protocol details, we're gonna start thinking about the parameters. Different people have different kind of back of the envelope feelings, but no one thinks that we should have less than a hundred validators. Okay. And Deven Dorff also wants to know, once this number is reached as you, which you say you'd like to come more toward the end, if there are a greater number of people wanting to be bonded than your desire for Casper, how will the selection process take place to choose from among them? Great, yeah. I mean, basically we're gonna have like an entropy source in the protocol, like a crypto economic entropy source, and we'll just do like sampling in proportion to, with your probability of being picked as proportion to the size of your deposit. And this is the plan at the moment, right? We're gonna end up, we're gonna target always having more than we need so that we can make sure that we have some buffer room, right? Because ideally we'd have like the same number of validators always. So people don't get scared when the number goes too high or too low or something like that, right? And ultimately this is gonna be probably based on overhead calculations. And the precise overhead figures aren't exactly known because we're exploring different data structures and algorithms for, for example, the bedding rules. And the bedding is mostly overhead really. And do tell me what the bedding is. I'm not sure what you mean when you say bedding. Yeah, so bedding basically is a process by which validators expose themselves to loss if something doesn't turn out to be the case. So the way that we come to consensus is by everyone synchronizing their, the conditions under which they would lose their deposits. So everyone is basically going to imagine, okay, when you mine a block in Bitcoin, you're placing a bet that the previous block will be included in the chain. Okay. And the more blocks are in the chain, the more of the network's effort has gone behind that one block that all these blocks are on top. In security deposit based for stake instead of creating blocks which require this computational expense in order to produce, we place our deposits against blocks in order to kind of give them, and give clients an economic assurance that those blocks won't be reverted, for example. And the bedding process is the process by which you, the validators expose their themselves to this kind of disincentive.  And what would NBR one bonehead wants to know? What is at risk when staking? Is it all of one security deposit, a portion of one security deposit or just the fees? And what would one have to do in order to lose? What they have at stake? Sure, so I mean, all of it is at stake but different types of behaviors can put different amounts of it at risk. So if you just never do anything, if you just place it on it and you never validate, then you will lose some but not all of your deposit.  If you engage in bedding, so you're actually online and you're engaging in a bedding strategy, if you follow a bad bedding strategy or if for some other reason the network can't converge or you aren't converging with the network, then you do stand to lose some deposit but it will be less than if you were offline the whole time. And now when you said if one is bedding but not validating one stands to lose one's security deposit, could one be not validating in the simple case of one's hardware accidentally goes offline? Would that count as not validating? No, what I mean is like specifically imagine it, like when I went forward to place a bet on an invalid block, that would be a sign that I haven't been validating and then that basically I've been bedding without validating, but if I'm offline, I won't, I'm not gonna get that, like I'm not gonna lose my whole deposit because I didn't actually sign an invalid block. But what might happen is there might be a problem with the hardware and you might fail to produce a valid block, it might, something like go wrong. And basically the bigger your security deposit, the more you need to put checks to make sure that your blocks are valid, your bets are good, that your note is online, et cetera. Okay. I know. If you have deposits, so like, you know, probably just running a note on your laptop will be fine. For large deposits, you'll probably want something a little bit more sophisticated. In the end, you might want something like, you know, a little consensus protocol between like for validating servers and then like another node that kind of filters everything to make sure that only valid blocks get through or something like that. And now, is all of this intended to also bring about a faster block time? Is that true?  So basically there's like this relationship in Casper and I think in general in availability, favoring consensus protocols that the higher overhead you're willing to tolerate, the lower latency you can have. Okay, and is there a number that's been estimated of what the overhead might be? I mean, the overhead is basically gonna be like linear in the number of blocks and quadratic in the number of validators. Although, you know, we have lots of like ways to get constant factor speedups. And basically this is the overhead of the whole network, not for any one validator, right? For any one validator, it would be, you know, you'd have linear in the number of validators overhead and linear in the number of blocks. Okay, and this leads me to some questions from Speedmax. He wants to know, do validators earn a block reward or do they earn transaction fees or both? Right, so this is something that's still kind of in discussion. And my personal preference is that they would earn only transaction fees. Vitalik and I have talked about mechanisms for doing enough issuance to have the amount of bonding that we would like to have. So there is a possibility that we will issue a variable amount. We haven't really talked that much about issuing a fixed block reward. And basically the main reason for that is that the economics are actually better when the clients are the only source of revenue for the validators because then validators can't profit by doing something that would cause the clients to stop making transactions. So for example, if validators like lied about timestamps, they could potentially get block rewards faster. I mean, this is true for work and for stake. And then clients might see these off timestamps and be like, okay, I'm not transacting, but the validator won't care because most of the revenue is the block reward anyways. And so basically clients will have to kind of reluctantly say, okay, I guess I'll transact anyways because there's nothing I can do with the validators, don't care about my transaction fees. Whereas if the revenue is entirely by transaction fees, client behavior will greatly influence the profitability of validators. And so we can use client behavior to do things like, for example, make sure that timestamps are correct. All right, so would it be safe to say that as of now, there is no decision on whether there will ever be a production cap on the number of ether in existence or that there is a set rate of inflation be that steady or declining?  The most of the design criteria for the consensus protocol are independent of these kind of like considerations of monetary policy and distribution of tokens. So, and most of the reason that people care about the issuance rate is because of, is for monetary policy kind of reasons, right? They wanna know that the currency has certain properties that they like in a currency. The thing that I'm worried much more about is the security of the market for basically transaction receipts. Okay, and crypto anarchy wants to know if Ethereum via Casper is planning to ever deploy a sort of decentralized governance process via its validators the way Dash does via its masternodes? So, if there is a decentralized governance platform it certainly will not be run by the validators. The validators are really like the biggest source of risk to the protocol to all of the guarantees we wanna provide. I mean, what we wanna do is provide a platform for people to deploy applications where they know that any transactions to their applications will be correct and where users know that they'll always have access to their application and that any transactions they make will be correct and that their transactions won't be reverted arbitrarily and all sorts of anything bad that can happen it would be because it's the validators fault, right? Because the validators are colluding to screw the protocol. And so I certainly don't think any governance any governance would be centered around validators. I mean, what we're trying to do is provide a platform for users and developers. I think most of the community would agree rather than a platform that is profitable for the people who run it. Okay. And I think actually this is one area where we're trying to learn a lot from Bitcoin where Bitcoin has a lot of basically political factoring along lines of personal business interests and mining is a very big part of the Bitcoin community because of just the sheer quantity of revenue that Bitcoiners are paying to the miners. I mean, 25 Bitcoins every 10 minutes adds up to a lot of money. And as a result, it's actually very, very difficult for the Bitcoin protocol to change in any way that obsolete mining hardware. Whereas like the kind of road from here to the scalable, general purpose, super easy to dev and use platform that we want, is gonna have lots of changes along the way. And we certainly don't wanna be stopped by the basically business interests of people who from a protocol point of view and from the user's point of view will represent the biggest risk. Okay. And that leads directly into Cat's Fives question which is in your view what is the greatest benefit of Proof of Stake and what is its greatest con or risk factor? Sure. So the best, so there's kind of the one best benefit. Okay. Or you can give a few of them. I'll give my top three hopefully in order of three, two, one. So low latency, we can do like lower latency. Although I guess arguably you could do that with Proof of Work too, but you end up losing a lot more work. So that's not necessarily the best one. Okay, let's restart one second. Transaction finality, okay. Transaction finality is like one of the biggest ones where you can actually have your transaction not be reverted, revertible. You can't have that in a Proof of Work kind of authentication model. And the other one is that we can make censorship actually expensive because we can see when people are being censored because we have access to the anti-civil mechanisms that kind of governs who can produce blocks. If someone's being censored, we can notice that and so we can make it expensive. So the fact that we have like economic guarantees of censorship resistance is something that is very much something that exists only for a stake land as far as any number of faults is greater than like 50%. And then there's a risk side. Sorry? Oh, I'm sorry, please finish. The other thing that people really like and that I really like is the economic efficiency of it. So what we wanna do is move from a model where consensus is expensive for everyone, more expensive at all times than an attacker might spend at any time.  Whereas in an improved stake model, what we'd wanna do in a security policy based for a stake model is move to a situation where it's cheap for everyone except for an adversary during an attack. So people are paying only a little bit to run the consensus. But then when someone does something really bad, they pay a lot. And it's a much more efficient model because instead of having to spend all the time more than an adversary might at any time, we can spend like a little bit and then the adversary could spend a lot when they wanna attack. So the kind of, it's much more secure, right? I mean, I worry a lot actually about double spans censorship, a lot of problems that existed before. I think that there are like super real problems that will happen. And with this kind of protocol, we can do a lot better on a lot of fronts, especially with transaction reversion. I mean, finality is something that you just don't really get improved work, but you can have improved stake. And actually you need to have further improved stake protocol. Okay. And then what about the risks or possible complications in proof of stake? What do you envision there? Yeah, so the main, difficulty with proof of stake is the conceptual end of implementation difficulty because this kind of like modern proof of stake protocol tries to get like very large coverage of Byzantine faults with disincentives. There's a lot of cases, there's a lot of edge cases and there's a lot of kind of reactive stuff that needs to happen. So if someone produces an invalid block, then someone who sees it needs to either forward it to someone else or produces a type of transaction called an evidence transaction, which would lead to the forfeiture of their deposits. And I guess what we could talk about is like the failure modes. Like what could go really wrong? There are basically, I guess there's two main failure modes to kind of worry about. One of them is consensus failure due to finality. So whenever you have finality means that like clients will refuse to accept the blockchain that doesn't include some block, some hash. Then, so if you have like a network partition and a super majority of Byzantine faults on both sides, you can have two different clients on different sides of a partition, each finalize a different block and then they'll never be able to come to consensus once the connection reemerges. But that does require a very large number, a finality threshold number of nodes to be finalizing the block on either side. And the finality thresholds, something like a very least 67%. And so you end up having a lot of security deposit loss from that, but you will have consensus failure. Another kind of concerning thing might be if some validators do manage the sensor, so you get a sufficiently large proportion of the pool that you could actually censor transactions. Then you can censor evidence transactions, you can censor bond requests from other validators and you could basically have like a monopoly on the validator set, which you could then use to do something like maybe jack up the fees and undermine user guarantees in various ways. And for both of these kind of failure modes, the last recourse would be a hard fork, right? I mean, if there's a narrow partition and finalized block on both sides, then one side is gonna have to hard fork to join the other. If there is censorship going on, then despite all of our best efforts to stop it, I mean, because we do do a lot in protocol to stop censorship, then we could potentially like hard fork and remove these validators from the validator set. There are, and some people find that rather unsatisfying, but for me, the important thing, the important question to ask about security of these protocols is what can an attacker do? How much does it cost the attacker? How much does it cost the community and what can the community do to recover? And having to do a hard fork while inconvenient and potentially politically difficult, that I think is much better than a very long history reversion. And the nice thing about it is that it happens at very little like economic costs to the community, whereas there's a lot of economic costs to the adversary. Okay, so I have just a few questions left then. pseudonymous Chomsky wants like a brief explanation of what the user experience will be for a validator. So basically what hardware do I run? I think you had mentioned that it would be possible to do this from a laptop and basically what interactions are required from me via software or not. Right, so it does vary for us large and small validators, but eventually what you need to do is have a correct validator online all the time if you want to be as profitable as possible.  And so you need to have like a correct validator online as much as possible. Now, what it takes to make a correct validator basically in practice, it'll probably be okay to just run in like Casper node on like one computer. You know, just probably won't even need to be that strong at first, but if you're staying with a lot of money, what you're gonna wanna do is have a whole lot of precautions to be really sure, you know, because on top of worrying about your thing being offline, you also need to worry about being hacked. You need to worry about it producing invalid blocks or misbehaving in some way. Like the cost of having a malfunctioning validator could be quite a lot higher than potentially the cost of having a malfunctioning ASIC. I guess, you know, in a case where an ASIC malfunction that overheats and burns and becomes unusable, is that's basically the worst case for Casper as well. Okay. And B Chain wants to know when a Casper white paper will be published? Yeah, so that's definitely a great question. I mean, the reason that it hasn't been published yet is basically because we have different design ideas for different parts and we're kind of comparing them. And, you know, so Vitalik right now is working as proof of concepts for like basically the simplest version of Casper that he can imagine. I was working on a white paper for basically a version of Casper that has a lot more kind of a lot more guarantees, but a lot more protocol complexity. And I think what basically what's happening is we're finding common ground and ways to move from like the simple with less guarantees to more complex, more guarantees.  I mean, so far every time I've tried and I've done it a few times, things end up changing and then I end up having to go back and it becomes dated, you know, within a month or two.  Yeah, but certainly we're gonna have everything be very well documented. We're not gonna ask the community to accept the hard fork without proper peer review, you know, work, right? So we're gonna have like a white paper for like the general public, but we're also gonna have like an academic quality like paper that's been peer reviewed by like, you know, experts in the field basically. Okay. And Tony McCarp wants to know approximately when Casper will be deployed? No, that's another great question. Essentially, there's as much stuff that needs to happen first. Basically the protocol needs to like stabilize. We need to kind of produce all the documentation and we need to like implement the protocol, you know, and probably multiple implementations will need to implement the protocol. And the other thing that I've been really working a lot on is that working, I've been working towards doing formal verification of the protocol, basically kind of getting proofs that various parts are correct and have the properties that I claim they have. And this is like a lot of the work, which, you know, is like, it's a way to verify things that is kind of in some way more exhaustive of edge cases than simulation. Like some people find like simulations sufficiently convincing, but, you know, I'm really trying to get formal verification of the protocol and I think it would be prudent if we waited for formal verification, although probably we're gonna have is like formal verification of some of the key components and then the coverage of the formal verification will grow over time, you know, and then during that process we'll probably be able to upgrade the protocol a bunch of times. Okay, so no estimation of time of deployment yet? Is that how- As an estimation, you know, like a year is like kind of, if it's not out in a year, I'll be, I will have not worked as hard as I should have possibly or maybe there's like a lot more stuff to do than I thought, but I certainly think that, you know, a year from now we will, I'm pretty sure that a year from now we'll have it. Okay, and a final question comes from wobbly bit five. I believe that's how it's pronounced, which is have you, first of all, are you familiar with Daniel Larimer's critique of proof of stake that is not delegated and what would your response to his critique be? Which I believe can be summed up as without delegation it can only lead to centralization. Oh, yeah. So I actually think that the important thing when you're thinking about any of these markets is basically the, you have to look at the like the dynamics of the market and ask whether, you know, people who are not in the majority will be represented. So it kind of for me, decentralization is kind of an economic concept where, you know, instead of having like oligopolistic competition kind of breaks down and ends up being more of like a competitive market where anyone can enter and leave and anyone, anyone's say kind of counts. So let me just like take a step back. So the reason that I would say that Casper is decentralized is because if you say have like 10% or 5% of the stake then like your bonded stake matters. There's nothing that the 95% could do to remove you without really screwing themselves. And like they're ever, they have every incentive to allow you in the protocol.  So the kind of economics end up determining actually whether someone with a lot of money gets to run the show. In certain protocols, as soon as you have a majority no one else's word matters. And in that way, I think is like extremely centralized. Whereas like, you know, on an economic front, basically with like, you know, thinking of Casper as like a consumer protection policy is like an 80 trust policy. You know, we do our, I do my best and we do our best to make sure that everyone is going to be represented. Whereas, you know, in my view, you know, looking only at the fact that it requires resources to earn resources would basically let you say that any economic consensus protocol, you know, favors the rich because they'll just make more make more resources. And I think that's like true on some level. But I also don't think we have a choice but to use economics in this consensus protocol. And I think that like very notably delegated proof of stake has a lot of economic problems. You know, you can buy votes. You know, the highest bidder just gets all the votes and then it's like super centralized. So under like an economic analysis, you know, delegated proof of stake falls apart rather quickly. And, you know, the wish that the stakeholders will give the delegate positions to people who, you know, somehow would expand their community or make it more decentralized, you know, isn't really guaranteed by the protocol at all. So really what we're looking for is in protocol guarantees of decentralization that withstand economic analysis. All right. Does that make sense? I don't know. It jives very well with the theme of this show which is currency competition. That is what we're all about. That's what gets us up in the morning. And that is all of the questions that I have for you, Vlad. So I want to say thanks for your time and thanks for coming on the show. Sure, my pleasure. Yeah. Are there any links you want to leave anybody with or anything for further research or how to follow you? Not really. I guess I should mention that, you know, we have CASPER research hangouts every week and we're starting to make them more and more open to the public and, you know, I think we'll have like a live one every week now. So anyone can drop in and just listen in on the research discussion on various parts of the protocol. Okay. And how would they do that? Is there a URL or should people just follow you on Twitter or? I'm not sure. Right now I don't think we have like a page for that. It's just gonna be like a different Google Hangout every Monday. And so, yeah, maybe I'll get into the habit of tweeting it. That's like, that's not a bad idea. Yeah, follow me on Twitter for news. Another thing that I should maybe mention is that I've been working on a lot on blockchain scaling and that's been pretty exciting. And so maybe you should expect something from me in that regard sometime soon. All right, right on. All right, well, I guess maybe we'll talk some day in the future via an update. And so until then, thanks for chatting. Nice to meet you. You too. Bye-bye. Today's episode is brought to you by BitShares, a currency which makes multi-signature transactions easy for everybody and offers a fiat on-ramp to its ecosystem via the Exchange CCEDK. The next version of BitShares this quarter will also offer confidential transactions to increase the privacy offerings for all BitShares users. You can find out more about the currency which runs on human-readable usernames versus alphanumeric addresses at bitShares.org. Well, the man-servant and I are headed to the Free State Project's Liberty Forum this weekend so you can expect event coverage when we return next week. Have a good one. I'm running the tide.