Welcome to the talk on the hand of for HCD data inconsistencies. Data inconsistencies are pretty is unique animal. So finding them is more than an art than strict science, at least yet. So hopefully after the talk you will have a better understanding of the topic and be able to find inconsistencies in your own system. I'm Mark Cherkovich, I work at Google and I'm one of the HCD maintainers. So topic for today is I would like to define what are data inconsistencies, what tools we use to hand them and how HCD has adapted to find the problems that we had with inconsistencies. So at the end I will do a short demo showing you how it works in practice. So HCD implements so-called distributed consensus. It means that multiple units, processes work as a, or multiple processes of HCD server run as a single unit, a cluster, that can consistently respond to user requests. So any user can observe the same data, any write by a user can be observed by all the users. So what is the inconsistency? Inconsistency is when one of the instances breaks loose and starts spilling nonsense. No matter how cool your free head dragon is, it doesn't, it's stopped working when one of the heads goes berserk. To give you more concrete example, here is a real production case that I noticed and tried to document where HCD inconsistency caused problems with HCD cluster, Kubernetes cluster using it. In this case, HCD or Kubernetes nodes were flapping from status ready and not ready. There were random failures, random timeouts. Authorization didn't work, sometimes it worked. So how do you know what is happening? Add-ons were crashing. Yeah, basically because of architecture, one of the API servers was totally misbehaving. And what was the root cause? It was just missing one, exactly one write. So as you see on the graph, missing one write can cause the revision of HCD to totally diverge. This is because how Kubernetes uses HCD and its revision is pretty crucial for Kubernetes correctness. To explain, revision is like global counter for each change that is happening in the cluster and revision is used by Kubernetes for optimistic concurrency control. So the example above was unrelated, but why I'm talking to you today is about the state of HCD 3.5.0. The release was done after a long time with a lot of changes in the project, including total change of maintainers and loss of a lot of knowledge that was unwritten between maintainers and not past. Addie, the work that motivated HCD to look into the problem further was two, let's say inconsistencies or correctness issues. One was data inconsistency on crash. So your instance of HCD server could be happily running and then it ooms or your VM runs of memory or it ooms or is killed. HCD could make incorrect write that results in this one instance being inconsistent with other members. In second case, apparently no one noticed that HCD for a long time didn't provide durability in some cases. Yeah, and we got and are still mostly not have not full explanation for multiple reports that are currently very hard to reproduce and understand. We are trying to build tools to understand them, but at the current state we have multiple reports that are unconfirmed, unverified, but also not fully understood. The problem underlying this is HCD doesn't have a test suit that is capable to detect these class of issues. I expect most of the projects and databases don't have these kinds of tests. But compared to most databases, they don't build for full ecosystem. So no matter what anyone says, HCD is the building block for Kubernetes, which is the building block for full cloud native infrastructure. Like we cannot deny it. We need to do better, like as a project, like we cannot let this slide. So how do we hunt for inconsistencies? What is the previous art? So previously, HCD was using functional tests. So functional tests that are pretty simple. They just run HCD instance. They do some failure injection and maybe when they finish verify if the state is consistent between the members. There were multiple problems with them. Like one main thing, it was they were written by one person and when this person left the project, no one knew how to use it, no one understood them. They didn't work, like they run. Like most of the time they flaked also a lot, but it was not solving the issue directly as, it didn't solve like the issue end to end. It just give us a signal that was very flaking. That if you go into state of the art, you can find Jepsen. So Kyle Kingsbury is person that is really interested in correctness of databases. So he built whole I would say project and community that validates safety of distributed databases and verify like checks if there are guarantees that they are saying that they're giving checks is those guarantees truth or is the database lying or selling there or is it selling more than the relative is? And there were multiple times used successfully to validate HCD project. But this solution is also not perfect for an open source project that is written in Go has limited capacity and doesn't want to learn new language be forced to work on AWS or learn a whole new domain, the domain like from distributed systems which are known to be pretty hard. They were also never designed to run in CI. So if they are, if at CD changes anything in API and it breaks the tests, it cannot be easily verified. Someone needs to run them manually on every time we make a change. So they cannot be used for continuous validation. And at the end, like maybe personal opinion, but we couldn't reach out to even ask CNCF to pay the owner of the system to help us. So what the tools that we need need what are the requirements? So what we need to build is something that is adequate. By this, I mean, not only reproduces all historical issues that we know of and in every, like reproduces them easily and every time and we can use that to always validate correctness of the tests not only, but also is able to go through all generic properties of HCD and find new issues that we haven't thought of. The tools need to be also accurate. So there cannot be, they cannot be just run and not return anything. And maybe they work, maybe they don't work. They need to be strict. So every time that we validate killing a machine or killing the process, they need to validate that it happened and the red seed responded because I know someone changes the, some code and it no longer skills the process and process that lives and we stop validating which happened to functionality tests that at the end they didn't validate because sometimes they didn't test or run the disruption that they were meant to do. And accurate, we need to know who if there is a failure, we need to be able to attribute it. Is it a problem with the test or is it the problem with HCD? Like if we don't have it, like we flip a coin and in finding the issue or interpreting or picking the issue and we flip a coin when when someone like spends a lot of time to try to reproduce it or understand it instead of getting a clear signal, is it, what, who is to blame? The tools also need to be maintainable. So they need to be run always by everyone. They can, they need to be run on each PR. They need to be able to be run every day for a long time. Part of the HCD testing is using the end-to-end suite which is just like wrapper for setting a process with configuration, but reusing the existing framework would save us huge amount of maintenance costs because we just need to run HCD. Like we don't need magic. We just need to have an HCD instance and check if it behaves correctly. Yeah, so for that I would want to go to robustness which is how I want to define what we are trying to validate. So robustness is ability of the system to maintain correctness under any condition. So correctness is any guarantee that we are saying saying to our users that we give them. And the any condition is any condition. Is it cloud failure? Is it process problem or disk or bytes get flipped? We don't care. This is, we just need a correct system. So to give a detail, here is like high level or overview with guessing how frequent the issues could be happening, but this is the level of granularity that we want to consider. So outside of normal operations which are the trivial case, we need to know if there is a problem if there is a normal packet loss that happens every day. We need to know if during upgrades is there an issue. And are we still correct even if people pick unsupported or not as obvious upgrade path? We need to also take account of standard failures like people just shutting down their machines. It's not always obvious that at every code line if the process is killed at any code line it will be still correct. And going into more obscure failures that happen pretty rare or even this is more number from on-premise, but when multiple bit flips occur you can get data corruptions of memory that fulfill the CRC code. So they seem from process correct. They could be random, sometimes randomly flipped. What does correctness mean here? At CD promises to the user two types of guarantees. One is about key value API and second one is about watch. They are separated because one is request response like and second one is more subscribing to some changes. So one is by delayed by definition. So key value changes need to be atomic. So either a transaction succeeds fully or it never happened. Key value changes needs to be durable. So if I make a right, this right is permanent forever. And no matter if your desk like disk fails or yeah, if process shutdown knew if I restored the HCD it should have the data that I wrote. And API needs to be linearizable. I will go into that further but it basically means that every change is ordered by the real time counterpart. So you can order the changes and buy time. The watch guarantees mean that watch guarantees gives you an global order. So at CD, all the changes to at CD have a revision number which is a global number and all the changes on the watch should be ordered by this number. At CD, watch needs to be reliable. So it should never drop any events or any part of stream within so you should get all the revisions. And watch needs to be atomic which means if there is transaction that changes a multiple keys, they should be sent as one unit within one response instead of being split between multiple responses. So basically you should observe every time you get an event or response, you should observe all the events that happened within this revision. So how do we validate correctness? We can take the part from the functional tests which is we take some scenario, we run at CD, we inject some failures and we send some traffic but how do we validate the correctness? This is the hard part. What do we take or what do we read from at CD or failures or traffic to make a decision how do we validate it? So this comes to broader topic of fact we cannot use traditional testing that is scripted. So tests like unit tests, functional tests, integration tests, they all follow a trajectory and they all follow some script that someone defined and can never derive. And the moment that the tests or what you are testing derives from what you are specced, the test fails. So we can't use those. How do we test generic properties? So for help, here comes exploratory testing. So if you ever did fast testing or ever did property testing, this is what I'm talking about. So not testing a scenario, testing some invariant, testing some property of the system. But the problem is how do we like, okay, so we have an approach, how do we do validation? Here comes model-based testing. So a model is a simplified implementation that is easy to understand by anyone should fit in a couple, let's say 100 lines and behaves like that full system, so full fancy, multi-distributed from multiple nodes system should at the end behave like a simple 100 line code structure or class. A model tests or using model testing requires us to collect operation history and from real system and valid replay it on the model. So if the model represents correct and desired behavior of the system, replaying the same, the history should give us answer. Is the system behaving as it's supposed to be? So for HCD, HCD is simple key value store. Like why cannot we use a hash map with a counter? That's all. But there is another problem. Which order, like here we have set of requests that were executed by multiple users. Multiple, there are concurrent requests that are done. There are different types of requests. How do we know what order they came in? HCD knows, but when we test we don't know what is happening in HCD. So how do we know? And here comes the linearizability testing or linearizability tracker, which is a tool that can take operation history and the model and find the order. So it basically, for the image here, it basically goes through every operation and creates a line that is consistent with how model would behave. So we don't know what order of the operations happened in HCD. We can just, based on history, derive one way it could do this, that is correct. If there is no way to connect the lines that is correct, it means HCD is incorrect. Or at least the test fail, and we can then go there and validate is the HCD or the model incorrect. So this gives us a full solution. The HCD robustness test is, as described, the part that starts the cluster, injects some failures, generates some traffic, then collects the history of both operations, which are key value put, get, and watch history. The operations combined with the model can be passed to linearizability tracker, which gives us the answer, is it correct or is it not? Watch history. It's simpler because it's ordered. We already know the order of these results. We can just write simple functions to validate it. So when we started this, or when I started this effort, we quickly figured out that no one knows how HCD is used by Kubernetes. So me, with Han Kang, we worked on defining the contract because without the exact contract, how do we know what to test? There were unwritten assumptions done long time ago that are technically correct, but no one wrote them down. And we discovered properties like renovability or watch being renovable, which means that Kubernetes supports bookmarks and it uses it to save progress. But if it's those safe or bookmarks need to be sent when they are sent, they need to guarantee that all the events before the bookmark were sent. And if not, Kubernetes will start from incorrect point. It will go either forward or far before instead of the last change. And we discovered that it was broken in HCD in some cases. So what are the results? We found, as of today, three issues most related to different parts. Surprisingly, there was one behavior that no one ever tested, which is HCD recovering its state from other members and we found out that watch can travel back in time. We found duplicated events. We found that diffrack can cause inconsistency. It was just really, really rare. So what we will be doing next or what I'm working on, what is the next steps for us? We had results, what we should do next. We need to codify the whole contract. We just touched it. It's not nearly done. We need to make the tests or failure injections much more advanced. We don't test disk connection or basically, yeah, mostly disk crushing and kernel losing the data that was unsinked. We are already working on Bebolt, which is the embedded key value store to have the same test suite or the same approach for testing to find really, really obscure failures that look like style and data corruptions, which means it's just a problem that most people don't discover. And, yeah, we can also validate the history. We can use the history validation in every test because every test that is an API test, so with tests goes to API or end-to-end tests go to API, is doing some operations, generating some history. We can validate the history that the tests are correct. We can check if our tests are correct or not. And the last thing, when we have contract and fully tested on XD, we can bring this contract, implement it in the model, have a model that fully implements the contract, maybe with some changes to make it more efficient. We can move it to Kubernetes and have Kubernetes verified back what is happening. And, thus, speeding up Kubernetes testing much more because it doesn't need to start full at CD instance. It can be just a process-level test. So, let's see how it works in practice. So, here I have a ready command that I can run which basically does three things, as described, and it's so fast. So, what is happening here is we are starting an XD server, we are checking its health, we inject the fail point, which is maintainers put in the XD codebase. Fail points in the codebase or comments that work as fail points in XD codebase that are in critical points and can be later used by this test suite with use of special library, they can be used to say to XD, hey, crash at this point. And we have many of those. And we then do a strict validation. If we set up the fail point exact here in raft before saved, which is a raft someplace when we call raft, we expect it to exit. And if it doesn't exit, the test also fails because we want to test that fail point worked. So, we should be strict here. The member exited as expected and this can be recorded history. We can take recorded history here 2,400 something operations. And we have some average traffic so we can simulate and require minimal traffic, QPS, so not only test low traffic, like playing around with XD, but also verify that traffic that we are getting is as high that we've seen in some cases and then validate it. And what the test does is model is not linearizable, which is incorrect. It means that there is a problem with XD. Here, I specifically took an existing issue and existing and XD version that was vulnerable to this issue and I reproduced it with one command and it took five seconds. At the end of the test, the test reports all the important information that anyone can use to validate. Is this the problem with XD or is it problem with the test suite or the model? So, we can take this URL that is here. I have it prepared but for demo sake, I will refresh it and we get this visualization. We can click jump to first error and we see that visualization has reported an issue. Here, we are using on the top or the name of the page is Percupine. This is the linear resolution checker that we are using and gives us this HTML file and it shows the durability issue I described, which is there was a put request that sets revision 601. Next put request sets revision 600 true and then it was never persisted. So, all the following requests have revision that is lower than that what client recorded, which means the model doesn't allow us to find any connection to the next operation and proves or gives a proof that a human can verify and read that there is a problem. Okay, so when you should use model best tasting, we can validate it's great for testing generic approaches to correctness. It separates validation phase from the execution phase, which means we can generate the report, verify that the operations or the model is the model incorrect or the HCD. If the HCD is incorrect, we have a proof. If it's the model, we can fix the model and rerun it on the same history. It should work. We can verify that we fix the issue. And as I mentioned, model is reusable so we can plug it in any one that wants HCD fast. Of course, not everyone needs 100% correctness. I mean, I hope you do, but I don't know. Model, I simplified the model. It can get really complicated mainly because it needs to assume that client can observe a data loss and this data loss not necessarily is lost. Server can crash before it responds the client even though it persisted the response. This means that as the next point mentioned, it's an NP complete problem. The state increases exponentially or validating linearizability. Cost of this increases exponentially, making the test pretty fragile if you have any bugs or any unoptimization. So it's not like one tool for everything. So if anything in this presentation make you interested in the topic, this is public. You can go to HCD code, read it and contribute new fail points. At the end, maybe I would think about generalizing it so not only HCD can be validated. That's the end of presentation. If you want to give me any feedback, you're welcome to make a screenshot and send it and let me know. That's all, thank you. Do we have any questions? Hi, Mike. Thijs, I was in the question. What's the maximum size you test HCD against in a number of objects? What's the size? Is it 10,000? Is it 100,000? The size doesn't really matter. It's about QPS. So if you have SSD, the more QPS you simulate, the higher the probability you will find the issue. So you can test even in memory. It needs to be persistent assuming that memory is... you trust the memory that is persistent. So it could be really, really fast, but I don't care about how, because it could be fully starting memory to make the test faster. Thank you. The mic is on this side, so if you have questions, it would be faster. Okay, sir. Hi, Mike. Thank you so much. I was just wondering whether you have already any plan for testing network latency and things like that, because I see that right now there are four points that you inject into the code base, but what about testing these other external conditions which are not necessarily in the code, like rashes? We are testing latency. I mean, latency is not an interesting part because raft is validated to be correct, but on an interesting point, we've found problems related to... it's more behaving like data manipulation, because if the packet is half-le-cast-cut, the packet matches the block, and then you lose one block because it's JSON, you could... it's correct, but there is no right. The missing... I expect this is the case with missing one right. You could cut... there's probably that traffic is cut in a way that allows requests to be missing, and RAV doesn't account for that. We have a... as they implemented already a proxy with very naive implementation, it's not great, and we... Yeah, I'm... I would want to improve to something generic. There is a custom solution which is not very advanced, but yeah, we are simulating network partition delaying and packets dropped. Hello. Thank you a lot for your speech. Next question. There is one slide which told us that our non-AWS setup was broken, and the question is, is there a more stable setup, configuration, or something else? I think... so we needed to fix... or I had a contributor, and that was trying to use Docker, and they apparently made... because they knew no closure, they fixed some parts of it, and apparently they were able to get results, but because they changed their project, I don't know if they left the... like, at CD community, so I don't know if they contributed the changes in... like, Jepsen infrastructure setup back to the Docker. So, like, AWS apparently worked great, but nothing else. Like, it's proven too. Hi there. I'd just like to ask about performance with ATD with splitting out. We're running non-prem, where... what's the name? All our ATD is running on one disk, and we've been advised to look at splitting out the well and data into separate drives, because it's all VMs, we can't exactly split that onto SSDs. What other performance tuning tips are there? I'm... Most of my work is in correctness. I cannot say about performance. There is a talk on KubeCon about tuning at CD. I would encourage to go there. I mean, I could give you examples from... So, what I give is usually look at Kubernetes scalability tests. They support 5,000 nodes. If someone claims that HCD is in performance, they haven't even read open source code that gives you ready HCD configuration that is... Kubernetes guarantees 5,000 nodes. PR release will be not cut if it doesn't support 5,000 nodes. Like, if you need more, we can ask Kubernetes, but it's ready configuration. There is no magic. That's some... Thank you. SQLite is famous for having one of the most comprehensive testing for correctness CI. I don't know, I wonder if you looked at the difference in practice if you could get inspiration from that or if it's too different. I read the call, or I watched the presentation by Kyle. I read some code. I understood some part of it. Collecting history is the same. What is done or what traffic is sent and what is verified is a little bit different. It's more advanced. Even in some ways, it's more advanced because it's not as fragile as the model-based testing I described. It more validates, let's say... For a limited model, there are some distributed system properties. And if it can... Because of limited traffic type or limited operation, I can send you to the... On the page of HCD or reports from HCD, the limited traffic allows you to make some assumptions about setting a variable or append-only traffic and give you properties that you can verify that if there is append-only operations, that means that it should always grow and some traffic's suffix properties. But outside of that, it's more... It's more advanced and more reliable, but I don't know if it's... Yeah. That's all I know. Okay. Thank you. If you have any questions, I can still answer them here.


Hello and welcome back to the craft box. My name is Brent enter a day We are on episode 35 of completionist crafter back in aquila My blood most plan worked really well. We are crafting the adepts mental of winter today. Obviously all of the things we need to hear And always as always on screen is everything you need all of the requirements and such. Oh a treasure chest Frost beetle TC not that important Yes, let's hit on home and craft this bad boy up, and then I'm gonna quickly show you my my Pink dandelion farm because that is the best way I found to get blood moss It's not really one of those videos, but I thought I should probably just mention it again Because in these you do actually need a lot of blood moss and blood moss is extremely difficult to get But yeah, otherwise most of everything else apart from red mandrake red mandrake can be a bit of a pain as well It's pretty easy to get Obviously a lot of it's just buying or transmuting, but there we go adepts mental of winter Let's let's have a look at that hats We have 220 damage health. I can't speak this morning 3% power put 2% resist 8% fire resist 2% ice Accuracy 3% ice damage and 8% storm resist obviously I school only level 30 on screen will be the comparison to the Zeus hat from aquila as well Just so you can get a juicy comparison between the two But yeah, I'm gonna quickly hit over to my botanical garden house, and I will show you The best thing about these pink dandelions Essentially these pink dandelions Get to roll the blood moss every single time you can harvest them So if you keep coming back every single day, you can just get more and more blood moss because you can harvest them At least four or five times before they come out I haven't even had this out of yet, and I've gotten so much blood moss. This isn't even the full farm yet Last time I think I showed you is like still baby, but now they're Not doing too bad. I've only just harvested them in the last like hour So they're still gonna be wild to go until they can be harvest again, but they are so so useful for blood moss I went from basically no blood moss to 30 blood moss in two days so Yeah, would highly recommend it if you're doing these Aquila crafting stuff or just any crafting to have a pink dandelion farm. It makes it so much easier Obviously you can find Ways to get this I just made this up myself this pot size and whatnot. So yeah, anyway Thank you for watching them to be like and subscribe before I subscribe press the little bell So you know exactly when I post another video and remember to craft outside the box


Obviously I'm honored to be here, so thank you to Julia and Dave for organizing this and Lionel for having invited me. It is two projects that I'm working on and because they're our current graduate students in the audience, I want to put them in context just perhaps to show you at least in my experience how sometimes ideas for papers or texts come about. And indeed they're in very different stages. I have circulated. There's a paper copy of the current iteration of the cloud piece, so please feel free to take one. There are incomplete bits in it, incomplete footnotes and that sort of thing. I'm still revising it, but it's there if you wish to comment. Please do so. I'm always glad to take your comments. So that particular piece, the cloud piece is in a more final stage. In fact, I hope to have it done very soon for the purposes of submitting it to a journal, in part because the topic is rather hot at the moment. The other project is a more long-term project that actually came from my default thesis as it's evolving. And so two very different kinds of work, two very different kinds of projects. One of them is very much a long-term project, the virtue piece, part of the book project. The cloud project is a side project that came up as a result of teaching a course last September in the law faculty at McGill. So very different origins and perhaps illustrative of how things evolve as you're teaching or conducting a career as an academic. So the first piece is the cloud. I guess as I had said, I was teaching a course in September about the time that Lionel Bentley visited us. In fact, it was towards the end of that week that Lionel arrived. It was called Analog Copyright, and it was a one-week intensive course taught with a fellow who produced music. A lot of music that I like including Black Sabbath in the Clash and Blue Oyster Cult. Maybe you wouldn't put Blue Oyster Cult in the same category as the Clash. I wouldn't either, but in any event, we had a number of exchanges on the cloud with our students. What is the cloud? Well, first we need to look at what the internet is. This is a picture that I took off of a website on the internet. The idea I want to convey to you is that the internet, as we all know, is a linkage of computers and servers. It virtually and physically, I suppose, circumnavigates the globe. All these servers, computers, link together either with wires or wirelessly. And there are a number of metaphors that I think are useful in understanding how this original internet worked. Now these are ideal types, so a few caveats. These are ideal types with respect to understanding the internet metaphorically. Second, we are still, for the most part, here. We're still in this internet age, if you will. I think we're moving towards an age that we can call the cloud, or the clouds, with what I will argue are a very different set of metaphors. But to some extent we have aspects of both, as they're ideal types or pointers, we're pointing more towards the original internet, as I speak. But it's changing rapidly. The metaphors that I want to deliver to you, first of all, based on that original idea of an internet being linked around the world, is the first metaphor, first of all, is horizontality. In a sense, all of the various links are on par with each other as they go around the world. If you were to send information across the internet, if we were exchanging a file, as I did this morning with Gaynor, that file would take a number of different paths in order to go through from my laptop, through the connection here in Cambridge, back to my server at McGill, and then back here. The file would be broken down into packets, and the various packets would take different paths to go back and forth across the ocean, and get here, and then would be reassembled into the document as I had sent it, for the purposes of downloading and copying. The idea of horizontality is, then in part, the idea that there isn't one path for the information to follow, very much multiple paths, and therefore very difficult to control any one path. Governments that try to set up barriers, for example, can do so to some extent for some period of time, but there will always be a way to work around it, if people really work at it, in terms of trying to get around barriers. The second metaphor is that the internet was meant to be open. Lawrence Lessig points out in his book, Code, that when the original programmers of HTTP and HTML had to decide on whether to use an open code or a closed code in working out these protocols, they opted for open code, which meant that the internet could expand rapidly, anybody could add to it. They built it to be open in terms of its architecture, such that anybody could add, and people did add to it. Brett Frishman, in his very recent book, Infrastructure, calls this end-to-end, the end-to-end architecture of the internet. So all the way from the hardware of the internet, the physical infrastructure, down through standards, the very standards that make it operate, to programs that we use, to the content that we put up, and then to social networking sites, was meant to have this open aspect to it, in terms of its very conception. That too is important, in the sense that we're trying, again, along the lines with horizontality, we built something that was meant to be open for, it was like a building that you could add on to, at any given point. Third metaphor is anarchy, this is obviously a bit overstated, but in the early days of the internet, people like John Perry Barlow were arguing that this was a lawless zone, John Perry Barlow was a lyricist for the Grateful Dead, and the first president of the Electronic Frontier Foundation. So anarchy is a strong word here. I think what I wanted to represent is the idea that there's a decentralization of power here, with respect to, again, the horizontality and the openness. So the internet wasn't meant to be controllable, in a sense, in a way that perhaps other aspects of governance that we deal with as lawyers, as legal academics, or think about as lawyers, as legal academics, would be controllable. Partly true because of the technology, it moves so rapidly that oftentimes the normative response is a step behind, or two steps behind, with respect to the WIPO copyright treaties permanently behind. So again, this idea that there is some flexibility here in terms of the way the internet operates and the normative responses to how it operates. Fourth point, robust interaction with the internet.  And for people that have the capability or the knowledge, they can do wonderful things programming on their own. They can, to a large extent, set the parameters for their own interaction with the internet. What Cori Doctorow or Jonathan Zittrain would call general purpose computing, and this power to interact, what Jonathan Zittrain would say would be that the power to create or to be generative on the internet, to have a generative internet. Finally, fourth, fifth metaphor, and this is, I think, the most controversial one. Again, it's an ideal type, and I don't pretend that this covers the picture completely. The metaphor of sharing, and I use it widely to mean, not just file sharing, but I use it widely to mean going on the internet, taking an object, downloading it, paying for it, or not paying for it as the case may be, depending on the business model or one inclination towards file sharing, as well as the legal norms that govern it. Making one's own copy on one's own hard drive, and then using it, perhaps, making it available to other people. So that's part of it, this idea of downloading, keeping a physical copy, sharing it. Also the idea of the software being shared and movements towards open source software. So taking those initial inclinations of the original programmers, and taking it one step further by creating kinds of software operating systems, as well as applications that could be used, modified, taken by other people, etc. So an important ethos. I'm not crazy enough to claim that it's the only ethos. It's a controversial claim to say that it's the dominant ethos. I'd be willing to defend that, but I also see, quite realistically, the weaknesses of that argument with respect to that fifth point. That's the internet I think that we've come to know up until now. Now, I think it's changing. It's changing with this system that we call the cloud. Basically the cloud is the idea that we delink the functions or that we would like our computer to perform for us, such as storage or computing capacity. We delink it from our own computer and we allow some other computer somewhere out there on the internet to perform that function for us. And we have no idea where that computer is. We have no idea if it's even one computer. It could be many. So that's the cloud. That's the operations that take place somewhere in the non-physical world of our own computer. It's obviously stored in a physical world somewhere. We just don't know where it is. It's probably in a Northern climate. It's probably in the UK or Norway or Iceland. But it certainly isn't somewhere that we can identify. The cloud came about because various internet players performing various functions, so sales, Amazon, research and information services like Google, realized that they had a lot of excess capacity and therefore decided to make this excess capacity available to people in order to make some money. Fairly well understood impulse in the market economies that we live in. And so they provided storage service and computing service and this began to move forward. Through an idea known as mirroring and virtualization, your computer gets tricked. Your laptop or whatever computer you're working on gets tricked into thinking it's dealing with computer X somewhere uniquely. But that isn't necessarily the case. It could be computers X, Y and Z. And you have no idea where they are. There's an exceptional amount of potential in this model. I'm going to make some comments in a moment that are skeptical about the cloud. And my goal here is to make you worry about them. But that's not to say that I'm not a great believer in the cloud. I would like to see this work. I think the capacity here, the potential here is limitless. I just think we need to open our eyes a little bit and be careful to, I think, give our movement to the cloud some direction to prevent what I think might be an era of increasing control over users. So far so good. There's a bit more technical descriptions in the paper if you need them, as well as footnotes to other texts that also describe the technicalities of the cloud. For my purposes, I think we're okay with just this basic idea of computing capacity out there somewhere that we have access to through the usual internet connections that we already have. I'm going to put you that the metaphors are changing. We're moving in this direction. We're not here yet, but I think already we can see a number of different trends. First of all, as opposed to horizontality, we're seeing a more vertical model, a more hierarchical model. In a sense, you have to move up to the cloud in order to have access to those services. And you move up into a cloud, it could be Amazon's cloud, or about half a dozen. It could be Microsoft's cloud, it could be Google's cloud, it could be HP's cloud. It's four of the six, there are a couple of others that escaped me right now. But you are moving up into that area, and you need to have access to get into that area. It's not something like the internet where you could find a different way to get in. You could find a back door, a side door, or your packets would move everywhere. Second, the architecture, the protocols of the cloud are not open in the way that the base internet protocols were open. The phenomenon here is one that's been called the wall of garden. Unfortunately, the real groundbreaking has been Apple, right? Developing, and I'm guilty of owning one, developing systems that are closed, that don't inter-operate with other systems. Meaning you have to, you know, if you want to buy into, you want to use Apple, you've got to buy into Apple. You have an Apple computer or Mac, you have the iPad or the iPhone or whatever, right? They work well amongst each other, and they present a wonderful degree of simplicity for the user, as between the various objects.  But you do have to get into their system in order to use it. And the same is true of their cloud. If you want to get into Apple's cloud, you have to use Apple. And it doesn't, it is important that that is important, the content is important from cloud to cloud, and indeed the hardware is inter-operable. The cloud makes it easier to do that, in a sense, right? You want to use Amazon's cloud and buy into Amazon's service. You're going to use Amazon's service, don't even try to move to Apple. So there's a great deal of potential here for closed systems to become the dominant model, in a way that they haven't been yet with the internet. It's true that Apple has always been there, but for the most part there are other systems that have been much more open, including open source systems, which I think work quite well, but even other systems, the whole PC system was a whole lot more open than the Apple system in terms of its interoperability. That creates the possibility for the third metaphor of control. You can have a degree of control at these nodal points as you move upwards into the cloud, that I think just wasn't possible on the internet. Yes, yes, you could control aspects of the old internet governments tried, others tried, but for the most part those were exceptions and not the rule. I think with the cloud control will be the rule and not the exceptions. Fourth metaphor, you'll recall I mentioned robust interaction with the internet, with the kinds of computing devices that we were using. That's already changing. We're more likely to interact with the internet with something like this, or an iPad increasingly than we are with a laptop. This is something that Jonathan Zittrain called the thin client. With an iPhone you become, I think more of a content taker than a content generator, but you also become beholden to the software that's used on that iPhone. It's very difficult. Apple has conceived it as a whole to make it easier for the user interface as between various products and software, but it means it's hard to change. It's very hard for you to get beyond the barrier in order to program things on your own. That's true with all of these various small devices. It limits what you can do. You really have to be good to get around it. You have to arguably put more of an effort into getting around it to do any programming on it, or to program it to make it do what you want it to do, then you could with a laptop. Again, Dr. Rowe and others point out that a laptop is a very powerful device for allowing you to generate what you want to generate. These other things are still powerful. Don't get me wrong, an iPad is still pretty powerful. You are conditioned to use it, you're controlled to use it in a certain way, and you have to be so much more adept at the technology in order to not use it in the way that Apple wants you to use it. This means that we become less democratic in terms of the internet. The ability to control from above, even as you have the device, and I think of the famous example with the Kindle eBooks in which George Orwell's 1984, quite appropriately, was pulled off everybody's eBooks because they hadn't quite worked out the copyright. So Amazon pulled it, even though people had already paid for it, so it just disappeared from your Kindle. They had to put it back. There was an app drawer, and Amazon worked out pretty quickly with the copyright holders what they needed to work out in terms of licensing agreement. But that goes to show you that these devices can be controlled from above in a way that was much more difficult to do when one was dealing with desktops and laptops. Finally, the idea that we are now streaming more than share. Again, this is meant to be an ideal type, but the picture that I want to convey to you is that stuff is coming down and we're taking it. We're no longer saving. You listen to music on Spotify or Rhapsody, depending on what part of the world you're in, and you can access all the music that you want. You pay for the service, but you don't ever make a copy. You can't make a copy. You move very much into a model of buying services, and there are technical terms, software as a service, infrastructure as a service, etc., a program as a service. They're explained in the text. But the idea is here, you become a contractual purchaser of services on the internet, as opposed to the old way of doing things which was to make your own physical copy onto your hard drive, and then you could do whatever you want with it. Totally, I think this may be a generational thing. I speak to my kids and nieces and nephews, and this is fine. Why would you want to have a copy? I think I'm still part of a generation that still like to have a copy on my own in case I can't get hooked up to the internet. You can always get hooked up to the internet. What if you can't? These are actual discussions that I've had with people 20 years younger than me who don't seem to see this as a problem. Maybe it's just my own bias towards having some kind of object, not necessarily physical. It can be a file on my computer, but I still like to have it. The sum total of this is, I think, we may be moving towards a kind of enclosure. We owe the vocabulary of enclosure to Carl Polany on the left, the Hungarian, Austro-Hungarian or Hungarian-Canadian political philosopher. A lot of people don't know that he actually lived in Canada while he was teaching at Columbia. So all the years that he spent in North America, he lived in Canada because his wife was a card-carrying communist and therefore couldn't get into the United States during the post-war period and McCarthy period and all the rest of it. So he taught at Columbia commuting from Toronto and had a capital influence on a large group of Canadian political economists at the University of Toronto. He famously talked about the enclosure movement here in England as the great transformation when lands were taken from a fairly complex normative system of commons and enclosed to allow for the industrial revolution to take place in England. James Boyle, who's on the right, used the metaphor as a second enclosure movement to describe attempts by copyright holders in particular on the internet through various kinds of rules that we see in the Infosoc Directive and the Visual Money Copyright Act to try to sort of add to their power, expand the power of copyright if you will into what's been called paracopierite or copyright plus controlling access on the internet, et cetera. My argument is we're making a quantum leap even from that in this regard because here with the architecture I believe is so much more controllable and the potential for control is greater, more effective than that second enclosure movement that Boyle was describing with respect to copyright holders. Many of those measures just won't work or can't work to some extent, maybe they have in some places, but certainly the idea that I want to convey to you is that if we are moving towards changing all these metaphors the potential for enclosure becomes much greater. So that's the fear that I want to strike into your hearts. Yes, there are privacy issues and a number of our intellectual property colleagues, privacy colleagues in both in Europe and in Canada, so people like Daniel Gervais now in the United States or Pierre Lagouche here in Europe have spoken about the privacy issues and written about the privacy issues. I share that. I think we do have a challenge there with respect to privacy, particularly in an age where we seem to care as lawyers about privacy and our 15-year-old kids don't seem to care when they're putting stuff up on Facebook. So there's a disconnect between what users feel they need to have in terms of privacy and what we feel we need to have in terms of it's because we're older or more mature because we're lawyers, but I share those various concerns, but my concern goes further. I'm concerned with the architecture and if we don't work at keeping this architecture open we could fail to realize the potential of the cloud as has been described by a variety of different people. This is obviously a very political question. It's not necessarily about IP as such. The original model of the Electronic Frontier Foundation was architecture's politics. It's now architecture's policy. I agree with Lawrence Lessig that I think the first model was better. So we need to think about the architecture of the cloud in another way. I propose that we think about it as a commons. Along the lines of a knowledge commons as Eleanor Oostrom and Charlotte Hess have proposed, I can flip through this quickly because we do have the other paper to talk about. I think we should think about the architecture of the cloud as a commons, so as something that we share and therefore we need to keep open. Such that private activity can go on in the cloud. I have no qualms with people making profit on the internet with a variety of different kinds of business models. That's fine by me. But I do think we need to keep the infrastructure open for others who want to use it in other ways and in particular to keep channels of communication open as between people. It's a flow resource as again Oostrom's work develops. Flow resources have a value that is a public value and needs to be shared and therefore we need to maintain I suppose open tabs in order to keep the flow of this kind of resource moving. It's a common good. This is something that's going on in Italy right now this whole discussion. Brett Frishman proposes infrastructure as a commons and I would tend to agree with him on this that we need to think about the net infrastructure as a kind of mixed semi commons. And this he's done in a number of articles but also in his book which just appeared about three weeks ago or four weeks ago. If we don't, and this is Frishman, private infrastructure owners have a number of reasons for choosing to reject a commons management strategy such as opportunities to price discriminate, vertically integrate and operate for a subset of downstream markets control future progress. That's exactly the fear that I've tried to describe in those metaphors that we have that we've just gone through. What might be solutions? I don't want to be completely negative. I think we need to try to make the cloud as, keep the cloud as accessible as possible. Beware terminology. When someone talks about the public cloud they mean Google, Amazon, Apple. Cloud that the general public can access even though they are privately held. They're owned by private entities. So I call that the privately delivered public cloud. So we need to try to keep that open as much as possible. I also think we need to try to work on a publicly delivered public cloud that will be there as a safety valve, if you will, to keep the privately delivered public cloud accountable. What do I mean by that? Well, with respect to the private sector and the privately delivered public cloud we have the option of competition law and incentives to try to keep moral persuasion, to try to keep technology interoperable, to try to make sure that there's no price gouging or collusion as between the five or six major entities that are operating cloud services. We could use other kinds of incentives to keep it open. So the usual things that we use with respect to private actors we've done it already with Microsoft, we've done it with others, sometimes with success, sometimes with less success. But in general we need to do that to keep the privately delivered public cloud as open as we can. I also think we need to have a publicly delivered cloud that at the very least provides services to people who might not be able to afford the services of these other private entities. So it would require at least some countries and here I think we have to push countries in Northern Climates in particular. One of the main challenges of providing cloud services is cooling, literally, and hence there's a great deal of cloud infrastructure that is moving towards places like the UK, like Norway, hope to think like Canada, like Iceland. And therefore I would hope that some of these countries would, using perhaps their universities, providing resources to the universities, computer science departments or whatever, to develop the structure and then maintain it. Now obviously this is a financial burden but my sense is it doesn't have to be all that big. It just has to be there. It has to be there to provide the service for people who otherwise can't afford it and to keep the private entities honest, if you will, make them accountable because if there is this publicly delivered option, chances are they will react by being more open themselves. So my view it doesn't have to be a massive cloud. It just has to be big enough to provide access to anyone who cannot or doesn't wish to use a privately delivered cloud service. I'll conclude with Lon Fuller, as I like to do. And you'll recall that in the morality of law, the first part of the morality of law, Fuller talked about law's aspirational morality or its morality of aspiration. But in the second part of the book, when he was trying to articulate the substance of natural law, he couldn't. The best that he could do was focus on what were effectively procedural principles. And the argument was, well, if we work on the procedure, hopefully the substance will work itself out in the long run. And that's true. I learned law in an environment where we thought Fuller won one of the heart-fuller debates and I appreciate that that would be different here. My own view is that heart won all the little battles, but Fuller won the larger point because at the end heart couldn't escape the idea that while there was no necessary connection between law and morality, there were so many contingent connections that in practice it was the case. Now, I presented this a couple of weeks ago and Kevin Gray was at a conference and pointed out to me that there was even a stronger argument in the second part of the morality of law with respect to what I was arguing. So I'm going to point out to you something that Kevin has pointed out to me. When asked to name one central, indisputable principle of substantive natural law, natural law of capital letters, what was Fuller's response? To open up, maintain and preserve the integrity of channels of communication by which men convey to one another what they perceive, feel and desire. If we were forced to select the principle that supports and infuses all human aspiration, we would find it in maintaining communication with our fellows. Communication is something more than a means of staying alive, it is a way of being alive. Even Fuller was pressured in that respect to have identified communication as being so central to the way we perceive, feel and desire. And I can't really say it any better. If we stress with the cloud communication and the ability to communicate, I think we can make the cloud live up to its potential and not allow it to slide into something that becomes another round of enclosure. So that's the first one.


All right joining us now to give more insight on the 2022 trade fine and survey are two gentlemen President of the Association of Small Business Owners in Nigeria, Asbon, Dr. Femi Egbeshola and the financial analyst Abiodun Olokho. Many thanks gentlemen for joining us on this discussion Thanks for the opportunity. All right. Can both of you hear me so far? All right. Abiodun, let me start with you. We're part of the investigation which brought out this particular survey and the report therein. Can you take us through the survey, the parameters and the highlights? Abiodun. Okay. Thank you very much. We have to take 227 exporters who completed the survey. It is pertinent to note that 65 percent of these exporters only do export while 35 percent do both export and import. In terms of the years of experience of being in the business, 0 to 5 years is 48 percent. Over 15 years is 18 percent. 6 to 10 years is 20 percent. Since we have 0 to 5 years of 48 percent, it is telling us that we have more exporters coming to the export business recently. So it's a good thing for Nigeria. Well, okay. Before I get to you Dr. Femi Egbeshola, let's talk. Let's stay with you, Abiodun Olokho. You said that the survey actually indicates that more exporters are actually getting into the trade space of Nigeria. But yet again, the country seemed not to be really having as much export in order to really balance trade when compared to imports. Thank you very much. It is obvious that the level of export has been increasing quarter on quarter. If we check the export proceeds survey recently by Central Bank of Nigeria, we have close to a billion dollars that has come in terms of export proceed in Q3. So it shows that we are actually making progress. This data has shown that we are making progress. When compared to Q2, we have less than or we had less than half of that. So we are making progress in terms of exports. All right, I will come back to you Mr. Olokho. Let's get to Dr. Egbeshola. Now major finance from this particular trade finance survey revealed that 94 percent of exporters experienced rejection of their request by the banks. You are part of the Association of Small Business Owners of Nigeria, Dr. Egbeshola, and all right, I think we lost Dr. Egbeshola, but let's stay with Mr. Olokho right now. I'll still throw the same question back at you from the major finance that have been revealed from this particular survey 95 percent or 94 percent of exporters experienced rejection of their request from banks. Would you say it's an issue of documentation or what exactly are the issues really?