Welcome to Warpforge. Welcome, Eric Meyer. Eric likes to hash things. If you can hash something and use it as indexes, then Eric wants to build software around it. And he also has two other talks here at GPN. You might want to check out. And the third talk now today is Warpforge. Which is a kind of, which wants to do reproducible builds. So if you do continuous integration and if you use containers and you ever had a problem with that or it's not as reproducible as you thought it would be, maybe Warpforge will be your solution. Perfect introduction, thank you. Hello, I am Eric. This talk is about Warpforge. It is a tool for building anything. Hopefully fast, hopefully repeatedly with hermatic environments, aiming for reliability and in contrast to some other systems, really paying attention to communicating so you can build things with friends. I also go by Warpforge on the internet. All of the slides are under that name. You can find me on GitHub and Twitter there if you want to bug me later. Hopefully this is a nice new monographic for that name. This talk is going to come in several phases. In the very beginning, I'm going to complain a lot about the state of the world as it is today because we have things that need improving. In the middle and for most of the talk, I'm going to talk about systems for building software and fixing the way we handle build descriptions. Focusing on how we can share things and how we can fix software supply chains. There will be a liberal application of cryptographic ashes throughout. In the end, I'm going to be stuck talking just a little bit about packaging and how we approach that so that it is easier to share things within the systems that I want to live with. There's going to be a tool called Warpforge introduced which is going to be a lot about fixing the bills and the supply chains. And then there's also going to be some conventions around how to package things that are more of a suggestion at the end but I hope a good and interesting suggestion. So, motivation. I use computers. This means I have to build software. I don't actually want to build software half the time, though. I would actually even just like to use software. In order to use software, I end up needing to install software. This requires the software to be distributed in some way. At some point it was built and compiled. Probably I have a problem with it someday so I need to patch it and then I need to compile it again and then I want to use it and share it. There can be a lot of steps in that process of making software available to people. And I wish that these were simple steps. I want to build all sorts of things. And once I've done so, I want to install them anywhere. And I want that to be simple. I want that to involve as few tools as possible. But I think the state of the world today is that building software is weirdly difficult. It involves a lot of manual effort. Usually if you're building some kind of open source software, you're going to go look at the project, read me.  Maybe you use some system package manager. Maybe you use some language package manager. Almost certainly you use both. The incantations might depend on what kind of machine you have. It's a fundamentally unscientific process in which you have to guess a great deal and try stuff. We manage to survive with this, but it's rough sometimes. And if you want reproducible environments and reproducible outcomes, especially if you want to try to detect bugs that only appear in some combinations or versions of libraries, good luck. Most of our tools will not help us with this today. This could be improved upon. My anecdote at a hammer this point home is I tried to build something from source the other day and it was rough. I picked a random project off of GitHub that I thought was cool. And I ended up needing to install GCC that I needed to make. A make file tried to get a different malloc library. It asked for root to do that. That was uncomfortable. I discovered I could use the system package manager but it was guesswork to discover that. By the time I got through building this one selected open source project, which is by the way really cool and I'm not picking on the one named here, but this is just like a real story that happened last week. I needed about 15 different things. I needed one language. I needed a package manager for that language. I needed a package package manager. Installing the latest package managers in Node was a journey of get Node, get NPM, use NPM to get CorePack, use CorePack to get Yarn, then use Yarn to get more packages. That was fun. None of those things were a pinned version. I ended up building stuff across multiple languages. Implicitly Bash was used as glue. GCC got involved and I definitely needed my system package manager for GCC. By the time I was done, I don't know how many other C libraries I depended on. I have no idea. It was unclear to me. I didn't S-Chase GCC so I don't know what happened. All of these things should have been versioned together. If I wanted any kind of reproducibility or like a clue what just happened in my environment and none of them were. I think the total count here is about 15. I might have forgotten some. So this is the state of the world that I want to improve upon. I want all of the supply chains, all of the things that went into this process to be identified and to be reproducible and ideally with no action for me. Like it should be a clear log. That's focusing on the build side. Even installing software though, if I've built it, can be very difficult. I tried to install Emacs the other day. I honestly failed. I found that app on my system. I'm a new boom 2 user. Make fun of me if you want. I know it's very common, very basic. It could give me an outdated version or I could install a PPA from some other person on the internet. That turned out not to work. I don't know why. I got tired of debugging it. Or I could find another version of this package in Flatpak. But then it worked strangely because it had some default sandboxes. It couldn't find my actual like .emacs in my home dirt. I don't know what was going on. Or I could use a totally different distro. You know what? I don't want to make fun of Lisp, but installing things should be easy. Installing things should be about as easy as unpacking a tar ball. And that's not the experience that I have today. I experienced complete system distributions that want to own everything about my computer and determine every part of the world in order to provide the first iota of utility. This is really frustrating. We should have more composable systems than this. Honestly, installing something should be open a tar ball. Okay, so these are the dreams that I have. And existing systems have maybe tried to improve the world. Like Linux distributions exist to try to make computers usable. But I don't think that they do. If you're going to use a Linux distribution, you have to buy into everything about that distro. It's a whole machine thing. And this means it's not really satisfactory if I want to have a dream of a universal build tool that you can use freely to do whatever you want and expect other people to come with you as well. I think Linux distros are even quite questionable as install tools because the norm in them is to make sure you only have one of a thing. And I don't want that, right? And I don't think anybody really does. I think NVM appears in order to help people get different versions of Node.js, for example. Like people want to bisect on different versions of things. This is normal. Linux distros fight you, almost all of them. It's really strange. In practice, the reason that this often comes about, I think, has something to do with ELF library headers and silly errata that are utterly fixable. The latter half of the talk will get into this. So what I want to hammer home about distros and what's not what I want about them is Linux distros try to solve problems of usable computers by making everything fit into their own universe and be compatible with themselves. I want to produce software that anyone can use in any universe. And that's a different challenge. Now maybe you've been listening to this series of complaints and think, ah, containers. Containers give us hermatic environments. That solves tons of things about building and distributing. Well, it's a start, but I'm afraid that containers are a necessary but not sufficient sort of thing. Containers are like having a clean room to do science in.  And now you try to build. There's twigs, there's leaves.  You don't know what's in that. And that's very hard then to work with to produce clean, understandable things. Having the clean room, good. Not knowing what you put into it. And then being stuck as many container systems are with a whole, just keep a monolithic image. It doesn't help at all. It also ends up not helping with the install story. Containers have caught on very nicely in like Kubernetes and things that are microservice networking. Because there they're composable. The actual file system part of containers, like if I want to have a series of processes exacting each other, there's no usable composition story here. So this doesn't solve a lot of problems that I want solved. We need more composable systems than this. Now I'm gonna jump ahead to a couple of other comparisons with other things that are more relevant real quick here. Because I think people probably won't listen to the rest of my proposal until they know that everything else has been considered. So Docker, containers. Containment is good. But I want something way more than monolithic images. Another tool that exists already and kind of exists in this build anything domain that I want to explore is Bazel. Bazel is really close to being what I want. Warpforge, this tool that I'm going to propose in a bit is similarly a build sandbox and has like declarative languages for what we're gonna do. But there's a couple of things I want to like ratchet up from what Bazel did. Warpforge is gonna focus even more than Bazel does on being hermetic. We're gonna use containers and we're not going to let things escape them. Bazel has a very easy way of putting like container equals nah in its config. I don't like this. We're also gonna focus a lot more in Warpforge on having a way to communicate partial build instructions and snapshots of data. So my experience of Bazel and watching people use it is that it sort of assumes you have a mono repo. I want a much more decentralized system. I want to let people collaborate even across teams and like without synchronizing, this should be possible. So we're gonna work a lot on that API. Warpforge is also very likely to be compared to Nix which is another functional declarative package manager and like Bazel close, really close to what I want. If you're familiar with Nix, you're probably familiar with the Nix language. Warpforge is not going to have that. Someone in the audience just says, yay, yeah. I'm not gonna pick on the Nix language too much because I think you can just like search for that on the internet already. Warpforge, as if I say, is going to focus on APIs, message passing, like plain JSON stuff. We might introduce a templating language but it will be optional. In contrast to Nix, Warpforge is also going to be content addressed by default and that has huge ramifications that I'll get into in a Q&A section if somebody wants to go through that. So all that said, yes, I'm proposing a new system. I have seen this XKCD. I'm gonna try to make something better anyway. I hope it works. So now onto the actual attempt at solving things. I'm going to offer us the system called Warpforge which is going to be a tool for building. And a set of suggestions called Warpsis for lack of a better title which is going to be a style for packaging. Both of these are going to be fiercely independent from each other even and everything. I'm going to try to build the smallest possible reusable bits and proceed with those. Warpforge, the build tool is going to be about building anything and solving that version together problem that like list of 15 different things that I wish were connected somehow. We're going to try to connect that in Warpforge. The Warpsis recommendations are going to try to solve more of those composable systems requirements. So basically this is solving builds and packaging. And I did have to cram both of these in the same talk because they end up functionally kind of interdependent trying to minimize that but you'll see where it shows up. So first builds. First of all I just want to demonstrate a tool exists. I am offering you working software today not just a series of pipe dreams. But I'll get more into demos later on. First I want to go through the goals of this system and then I'm actually going to drag us through the APIs of the system because I think that's going to be the clearest way to see what it can really do. And then we'll have more demo screens introduced in the middle of that. So the goals. I want zero parameter total environments for my systems. I want to run things with no arguments and have them work. Once I've written some config files but then I version control those, right? I want a software bill of material for everything that ever happens in the system. And I want this to be load bearing. I want the bill of materials of what goes into this to not just be like a log that comes out to the side making vague statements about maybe what happened. I want it to be something that I execute to prepare the environment and then it is load bearing. So we know it's the truth. I want this to be done with hashes as I was introduced. I really like hashing things. I want a completely decentralized system. So every time there's data going in or out I want it to be covered by a cryptographic hash. So I know exactly what it is. No DNS lookups should be involved in determining what is going on in this system ever. I want granular inputs. I want to avoid the monolithic image trap. I want to say where my input data goes. And sometimes I want to say which output data I save. Again avoid the monolithic image trap. I don't want to have to snapshot an entire system every time I want to say, okay the build output directory, just that please. That should be natural. Now the requirements start getting more complicated. I mentioned I want people to be able to collaborate with folks without being in lockstep in a monorepo. So I'm going to need some kind of format in the system to map human readable names to these hashes that are the identity of data that I'm passing around and also to the identity of build instructions. And I need some way to share those. That's going to be fun. I want those things to work at all scales. So they have to be decentralized. I'm going to put more hashes on it. There's going to be merkle trees for days here. And because I want this to be able to work for almost any sort of process, we're going to need graphs and pipelines. And then we just got the soft goals of be usable and be optional actually. I want something that you can drop in every get repo you ever use. And it's helpful to you. Or if you don't want to use it, okay, fine, you can go back to reading the read me the hard way. So these different goals are going to be covered by several different API concepts in the system.  There's also some goals that just require being careful. And there's one more huge goal that I've already alluded to several times, which is everything must be API driven in the system. I want messages. I do not want programming languages. I don't want my favorite programming language. I don't want anyone's favorite programming language. I want APIs. It should be easy to give instructions to the build system by API messages. It should be easy to get results in API messages, plain JSON. I should be able to pipe things from this build system to JQ. That should work. If that works, it should also make the system very easy to debug and easy to have audit logs of. That just seems natural, right? Okay, moving on. That was a lot of goals and credit to every other project in this space because I know that this is competitive to many other things that many other people work on. This is a very hard space, solving all of these goals, holy smokes. So thank you everyone who's ever tried to build software. We all try. Okay, so Warforge has a series of API layers. I'm going to start with the simplest ones. They're the most boring. They'll have the most hashes. Then in each consecutive layer, we're going to add a little more flexibility and utility. So in the beginning, Warforge has formulas. They mean very simply hashes go in, you do something, and hashes are going to come out. It looks kind of like this JSON object. This is an executable thing in Warforge. In the inputs map in JSON, this is a boring one by the way, it only has one input, but that slash there, that's a mount path. You can put more stuff in this map. Content comes in by hash. You can have a section where a bash script or your choice of thing, Exit, there's options here, does something. And once you're done, you can say I want an output from this path, pack it in a tarball please, and put this label on it. This is the fundamental unit of execution in Warforge. And you can run such a thing as this. Warforge run, the formula. So here's a demo that runs around with some CSV. You can see at the bottom, this also kicked out another JSON object. Yes, APIs. This JSON object that it kicks out is called a run record, and it has a section of results where the keys in this map match what you asked for. That's it, this is a simple primitive. Also, exit code, useful. This is gonna be the foundation of everything else that happens in Warforge. These things are at the bottom of everything. If you want the most granular logging imaginable, you just log these things, and you will see exactly what is happening. Because there's also hashes all over the place, it should make it super easy to check anything you're doing for determinism. If you wanted to know, I've lost the screen, how did this happen? Okay, that double time, cool. All right. I'm gonna hope that the formulas thing was on the stream already. Okay, cool, JSON everywhere, great. Moving on. By the way, I did use containers for that. Containers aren't lame, it's monolithic images that are lame. Composable input system makes them good. So, these formulas, these JSON objects, these are the bottom of our universe in Warforge. If you're thinking that they're verbose and that they're limited, yes, that is also the case. If you use just this API level, you would be copying, pasting hashes all the time, and that would obviously suck. You'd have work to get done, so do I. So, more abstractions will come. I don't know why I put this slide in this order, but by the way, there's schemas for everything. That's good, right? The next layer of API up, stepping to like layer two of API in Warforge is something called plots. These are gonna let us do multiple steps of things, whole graphs even, not just linearly. And it's going to start introducing features that will help us avoid copying and pasting hashes. So, it'll start associating human readable names with inputs and outputs, and some more stuff too. So, here's an example plot object. This looks a little bit similar to, it has things that look a bit like formulas in the middle, right? This has two formulas, step one, step two. These are obviously all dummy values, but you can get the idea. These things are wired together by pipes, if you will. So, you might have noticed that- Is my cursor show up? Yeah, great. Inputs in formulas were always the keyword where, followed by a pack type instruction, so like please use a tar ball, followed by some hash. Now, in plots we can use other things in addition to wares. For example, this pipe reference. This one has a empty in the middle here and then the word thingy. It's because it's coming up to the top level and putting thingy in here. Down below in this step two, this pipe is referring to step one, the output called stuff. All right, you see how this goes. You can build graphs. You can also run plots. So, this is a bigger example of doing some stuff with a shell, counting words. And you can run a whole plot and it will run a series of containers in sequence. You can see how the hashes are coming out in the middle and it's basically just getting stamped into the next part of the plot. So, this lets you do bigger, more interesting stuff. Here's an on moving snapshot of the same thing. You can see how this hash is coming from right up here. Cool. You could have glued this together by using the execution formulas and like JQ and some bash glue and stuff, but everybody would have to write that same glue. So, we build it into the system. Now, more interestingly, plots are gonna start using some of the communication tools that are so important to letting people collaborate without wanna repose and so on. That first example just used a where ID in the beginning. Soon, we're gonna introduce another system called catalogs which are going to map a human readable name to a hash like that. Catalogs are gonna get another heading momentarily. Other stuff that, why does this need so many transitions? Plots can also use another system called ingests. Now, this is a little bit spicy because this is going to let you do impure things and it's going to let you look at the host. This keyword at the bottom here. You can use a get ingest, for example, to look at a get repo in the current working directory on your host and get the head commit. Warp Forge knows what get is and it can do this. The key concept of using ingests is that first of all, Warp Forge is going to warn you because it's doing impure things and it's not going to let you go wild with this. And also it's going to immediately turn it into a where. So this will get sort of transformed into the string where colon get colon the get hash. So now it's time to move on to the communication parts and introduce what that catalog bit was about. Catalogs are going to be a data structure for sharing information about both data that you have produced in the system or imported into it, as well as, how's my time? How's my train of thought? Bringing data into the system, identifying it, also tracking, rebuild instructions for anything that we've built in the system. So these are just going to be a huge repository of knowledge. And they're going to be designed as a merkle tree. If you're not familiar with that, in brief, it means a series of hashes where each thing covers the thing under it. Git is a merkle tree. It's why you can trust Gary Pose, generally. So a catalog is going to start with a human readable name for some project. It's going to contain another human readable string for release versions. And then it's going to point to another document by hash. In that next document, it will contain a third tuple of the human readable name. So attributes like source and architecture tuples go in this part of the system.  There's also some free form metadata that can be inserted in here where this is meant to be an extensible system. In this example, there's also something called a replay shown here which will be how we do sharing rebuild instructions and more examples of that will come in a minute. Since catalogs are about sharing things with other people, it's also going to have to start talking about where to get data. This is a practical reality thing. So you can associate a mirrors document with a catalog and give URLs for fetching individual content hashes or you can say anything under this like glob of name go get from this content addressable bucket. There's a bunch of options there. And this is how we then make these names usable in plots. So when a plot uses a catalog reference, here's your three tuple. You split this on the colon character and then go look it up in a catalog. Now this seems like maybe I did something impure, right? The key element of catalogs is that we're going to make sure that every time you have a plot, that's a file that you put in version control and every time you have a catalog that needs to be in the same version control structure, meaning you're going to have one hash that covers all of these things. Assuming you're using a sensible version control system. So these catalogs can be projected onto a file system. I'm going to start going really fast because we've lost a bunch of time in the technical difficulties. And catalogs are where you can also bridge data in and out of the system. For example, there's a bunch of sub commands of Warp Forge which will let you look at a get repo, get all the tags out of it and instantaneously transform those into a catalog. That's cool. Here we're doing it on Linux. All right, recap of the API structure so far. Formulas, do one thing.  Catalogs, start communicating. The other thing we're going to put in catalogs is the replay data structure. These are to communicate builds that you have done before and let you do them again. This is actually really easy. I don't even need to show a new API object for this because it was already plots. If you use any ingests, that would be unreproducible. So we're going to freeze those, but then it's already a replay. This is something that we do in practice if you want a get repo for your source on your host, you can say, yeah, use that. And when you're releasing as long as you put an output in the system that contains that same information, we can just automatically freeze that into a self-reference. This lets anybody else, whoever wants to reproduce this run, do so from the information already available in the catalog because that will then be the get content hash. These replay files go in the catalog along all that other data. And something I want to highlight is cool about how this works, especially this is contrastable to other pure functional build systems. Catalogs are based on this concept of snapshotting the data and having the hash of the data and then associating the replay instructions with that, which is cool because you can actually have cycles, right? We kind of just saw one right here. You can make a cyclic reference and that's fine because you're snapshotting it after the cycle is completed. This means we can explain even build graphs that have cycles, which is really cool if you wanted to say, fix the trusting trust attack of compilers. Replays also, of course, let us build stuff. If you're missing some content and you want to recursively build it from the replay instructions, you just add dash R to the run command. It's great. All of this stuff, when you're running it in practice, takes place in a workspace, which in Warp Forge is basically any directory that has a dot Warp Forge directory. This is where the scope of name lookups are. So that catalog file system, it goes in this directory. Workspaces can be nested for practicality reasons, so you can have one root workspace, which contains a bunch of catalogs that you imported from the internet. And these can be used as auto-complete suggestions or so on. And then our recommendation is that you should put another... No, that's not the slide head. Every Git repo, every version control root should itself also contain a workspace because we want that catalog snapshot and the module to be next to each other, the plot next to each other. So we have a vending tools that can take all of the references from your root workspace and copy them into a smaller one. So when you're using the tool in practice, you can ask it for the status of some module like our build instructions for Python and it will say, yeah, you have these references to these catalogs. I would love to explain these things more slowly, but I'm quite worried about my time. So put these in a Git repo, you're off to the races. It's a vending of lightweight metadata that's gonna let you reproduce stuff. Now, you're still writing lots and lots of JSON and if you want to do lots of things very efficiently, you probably want to write less JSON or you want to have some reuse layer. Now, at this point, I'm actually going to give up a little tiny bit and not necessarily solve it totally in warp forge because I want to avoid having my own personal opinions about programming languages, reduce the number of people who can share this tool. So bring your own JSON object generator, please. Or okay, yes, I'm going to include at least one batteries included answer. Starlark from Bazel is actually kind of cool. It's a Python dialect. We're going to introduce support for this into the core of warp forge. It's going to be non-special. It's just going to emit JSON objects.  This is still a work in progress. So this syntax is wildly non-final, but the idea is having a language gives you some degree of being able to reuse concepts and components. Yeah, so in recap formulas, all you do basic stuff, plots and these other structures let you do graphs of it and then catalogs let you communicate it. And then at some layers above this, you can introduce whatever kind of templating you want. Maybe I'll offer some Starlark and that's the batteries included easy option, but anything you want to do is fine. As long as you're producing these structures like plots and replays, the whole system can still introspect, understand what you're doing, produce good audit logs, et cetera, et cetera. There's a couple of other cool things like we have quick commands for like, oh, you want an interactive shell? All right, fine, we can have this. You can have local host CI in this system basically for free. You use a git ingest. All of this works rootlessly just for the record. You never need sudo for any of this. Now as I continue to run low on time, briefly packaging. Yes, we're starting to package things in the system. Mostly I'm trying not to care about any details of that except for one thing. I want binary packages to be path agnostic, by which I mean if I put them in some install folder, I want it to work. This is a handy attribute because it makes it easy to compose things with this kind of JSON. And it's also a very high and mechanical sympathy to containers in general. I want zero step installs. I want no post install hooks so that when I put an application in a file system, it should be a read-only mount and it can go. That just makes sense to me from first principles but it's also fantastic for container usage because it means you can use a bind mount and just go to space. If you have things that need to be moved around on a file system in a container, an overlay FS will cause all sorts of invisible copy operations on the bottom. Things get really slow.   ELF headers are kind of nuts. I'm not going to explain every part of how you fix this but long story short, there is a way that you fix this. You can play with ELF headers in Linux such that you can have libraries be relative to the path of the binary. If you follow this to its conclusion, you can have directories that you can mount in any position and they will work. This is a way to start building reusable components that do not need any support from the system around them whatsoever. This is a way to get out of the trap of distros building into their own universe. I've given a talk about this before as well back at All Systems Go in 2018. There's a much longer rant about the needs for this and the effects that it will have if you want more information. Or you can follow these links to like working implementations. There's one more thing that comes up if you're stitching together lots and lots of applications each in their own directory like that. At some point you need a path. The thing that your shell looks at to pick where a binary comes from. Use a Simlink farm. This is a solvable problem. That's it. I don't care about anything else in packaging. I just want things to work together by being isolated and it can be done. Okay. All of these systems that I have introduced are early. There is working software and if people want to come to those links and like get binaries, try them out later. Please do. However, it's early. We're trying to build APIs and like sort of functional spaces where people can get together and build stuff in the future. As demos, there are some packages for like GCC and Python floating around, but there is so much more stuff to do to make a practical environment with lots of packages. If anyone in this room or anyone watching would like to come along and help with that, please. We have lots of future work in the tooling, the Starlark thing, like I said, is coming up many other things that we would like. And I would like to package more stuff as well. We are currently resisting building a distro and I hope we continue to resist doing that. But if we follow some of these packaging recommendations and we can produce things that are reproducible, hermetic and easy to rebuild with Warp Forge and are usable everywhere, gosh, that would be cool. So anything that anyone sees in here and they want to help make it better, please come talk to me later. It will take many people. Hopefully, if we are successful in this project, it will produce a cultural shift. I want reproducible builds to happen. And I hope this tool and the way that it exposes hashes and makes things visible and debuggable will actually help us get there. Not necessarily by solving reproducible builds themselves, like patches to everything that has some random component is necessary to fix that thing. But this tool chain should help make that visible and help make it fixable. I'm hoping that this eventually makes computers easier to use and I'm hoping that these packaging conventions mean more people can share more stuff and honestly make the world a better place. I think I've made it through in a halfway decent time. Does anyone have any questions? Maybe, how is my time? Your time was great. So we have 12 more minutes for questions, which is more than we expected. This is great. You made up time great. So yeah, raise your hand if you have a question and I will pass you the microphone. Yes, you can. What kind of tools or languages are you using to build this? Oh, good question. I also think you know about the same language. Yeah. Is the mic, can you throw me the mic or something? Sorry. Okay, try again. So what kind of tools and languages are you using to build this? So all of the Warpforge tooling is built in Golang at the moment. I try not to treat that as too interesting of a choice. But lots of things in Golang are very practical. It's also very easy to get reproducible builds out of the Go compiler. So we have Warpforge building Warpforge with a consistent hash already, which is very pleasing. The container system, we're reusing RunC. It works. It's the same thing that's at the bottom of Docker stack in practice and almost in install. But RunC is just the containing parts and none of the image transport parts, which means thank you, whoever isolated that API. It's like very correct. Okay, awesome. Hopefully also lots of things are pluggable. So like where the hash said where, call in tar, call in plet. There's a whole plug-in system there for these transport and hashing systems. So if you wanted to plug in somebody mentioned IPFS earlier, that should be easy. There is like a switch case where you can start dropping more code in. Any more questions? Yes, please. Hi. Is it on? Yeah, it is. Thank you first of all for this quite interesting work. And the first thing I was wondering if I want to start putting my tool of choice which I wrote into the system. Do I really have to make packages for all of the dependencies including GCC and all of the compilers? Because I think it kind of comes down to like everything should be built in that system, isn't it? So you can use the system in a pretty degenerate way of just like do the whole monolithic snapshot thing like other container systems and then build. If you want like a quick escape valve to let you do something without thinking about it too much, you can do that. And of course we're doing that like constantly as we bootstrap things as well. I'm hoping that we will build a larger ecosystem of packages very quickly that will make it easier to do more stuff faster. Like we do have a GCC and a Python already for example. We even have a path agnostic Python by the way. Which is really cool. I don't think anyone else on the planet has that. But yeah, inevitably there's going to be a lot of stuff that needs to get built in order to have the nicest possible experience. I hope the fallback to monolithic snapshot helps people accomplish things and increments as well. Okay, more questions. Yes, yes. So at least if you are using something like tend to you probably ran into issues with like GitHub changing the table beneath your wall. But basically, and it obviously breaks, hashes. You got any solutions to that one? Because one thing which could work for example is not relying on the table GitHub for example provides but doing a shallow clone of the repository itself or something. Yep. Yeah, I remember the day that GitHub changed their hashing algorithm or their exact export algorithm. That was fun, good times. There's a lot of different ways you could solve that. I think just mirroring the actual binary data is like always your ground truth. This can never hurt you sort of choice. The transport plugin system is also part of my answer to that. So this tar system we're currently using actually does a semantic hash of the tar ball believe it or not. I wrote that code. It means that you're immune to some of the details. I don't know, you're giving me a thumbs up but I actually don't know if I regret this or not. It's more code to maintain and that's worse. But it does mean that we're immune to some things like the tar file order. The actual hash in that system is based on the syscalls that I give to the file system. So it's correct and nothing additional. Nothing about the compression algorithm either for example. But yeah, at some point you just have to have the data available, right? We're gonna use cryptographic hashes so if you can't keep that data around at some point you're gonna have fragility issues. Yeah, since we're on the roll, more questions. Oh, yes, yes. Also, you can repeat the questions that the stream didn't hear. How would you compare Wobforge to BitBag from the open embedded ecosystem? From compared to what? BitBag. Oh heck, I'm less than familiar with that one. You should look at it because it's basically similar in many, many ways to what you're building here regarding repeat usable builds using hashes to exchange artifacts to see which parts change and which parts can be reused. Cool. If more people are trying to approach the same goals that's fantastic. Mostly what I have seen from other systems and I'm not sure if this is true of BitBag is that people are working on having build trees and there's less emphasis on the communicable subcomponents and like being able to detach whereas something that's very unique about Wobforge is every time there's a catalog, you can detach. Those are like points where you can collaborate with other people asynchronously. But I'm less than familiar with BitBag so I'll have to research that later. Okay, we might have time for one more question if there is one. Yes, last question. Maybe it's not really a question but an idea. I think you have a lot of work to do building a very large catalog and maybe you already came across the idea but did you ever thought about there are a lot of distributions which already have a lot of recipes how to build software and trying to invest into writing a software which converts that build recipes into your recipes? I think it will be kind of hard because most of them are having like programming languages which are too incomplete, that's a little bit annoying but I think maybe you can make a first guess and then get a recipe which like 90% works and then you do some hot fixing or things like that and then you can build a large catalog fast. I've definitely not tried to automate things at that tier but I will say that there are some distros that I enjoy looking at the source code of a lot more than others. Looking at Arch is usually a fantastic source of inspiration. It's bash, oh no but it's usually very linear and you can tell it's going on very quickly so those have been generally inspirational but usually you still end up with some manual transformations and I have no idea what we're going to do once we get Starlark in the system by the way. All the stuff we've packaged so far we're writing JSON files and so our copy pasta level is still like a little bit high like there's certain incantations for how to do the R path twiddling and then the LD shim thing that are like manually copy pasted right now. So I think we're going to end up with some sort of functional templating system to do that and I don't know what it's gonna look like yet. So there'll be lots more to discover once we get there. Okay, maybe one short more question if there is one. Otherwise I think this is a good point to end the talk and give the next people more time to prepare or at least some time because we had the technical difficulties. Thank you Eric, it was a great talk. Thank you everyone for the excellent questions. And thanks everyone for bearing with us with the... Oh, I had one more slide that I should flash. Here's all the links to all of the things. You can get more documentation at projectname.io, warpforge.io. Source codes here it's under my personal username on GitHub. Here's where we're packaging stuff so far so there's a bunch of JSON files there and if you want to chat, please do. I'm a Matrix fanboy so that's where I'll be but it is also bridged to the same name on Libera if you're IRC people. So please come hang out. I'll be lurking there momentarily. Thank you again.


Hi, I'm Mark Hall with the Alabama Cooperative Extension System here with Ohio State's Brighton Shining Star, Dr. John Fulton. Our next lesson on advanced precision agriculture series is digital agriculture. John, as we look ahead to precision agriculture, the importance of data is growing and has a larger significance. Can you tell us how this evolution is going to work out? Mark, what farm press doesn't report today on something about not only technology but it has the word data in it. And as we think, not only today, but really what's happening kind of in front of us, just to kind of keep things progressing here, we're starting to call it digital ag. And so, you know, we had talked about just kind of thinking ahead, where we're at in thinking ahead in digital agriculture is where I think we're heading and so just kind of going through. But as we look, you know, this is kind of, there's not a company out there, doesn't matter what continent or type of company, Brighton Ag, that doesn't have a, you know, something like this that they've shown. And so this gives you insight about where the things are going and for some growers, you know, that maybe they've already got quite a bit of connectivity on their farm. So perhaps leveraging consumer devices like our smartphones and iPads, I mean, that's just growing. I mean, I can't think of the number of apps today that you can utilize either for information or connect right to the machine or visualize data or get access to data. So, you know, but one of the questions not only for farmers, I mean, Mark, you know, you've been in extension for many years. Where do you see yourself, you know, in that picture? Looking at a smart phone, talking to farmers about what they're doing when I go to party and farmers are there, you know, what we'll do if we get out our smart phone, hey, Mark, if you're seeing this, it shows the right, it shows what the weather's doing wherever you want to look. It shows this, it shows that. Boy, this technology is making life better, John. And the old guys, young guys, everybody just give it to me. There's not a probably a commodity meeting that you don't go to that they probably got smartphones and 70% of them have a tablet or iPads with them. And so, you know, this is real. This whole idea around digital ag is real. We're at the forefront of this. I pinned 2012 as a real key where the industry said, you know what? We've talked about it. We've tried this multiple times, but today in 2012 or that year, they made a commitment and the industry's just kind of followed it. And so I don't think the idea around data and the components of technology being embedded in machines is going away. This thing's here and it's here to stay. And so I think as whether we're extension, we're in the industry, you know, you've got to be thinking about where you're going to plug in and how either you're going to supply information or help growers make money in a digital way. John, the data, one thing that just baffles me is the amount of data. I was talking to Christian yesterday, and he was talking to the IT guy for EuroSystem Biosystems, talking about terabytes of information needed a thumb drive or a drive, external drive with terabyte. Man, that, wow, so much data. How are we going to manage that? And that's the thing. It's a bit overloaded today, and if you take that terabyte and try and upload it, we're talking about some of this cloud technology, it takes a while to do that. And so we've got some things to still overcome. We think about rural coverage of wireless and even cellular connectivity. I mean, once that happens, I think this will even accelerate. But you know, this is the ability that, like I said, we've talked about this. I mean, as people were connected almost 24-7, these machines are now connected, and it's just a matter of time where we start to see more sensors, moisture, temperature, and similar type devices that are going to be in the field connected in the field. But I think the thing is, is apps and kind of thinking, if you're going to serve this industry, whether it's here in the US, South America, Europe, or wherever, I mean, there's some countries today in Africa that we've worked with that they don't have all the nice machines and things, but you know what they have? They have smartphones, and they have cellular connectivity, and they're taking advantage of apps. And so this thing's worldwide. And so, but let's just explore it. This is how I kind of based upon a report out of Iowa a couple of years ago kind of pulled this out. And this is how I kind of begin to explain what digital lag is. Let's just step through and talk about this. And so I see it as kind of four buckets. And we talk a lot about big data, but you know, in reality, when we look at some of the things in the medical profession, the retail profession, they're utilizing big data, but big data is really not here in Ag yet. We just are not aggregated. And we'll talk a little bit about that in a second, but we'll start with precision agriculture. We've been at that for a long time, Mark. We've been at that since, you know, the early-mid-90s in a lot of areas of the country. But precision ag, the other thing that's probably the fastest growing area in this country is prescriptive agriculture. If you go and you listen to the precision seeding, prescriptive seeding, we touch on that a little bit. It's really utilizing the data to be informed or make recommendations. The other third part of the lag, and you're seeing companies provide this, is really enterprise agriculture. And that's really bringing the business. And in that case, I think about it as not enterprise level, but it's field by field understanding. It's management of logistics of your machines, your grain, and bringing all that in. And an important ingredient to that is it's being informed by the precision ag technology, pulling data from it to do a better job of characterizing fuel and things of that nature at a true cost on a field by field basis. And the prescriptive agriculture. This is what I'm doing now, and the feeding that in. And so it's kind of a field, what I would say is it's business, but it's really getting down to the detail of field by field. What kind of money am I making? And then, as I mentioned, big data is where we're heading. There's promise. We'll talk lightly on that in a little bit. But those are the four buckets. As I explained, that when you put them all together, that's the world of digital agriculture. Kind of to bring some definition. To me, it's precision agriculture is quite simply this technology that we've talked about in many of our sessions, Mark, that just the technology itself, we've been at this for nearly 20 years of bringing technology to the farm. Now it's embedded in these machines. Farmers are used to doing it. They're comfortable with it. And whether I buy or release a new machine, it's already there. It's readily available for many years, just like we talk about air conditioning in a car. You know, I don't check that box anymore. It just comes. Yes. And then the other thing is really basic site specific services. And as an example, it's going out in either grid or zone sampling and then putting a recommendation of that. That's very basic. I measure and I react or I determine how much. And so that's precision agriculture, Mark, in just very simple terms. Today we talk about this. This is in the North America market. And I think even in Australia and Europe, you're seeing growth, but this prescriptive agriculture. But that's where we're really utilizing data, collecting it to drive information, drive recommendations. A lot of times we think about prescriptions, but also being able to make those site specific decisions. And so really it's getting down to that subfield level of being able to manage variability. And I would say a grower in a lot of areas of the country today are probably sharing data with three people in their area. They've got their seed person. They've got their retail or cooperatives that's providing potentially fertilizer. They could have their dealership. And they could have their agronomists. I mean, there's easily three or four people that are beginning and are taking that data and are generating prescriptions or some kind of information from it, utilizing it. And that's what I would call prescriptive agriculture today. Kind of the third. And this is just as an example, in the United States it's very common that folks that are implementing verbrate seeding in the prescriptive world, verbrate nitrogens becoming popular, spacing in corn. And then adapting nutrient management. Maybe as an example, as I'm taking my yield maps and generating removal maps, I'm coupling that with my precision sampling schemes and looking at that. And so all of a sudden I'm using the data and I'm refining my fertility application program at a site specific level.  There's many more out there, fungicides and et cetera that we see. But those are, to me, those are the three main ones that you see folks are really starting to utilize out there as relates to prescriptive agriculture. The other thing in enterprise agriculture, we talk about ERP and again, this is kind of the business field level management at the farm. Bringing that to the forefront, I can look at things like not only cost and if I add, you know, if I mix up my crop rotation, I can model that. I can look at scenarios and decide what I want to do.  I mean, these are things that I see again that we can do. But this is a heavily growing field and at least in the US you're starting to see larger farmers take advantage of this for them to better manage their bottom line. And so that's kind of the enterprise agriculture. And then the thing that gets a lot of attention, especially since 2012 is this thing about big data, Mark. We've got a lot of data. In fact, you were saying, well, how much data are we generating? Well, I can tell you, based on some of our projections, today a corn grower can contribute a half kilobyte of data per corn plant per year. Wow. You say kilobyte, but well, how many plants are out there? 42,000 an acre. You know, or 35,000. You know, and I'm doing 2,000 acres. You know, you're talking about thumb drives earlier and about the size. Well, you take a two-gig thumb drive, I can fill her up real quickly, just annually. And so, but big data is this idea that where it's just not my farm, it's aggregation of data and then beginning to query and learn from it in different ways than never been. If you can pull it up in a spreadsheet like Excel, that's not big data. Using some of our traditional databases, that's not big data. This requires all new kinds of cloud technology and such. But big data, as an example, the retail sector, you know, uses it to see trends and the likes of Google, the medical profession, beginning to do it, but it brings a whole higher order of learning and value potentially, but we're not quite there today. I thought this would be one good, you know, kind of answer to that is that officially when we get to the big data, you know, you think about when we work on projects, Mark, I send you the data, right? Hey, you give me the summary and the analysis and send it back. Well, once the data gets aggregated and it gets so big, you're gonna have to bring those analytic pieces to the data because you just can't efficiently and effectively transmit that anymore. And so we're gonna have to take the analytical tools to the data versus what we've traditionally done is taking the data to the, you know, loading it in Excel or loading it into R, SAS, or whatever we were, GIS. So that's a good way to measure what the difference is. John, do you see artificial intelligence coming in and looking at all this big data and be able to say that that's problem A and that's problem B and that's this and get that back to the farmer because it's just so much data. How's anybody or how are we gonna deal with it? Yeah, absolutely. And I think you're seeing companies trying to do that, but the requirement there is you gotta have an aggregated data set in order to explore and begin to learn. And so it's gonna take time. I think we see those tools being built today by companies, we call it machine learning. Yeah. Those are the processes, the analytical pieces that are being implemented on it. But we're still, I think, trying to get it to a level and learn how to really draw value out of some of these data sets once they are built. So that's, you know, the four components as I see it, the digital ag. And so that kind of brings us to this, you know, again, I think about this. So now farmers, as they look at it, they're using all this technology. They could be using like the telemetry pieces technology. And now all of a sudden I got all this data. So where am I gonna store it? How am I gonna share it? And if I am sharing and I'm doing it in a big data scheme, how do I remain anonymous if I don't want others? You know, I'm willing to share and I'm willing to learn from it. You know, these are now the questions.  You need to, you know, where am I gonna store it? We're talking about using like Dropbox and Box and you know, there's all these cloud storage, but you know, we've got companies offering their own platforms, but you know, that's something that you gotta kind of sit down and think about today. Even as an individual, right? As a consumer out there. So, but that kind of, you know, as the first part of this discussion comes in, I think we've kind of covered at least as I define it what digital agriculture and now we're starting to move in to all these issues around data today. This is exciting stuff, John. Thank you so much for sharing with us and please watch all our Precision Ag videos and we're gonna have another one on dealing with data. So watch that. Thank you.


Hi team and welcome. I'm Jeff Walser, the Vice President of Exploratory Science here at IBM Research, and I'm really happy today to start with going to be a series of periodic talks we're going to be doing over the next several months. Hopefully many of you got to join our first Distinguished Speakers series with Yoshio Benjio a few weeks ago, and of course we'll continue to bring in distinguished speakers from outside. But as IBM researchers, we actually have the pleasure each day of working and collaborating with an amazing array of the world's most brilliant minds right here in the labs. Some of our colleagues have been recognized as very high honors by the external scientific community for making foundational advances which have transformed a technical area or an entire industry, or even for starting an entirely new field of research. In some cases we may work with these individuals for years without fully coming to appreciate the contributions that they've actually made to science and IBM. So these stories are an inspiring example internally, but of imminent careers of research and stories which we can all learn from. Therefore on occasion we'll be inviting distinguished members of our community to tell their story and what we're calling an IBM Research Special Seminar, describing their notable technical accomplishments reflecting on their career path and sharing the inspiration that made these possible. So today I'm excited to welcome Ron Fagan, IBM Fellow at the Almond Research Center. Ron is recognized in the computer science community as a founder of relational database theory and the creator of the field of finite model theory. He's also authored several work in the field of information integration and aggregation and of reasoning about knowledge. His work advanced both the theory and practice of modern computing systems, especially data management systems. Ron's key inventions used in IBM products include extendable hashing, differential data backup, and critical tools for database design. Ron has earned a long list of distinctions through his 47-year career at IBM Research. He is an ACM Fellow, an IEEE Life Fellow, and a Fellow of the American Association for the Advancement of Science. The work Ron has published with his colleagues has been awarded the 2014 Goodwill Prize and the 2020 Alonzo Church Award, some of the world's highest honors for research in theoretical computer science and logic. And he's a member of three prestigious national academies, National Academy of Engineering, American Academy of Arts and Sciences, and most recently the National Academy of Sciences. So that's a three-fer for the National Academy's forum. Ron is also known among his co-workers, anyone in Almond in particular, for his unfailing cheer and kindness. He's always a favorite with the interns who are in town. When we used to have all hands in the auditorium, he has a special seat right in the front where he could always ask the very first question after every talk. He's always open to answering questions as well. He's known as well for his remarkably quick wit and his remarkably slow driving speed as anyone can attest to the propulsion of the Almond and Hill. But today, Ron will share some perspectives on applying theory to practice and practice to theory, gained through the course of his notable career. In this talk, Ron will describe two case studies from his career, in which the mutual benefit close parallel aberration between theoreticians and practitioners, combined to propel significant scientific and technological advances. I'd also like to introduce Alex Gray, the VP of Foundations of AI at the IBM TJ Watson Research Lab. He'll be my co-host. Alex will be moderating the question and answer period at the end of Ron's seminar. Please use the Q&A panel in your console to submit any questions you have throughout today's session, and then we'll get to them at the end. So with that, I know we'll further ado. Ron, welcome, and thanks for sharing your experiences and inspiring story with us today. Great. Well, thank you so much, Jeff. Appreciate it. I was not ready for the driving part, but that's interesting to hear. Anyway, so hello, everybody. So, welcome to my talk. Now, the purpose of today's talk is to, a purpose is to encourage collaboration between theoreticians and system builders. I call them practitioners, but Laura Ha's objected. She told me, call them system builders. It sounds better. So system builders it is. And I'm going to do this, as Jeff mentioned, with two case studies. One of these case studies was initiated by the theoreticians and the other by the practitioners. Now, there's several messages here for theory people and one message is how to apply theory to practice and why this leads to better theory. And for practitioners or system builders, the value of theory and the value of involving theoreticians in your work. So in the first case study started way back in 1996, garlic, and Laura Ha's came up to me. Now, Laura Ha's at the time was just a first level manager, but later on she became an IBM fellow. She became the director of computer science. But she knocked on my door one day way back in 1996 and said, okay, Mr. Database Theoretician, we've got a problem with garlic, our multimedia database system. Now, what was Laura's problem? Well, here's what it was. So garlic is a middleware system. It's on top of other systems. So it's on top of things like DB2, which is a relational database system, on top of cubic, which is queried by image content where you can query on the basis of color, shape or texture, and top of other stuff too. So, so now the problem we're ahead was that the answers were not compatible, which is getting the answers and DB2 were sets. And we'll see an example in a second, whereas the answers in cubic are sort of sort of like what you get in Google, you know, you get a list of answers, you don't get a set. So, so now, so let's look at an example. So let's say we have the CD database, and you're searching for artist equals the Beatles, say DB2.  Now this is from MusicBrainz, which has 12 million CDs in its collection. Now, so let's say we have the query album color equals red. Well, you get a certain list. And here's your history should get back from MusicBrainz. The reddest thing first, I guess counting red pixels and with a redness score there. And so now, how would you make sense of a query like artist equals Beatles and album color equals red? Well, you'll say, come on, Ron, big deal. We'll just have a list of albums by the Beatles, sort of by how red they are. What could be easier than that? Okay, smart person. What about artist equals Beatles or album color equals red? Not at all clear what you do, is it? And how about, let's say we're doing this multimedia query color is red and shape is round. How would you make sense of that? How would you give an answer to that? So that's what I dealt with. So what was my solution? Well, first of all, it's often the case that when a practitioner asks a theoretician a problem, they give out something important. And what Laura left out was that these weren't just certain lists, they were scored lists. You saw with the redness thing, there was a score there. I thought, aha, scores. And so I thought, gee, now we're in there, we can match these things up. Sets are scored lists also. Zero if it's not in, one if it's in. So they're both scored lists. So I can combine these scored lists. Wonderful. And that reminded me of real valued logic. Some flavors of that are called fuzzy logic where things have scores, not necessarily zero or one, but in between zero and one. And you deal with that. Now in real value logic, conjunction is often taken to be the man. This is what was the original motivation in doing fuzzy logic by Zada. He took conjunction with the man and disjunction with the max. The people use other rules and other real value logics. So then I said, great. I went back to Laura and said, Laura, good news. I've got a solution for you. Use real value logic. I was happy. I thought I was done. And Laura said, well, hmm, she says, you know, I like your solution. She said, but you know, we need a practical algorithm. We can't just look at every object in the database, get it, say, ready to score and round the score if that's what you're looking for. It scores in every attribute. And use real value logic. We just can't afford that run. We need an efficient algorithm. So I thought, hmm, so I scratched my head, came back to her a couple of days later and said, Laura, good news. I've got an efficient algorithm instead of winning your time. And we're ends in number of items in the database where you have to look at everything. It's squared event. So you can use squared event accesses and get the top K. So that's great. I was very happy. But Laura said, and I'll never forget her answer. She said, good. She said, certainly better than linear. She said, but you know, we database people are spoiled. We're used to log in things like B trees. So she said to me, and I really will never forget it. Be smarter. Go get me a log in algorithm. So I thought, hmm, so scratch my head, came back to her a couple of days later and said, guess what, Laura? I can prove you can't do any better than squared event. That's it. No better algorithm. She said, fine. Okay, we'll take it. And they implemented it in garlic. Now the, let's just see the difference between n and squared event time for the accesses. Let's go back to music brains with this 12 million CDs. So now if you've seen the thousand accesses per second and you know, searching every item of the end items, the database would take in the naive algorithm and take about three hours squared event, about three seconds. So it's a huge way to do square event versus add. So generalizing the algorithm. Well, you know, we see your dishes like to generalize things. So the algorithm would work not just for men or anything like that, but any monotone scoring function, one where if you increase the scores, you can't decrease the overall answer. So influence. Well, first of all, the algorithm was implemented in garlic. And it's influence a number of IBM products, including ones on this list. Watson, a bundle search system, infrastructure, Federation server, Webster commerce and others. So it influence a number of IBM products. And the paper that interests the algorithm, which by the way is now called Vegas algorithm has about 2000 citations, according to Microsoft academic. By the way, I use Microsoft academic instead of Eagle scholar because it gives more and gives a bigger number. They search more stuff. So, you know, why not give the bigger number? Okay. Okay. See a comment on the screen. I can't read it. Okay. So now. So now something happened though in 2001. So I'm not a lot of money. Now our and I came up with a new algorithm, which we call the threshold algorithm. And just to just tell you the algorithm. I'll tell you what the problem is. Just be more concrete. So there in other M attributes, let's say a redness and roundness will take M to be two in our example. And every object has a score for each of these attributes. It's got a redness score and a roundness score, for example, in our example. And now.  And now our goal is to find the top K objects according to some monotone scoring function while minimizing database accesses. That's our goal. That's, that's the problem. So here's an example. Here's a table with redness and roundness. So object 177 is the very reddest objects. It's got a score of .993. As you see in the upper left, in the bottom right, you see it's not so round. It's got a score of .406. So it's a little bit round. In the upper right corner, you see object 235 with a roundness of .999. That's probably a circle. But in the bottom left, you see it's not so not so red. There's its redness score. So, so now. So now let's talk about scoring functions. So let F be a scoring function. So as I said, the one commonly used, like Zata used in fuzzy logic was min. There's many others you can use, one of which is average. It turns out that in real life they've done psychological studies on people. And when you use average and give the answer with the highest average, people like the answer better. So average is a good, good choice. So let X1 through Xm be the scores of object R. It's redness score and it's roundness score, in our example. And then F of X1 through Xm, you apply your scoring function F, is the overall score. We may write that as F of R. So that's the score of object R in this, under this function F. And so now, now here's a formal definition of monotone. A function F is monotone. If whenever the XIs are less than or equal to the YIs, then F of the XIs is less than or equal to F of the YIs. So modes of access. Well, a cube is limited in its access. It has two flavors of access, sorted access, where you say give me the reddest thing, the next reddest, the third reddest, and so on. That's sorted access. And also random access. Maybe you just saw some object with a high redness score. You say, what is this roundness score? So you can go ask, what's the roundness? You can go ask for that. And now our goal is, I said, to minimize the number of accesses to the database. So we want an algorithm to find the top K object. K is 10, for example. Now, the naive algorithm goes and looks at absolutely everything, finds the top K, and gives the answer back. But that's way too expensive where it made very clear to me. So we wouldn't do better than that. So here's the algorithm that we came up with. And then look at Moni and I, the threshold algorithm. So how does it work? Well, it does sort of access in parallel to each of the lists. So it does a sorted access to the redness list and to the roundness list. And then as soon as you see an object under sorted access, in one of the lists, you go get a score in the other list. So if you get an object appeared in the redness list high at the list, you go get its roundness score. And then you can compute its overall score, because you have all of its values and all of its attributes. So you get its overall score. And then if this is one of the top K objects you've seen, great. Remember it. This is a candidate for the final top K. It's one of the top K so far. And you throw away anything that it replaces. So you have your current top K list. And now we need a stopping rule. When do we stop? Well, for each of these objects, you see, for each of these lists, I let T sub I be the last object you saw under sorted access. In the redness list and the roundness list, you have those, the scores. And then you define the threshold T to take your scoring function F and apply it to those thresholds. Now, that is not the score of any object, but still it makes mathematical sense. You can apply F to these numbers. And we'll call that the threshold T. And here's our stopping rule. As soon as you've seen K objects whose overall score is at least T, stop and output your current top K list. That's the algorithm. So let's see why it's good. First, let's see an example. So here we go. So here's the redness list and the roundness list, object 17. We got the top red, the redness object and the roundness object. And now for the top red object, you get a score in the roundness list down there at that bottom right. And for the top object you've seen so far in the roundness list, you go get its redness score. And then you compute their overall scores by applying your scoring function F to them. You get their overall scores. So we're saying min for simplicity in our example. So object 177 is the min of 0.993 and 0.406. So 0.406, so that's its score. And this is the threshold so far. You take the scoring function F, which is the min applied to the last things you saw in started access. So it's the min of these two numbers, 0.993, 0.999, which is 0.999. And you keep on going. And you keep doing that process. And you'll notice the threshold keeps dropping. Now it's the min of these two smaller numbers. You keep going. And now it's the min of these two smaller numbers. So the threshold keeps on dropping. And you're watching carefully to see if you have k objects whose score is at least that threshold. So now let's say, why is this halting real correct? Why is the threshold not successfully stopping then and not making a mistake? Well, suppose you have k objects so far whose score is at least t the threshold. What could go wrong? Well, what could go wrong is you have some object r you have not seen yet somewhere deep in one of these lists. You have some object s in your top k. But the score of r is bigger than the score of s.  So let's show this bad thing cannot happen. You cannot screw up like that. Well, let's say r has scores x1 through xm. It's got a written score and around the score. So now, since you haven't seen object r yet, the xi is less than or equal to ti. You haven't seen it yet. So in each of the lists, its score is less than that threshold value ti for that list. So now we're going to apply monotonicity. The score of object r is f of these xi's, which by monotonicity is less than or equal to f of the ti's. Well, my definition, f of the ti's is the threshold t. And now we know that the score of s is at least t because that was in our final stopping list. So we stopped when this object, we had k objects whose score was at least t. So, but now you see there's a contradiction. Up above we have f of r is bigger than f of s. And down here we have f of r less than or equal to f of s. Contradiction, so this bad thing cannot happen. So indeed the stopping rule is indeed correct. So now we found this wonderful notion in working on the threshold algorithm, something we call instance optimality, a wonderful magical property of an algorithm. So we tell you what it is. So let's say A is the class of algorithms, legal algorithms for a problem. D is the class of legal inputs. I'm thinking of these as databases. I may call them databases. Now for each algorithm A in the set of A algorithms and D in the set of inputs or databases, it's got a cost of applying algorithm A to input D. And this cost may be the number of accesses. It may be time. It may be whatever. We have some scoring, some cost. And now here's what it means to be instance optimal. So this algorithm A in bold A is instance optimal over bold A and bold D. The set of possible algorithms and possible databases. If there's some constant C1 and C2, such that if the adversary picks an algorithm A prime and the adversary picks a database D, that's his choice, the cost of running your algorithm on the adversary's database is less than equal to C1 times the cost of the adversary running his algorithm on his database plus C2. I mean, that's pretty heavy duty because the adversary can fine tune his algorithm to his database. He can fine tune his database to his algorithm. A pretty brutal condition to have to be able to do that. And we'll refer this number C1 as the optimality ratio, this multiplicative factor. So now I'm claiming the threshold algorithm is instance optimal. So now you might think that follows because you might say, well, the next object you see might have score equal to the threshold values. So this is why you can't do any better, but unfortunately that doesn't quite go through. Wife is a little more delicate. So we need another notion that I'll tell you about that we're going to forbid, something called wild guesses. So wild guesses is you ask for a score of an object you've never seen before under random access. So not something you've seen under sorted access, but for some reason just out of thin air you say, you know, I'm out of the blue, I've never seen object 393, but tell me it's redness score. That's a wild guess. You have no good reason to do it. It's not appeared in any of the lists. That's a wild guess. Now neither thing is algorithm or the threshold algorithm make these wild guesses. And in fact, your system may not even allow wild guesses. So now here's the theorem for every monotone F, A be a class of algorithms that are legal. They always give the right answer for every input and it does not make any wild guesses. Then and D is a class of all inputs or databases. Then the threshold algorithm is is this optimal over these. So we actually a theorem proving it is is this optimal under this assumptions. The only tricky assumption is no wild guesses.  Oops, let me back up a second. Sorry. So sorry about that. So now the late David Johnson came to my office one day and I was very proudly told in this theorem about this is optimality. And he said, you know, Ron, he said, I would have a stoner notion of this is optimality, not only the stuff you say, but it's got to have the best possible optimality ratio that constant C1. I thought, oh man, that's an interesting hard problem. So I went back to my office or the next day and I found the optimality ratio. It's this complicated thing. M is a number of lists, like two of its radius and roundness. C sub r is the cost of random access, C sub s is the cost of asserted access. This is the optimality ratio for the threshold algorithm and I proved it's best possible. So it's among all its optimal algorithms. So it's it's optimal in that strong David Johnson sense. So then, great. I went to Laura said, Laura, new algorithm, threshold algorithm, even better than thinking it's algorithm, because it's optimal in its stronger sense. It's not a worst case sense, the usual method, or average case, but in every case, it's optimal at every instance. And Laura said, Ron, we implemented your algorithm all over the place.  What's going on? I said, well, Laura, there's optimal and then there's optimal. So that was the way it was. Influence, well, so we submitted this paper to PODS 2011. It's the top database theory conference. And I was really worried. You saw the threshold algorithm. It takes about 11 lines to write it down. And I thought, man, oh, man, the program committee is going to look at this and say, are you kidding me? We're the top database theory conference and we're going to accept a paper with an 11 line algorithm. Forget it, reject. So I thought, how am I going to get around this? So I thought, aha. So what I did is in both the abstract and the introduction, I called it a remarkably simple algorithm. Turning a bug into a feature. That's something good to remember if you're trying to get a paper accepted in the conference. Turn the bug into a feature. A remarkably simple algorithm. And the paper not always accepted, but it won the best paper award. So it really, really worked. The paper was very influential. It's got 3,800 citations. It won the Test of Time award 10 years after the conference. And I won the IEEE Technical Achievement Award for figures, algorithm, and threshold algorithms together. And then the biggie, the Gertel Prize. This is the highest prize for paper at theoretical computer science. And this paper won the Gertel Prize. You can't do any better than that for paper at theoretical computer science. And I've mentioned that there's something called Jim's of Pods. They looked back over the last 20 or 30 years and said, what are the best papers that have ever appeared in Pods? Let's have it in 2016, but the very first time they did it. And this is one of the two papers selected. And they totally by there was a top rate among those two for Jim's of Pods. So it's got a gazillion applications. I'm certainly not going to read this to you, but one reason why it's been so influential and one reason around the girl prize is people use it all over the place in many, many, many different applications. Measures of success. Well, making our products better. That's certainly a ultimate measure of success for a practitioner. Creating a new subfield that is an ultimate measure of success for a theoretician. And so we did both of these. And if I want to make an argument to a theoretician, you will do a better theory if you just talk to those practitioners. I cannot make a stronger argument than this. Here's a paper that arose from a very practical real world problem and by golly it won the girl prize. What more could you ask? I mean, that's strongest argument I could get to a theoretician. Talk to those those messy old practitioners. Second case study. Clio. So, now Clio deals with data exchange. I'll explain in a moment what that means. Where you convert data from one format to another. Now, Laura has left garlic and after it's successful and implemented and started Clio. And I followed her. She was my manager. I was in the theory group, you know, I had a wonderful success with Laura before. I will start going to Laura's meetings and I actually sat in Clio meetings for a full year. And then something happened. So four of us, Vicki and Colitis, Renee Miller, Uchin Popa and I said, look, let's do how we do data change from scratch. If we didn't have any, you know, thing that all this worth has gone into Clio, just start from scratch what we do. By the way, Renee Miller is very sensitive about data change. She doesn't allow her picture. She will not allow her picture to be public. So that's a picture Renee Miller aged three. So she allowed me to use that. So she was not a child prodigy. That's the picture she allowed me to use. So data exchange. So you have a source database and a target and you're trying to convert data from one format to another. I'll give you an example in a second. So here's the example. One of them has an employee manager relationship and the other has an employee department department manager relationship and we want to merge these two. So we want to somehow combine this. So here's what we're going to use data exchange. What do we do? So now there's something called double generating dependencies or TGDs which has been used in the past for a number of contexts, including normalization and this is what we're going to use to describe it. As I said, it's originally used for other senses. So here's the double generating dependency or TGD we'd use in this case. We say if a double EM is an employee manager relationship. So employee manager M. If there's some department D where employee E is in department D and department D is managed by manager M. So now here's an example. We have this database here. Gertl is reporting to Hilbert Turing and Hilbert is reporting to Gauss. Now I'm going to give you three possible databases that you can convert this to and you're going to think about which one you want. Now if I were doing this in person which I might give this talk in person I'd make people vote. I say here's the three answers how many vote for number one, number two and number three and people don't do very well on that but here you have to privately vote yourself and see how you did. Okay so here's solution number one. So we name the department after the manager we say Gertl is in the Hilbert department which is managed by Hilbert. Turing is in the Hilbert department, managed by Hilbert and Hilbert is in the Gauss department, managed by Gauss. So that's solution number one. Solution number two you create these dummy values D1, D2 and D3 and you say okay Gertl and Turing are in department D1 which is managed by Hilbert and Hilbert is in department D2 managed by Gauss. So that's solution number two. So solution number three you say well wait a second just because Hilbert is a manager of both Gertl and Turing they may be in different departments maybe this guy's managing several departments so we have D1, D2 and D3 Gertl is in D1 Turing is in D2 Hilbert is in D3 and D1 and D2 are managed by Hilbert and D3 is managed by Gauss. So decide for yourself what your solution is. Would you take number one number two or number three? So we have good reason for which one we choose. We have this notion that we call a universal solution. A universal solution is something that is as general as possible makes as few assumptions as possible. In this case solution number three is the universal solution for example it's not assuming that each manager manages only one department. Now if we had this other restraint that says here that each manager manages at most one department then in that case we would say solution two was the universal solution. Okay now how do we obtain a universal solution? Well there's this well known method called the chase. It's been used for many many years in database design and used in normalization theory in other places and so we're going to use the chase to create the universal solution. So how does it work? So let's go back to our TGD TumbleToonerIndependency that says employee manager manager relationship with department D where employees in department D and D is managed by M. So what we do now is we take each tuple in that first relation like Gerdo Hilbert and now we create this W value D this so-called label null and say Gerdo's department D and D is managed by Hilbert and these are called label nulls so that's what we do and then if there happened to be the EGD is like each manager manages at most one department you equate some of these label nulls you equate both the label nulls dealing with the same manager. So now something else happened when you have a problem many problems arise afterwards so one day we should public came to my office and said Ron we got another real real problem with these mappings. He said we use these TGDs to go from schema one to schema two and we have these TGDs going from schema two to schema three. How do we describe how you go from one to three directly so my first thought was well there's going to be another TGD it's more complicated than you know complicated but still some other TGD will describe directly how to go from one to three. Now let me remark that TGDs are very special cases of formulas of first order logic. If you know the first order logic and you saw it from running down to use his first order quantifiers for every X and stuff like that so this but here's what we discovered. So Fouquien, Colitis, Lucian, Popo, Wang, Chutin and I studied these things and we found the following it's a very surprising thing when you compose these things you leave first order logic. So to use it in this simple TGD a special case of first order logic from schema one to schema two uses very simple TGD from two to three but going directly from one to three you need second order logic where you quantify over sets and over relations and over functions. That was quite a surprise. And so we invented something we called second order TGDs which is a very special case of second order logic and we showed that this exactly does a job that you compose schema mappings with TGDs and given any second order TGD it's resulted in composing first order TGDs normal TGDs and so so now we gave an algorithm for doing composition for computing these so measures of success well our first of all our stuff is used in these various IBM products that I've listed here DB2 control center rational data architect and control manager by use I mean the following so first of all they use universal solutions they decided that's a good answer the result we're going to take will be a universal solution people weren't doing that before even at Alamedin when they were working this originally they were doing kind of crazy solution sorry and but they weren't universal so they not only used our our universal solutions but they used our algorithms they used our algorithms for producing universal solution and they used our algorithm for composition so it was really heavily used our stuff now our paper our initial paper won the test of time award for the international conference on database theory that's like the second best database theory conference that's when we submitted that paper to and now something funny happened I got a letter one day from the editor chief of the journal theoretical computer science which is where the journal version appeared he said congratulations your paper was the second most highly cited paper of the decade of papers that appeared in theoretical computer science so great but I just out of curiosity I sent him an email saying okay what was first he said well it was a survey paper we felt a little bit bad about giving it the top number one but we had to it was in the journal so what could we do so that was the top rated paper now our paper on composition won the test of time award for international conference on database theory ten years after it appeared and I wrote a follow up paper with two other people Marcello Arenas and Alan Nash that won the best paper award for another international conference on database theory this was a follow up paper on composition that showed some surprising things that like a second order TGD can be taken by composing only two first order TGDs you didn't need some large number of them but anyway won the best paper award influence well this created a new subfield you know I told you that's a great measure of success for theoreticians in fact every major database conference suddenly had sessions on data exchange and now here's something that Jeff mentioned that made us very happy that we just won this award the Alonzo church award which is given for outstanding work in logic and computation they can look at a paper a body of papers they look over a 25 year period and see what's had the most influence and we won by the way I was told that in these committees that normally people look at things near the borderline like if it's a 25 year period they get paper that's about to expire because they want to give that paper a chance so ours was way less than like half of that roughly and yet we still won which is really great so before I go on to very recent work I was told to put in a slide here on very recent stuff so I did that but first I want to say one more comment and that's the following so one important issue is to figure out what the real problem is that theoreticians have to do, practitioners have to do what is the real problem so I want to read to you a quote from my academic grandfather Albert Einstein pretty impressive academic grandfather and in the Q&A I'm hoping someone will ask me Ron how was your academic grandfather and I will answer that question but let me read to you Einstein's quote this shows by figuring out what the problem is is important Einstein said if I had an hour to solve a problem and my life depended on the answer I would spend the first 55 minutes figuring out the proper questions to ask for if I knew the proper questions I could solve the problem in less than five minutes that's what Einstein said what more do you want this is a quote from the great man Albert Einstein himself okay so now recent work so I was told to add to this one more slide about what I'm doing very recently and I'm very happy to do that very recent work, very very recent like in the last few months so I'm working with Alex Gray department on Neurosynbolic computation and one of the important issues that arise there is real valued logic because you have uncertainty so you have inputs and neurons and they have scores between 0 and 1 it's like a measure uncertainty or probability these things we wanted a logic for dealing with these for dealing with this combining how much does this uncertainty affect that uncertainty now it turned out that what people have done in the past was the following a very weak form of axiomization they found a bunch of things that were tautologies in their logic they were always true in their logic no matter what truth values you assign to these things and from that they could prove everything else that was a tautology that was proved not good enough for us this uncertainty implies that uncertainty and I did that very very first example soundly complete means everything that proves is correct and anything that something is an implication of other things you can prove it in your logic so I gave the very first one that shows this amount of uncertainty here that amount of uncertainty there how much uncertainty does it yield here by the way I read this by some experts in the field experts in real valued logic that says has anyone ever done this no no no it's the very first time anyone's ever done that so I was very very happy the very first soundly maximization that deals with complete this uncertainty implies that uncertainty so that is my recent stuff so that's it and we will be in an archive paper we're still having to put the final version in an archive but if anyone wants a copy I'm thrilled to send you a copy of it and any questions of course I'd be happy to answer either now or later on so that's it that's my talk thank you very much


In Lake Ontario, a few years back, we had blue-green algae warnings at all the beaches. You can't go in the water because it's toxic to humans. We had blue-green algae warning. Yes, in Lake Ontario, and also Lake Huron, and also in Trent Suburban waterways. Pigeon Lake, Buckhorn Lake, and Shemong Lake, there was a no swimming warning saying, you cannot go in the water. This blue-green algae is toxic, and that's due to overloading of your nutrients from runoff. So if you're getting a lot of water coming in from everywhere else, and also a big factor on this, too, is old septic systems that are on the lakes. So, for example- Old septic systems that are on the lakes. So people who have the cottage, they got a septic tank that's been around since 1950. And they're literally, the weeping is right at the lake. The weeping is right at the lake. So what happens over the time is they start to leech into the water, okay? So now you're getting bad bacteria, you're getting fecal bacteria. Well, this is what I'm used to as a kid. I mean, you guys, when you were kids, we used to go down to the beach and swim at the beach, and all of a sudden there's a flag up there that says, no swimming.


Mike Vrabel, can you guess this tight? Yes, I can. Well, brought to you by Del Tadeno. Go ahead then. That would be Cincinnati, Ohio's finest, David Long. David Long? He's two for three, the head coach. How about that, David Long? He's continuing to improve. He's getting opportunities, making the most of them. Just a great kid, you know, it's his passion for football.  And you can tell that he flies around. Triggers, he's playing fast for us.  We've won both of those games. My broadcast partner, Dave McGinnis, coach linebacker, Coach Mack. Coach Mack. He likes him some very long. He likes him some David Long. He says he's become a more complete player in year three. I can't wait to talk to Coach Mack on Friday. It's just to talk about what he thinks is David as a blitzer. You know, they didn't pick him up one time. Now, they might have slowed him down, but he affected the quarterback every time that he blitzed against the running back the other day. And even one lineman, you know what I mean? He went in there and then wrapped around. The quarterback scrambled and threw it away.


Think you should treat yourself. I know you were thinking about it and you were like, I don't know do it do it. Who's gonna stop you?