Thanks, thanks for joining me this afternoon. It's really wonderful to be back in Berneau for DevCon. My name is Will Benton and I work at Red Hat where I lead a team of data scientists and engineers. So that's all you need to know about me. I'd like to know a little bit about you. Who in here considers themselves comfortable with data science or primarily a data scientist or working on machine learning? Okay. Cool. How about people who sort of spend most of their time developing software or as a software engineer? How many people think of themselves as infrastructure experts or SREs? So we have a really great mix of people in the room. The title of this talk is container workflows for data science and machine learning and it's a brief talk. My hope is that it will have something to offer each of you. If you're really comfortable with data science you'll learn about why containers are useful for what you do and some high-level tools that will make your life easier. If you know a lot about containers, you'll learn a lot about how you can integrate data science and machine learning into your environment and really benefit people who work in those spaces and make it easier to collaborate with data scientists and machine learning engineers. So I want to start though by looking at to provide context for people who aren't as comfortable with data science, looking at what data scientists do. Fundamentally, data scientists just use statistics and machine learning to identify patterns in data and solve problems. But let's look at what that looks like with a concrete example. So how many of you are already looking at a web page on your computer or phone? Be honest. Okay. So if you're looking at a web page, are you seeing an ad? Okay. So the company that's showing you those ads is using machine learning somewhere to try and improve the chances that you'll click on those ads so that they can make money and I guess keep showing you ads. Let's say that we're a data scientist and we're working on this problem. We want to decide whether or not to show a particular user an ad for a bicycle. So we'll need to solve a few problems fairly early on, like how to define what succeeding at this problem looks like. Does it mean that we have a model for who's going to click on this ad that's better than another model that's better than random chance? But once we figure out how to evaluate our results, we can continue. And one of the fundamental problems that data scientists face is called feature engineering. This means taking all of the information that you have about a user and encoding it as a point in space, ideally so that similar users correspond to similar points. We call these points in space feature vectors. And really the information that we're encoding could be anything, right? It could be just the content on the page where we're going to show the ad, or we could be surveilling the user across the entire internet and building a profile of them so that we could target the ad to that individual. I hope we're not doing that in this room. But once we've figured out how to encode a user or a page or whatever information we have to target this ad as a point in space, we can continue with the process of trying to understand the data. Ideally, this encoding is going to expose some structure in our data so that when we look at users who are ultimately going to click on an ad, which we have as red points here versus ones who aren't, you can see a clean way to separate them in space. And in this case, we have a group of clicks in a certain area of this space of possibilities. So given a way to map from users to these feature vectors and some historical data about what the users who clicked on an ad look like versus what the users who didn't click on an ad look like, basically a set of examples, we can construct a pipeline that processes the historical user data, generates feature vectors and trains a model, which means using a machine learning algorithm to automatically generate a function that can identify feature vectors that represent users who are likely to click on an ad and then, you know, by extension ones who are not. So given that model, we can apply it to user data and identify which users we should actually show this bicycle ad to. It won't be perfect. It'll identify some users who aren't likely to click on the ad. And of course, not everyone clicks on ads anyway, just by principle, some people don't click on ads. But if the model were perfect, it would mean that we just memorized the data we'd already seen. And we wouldn't have generalized from that. So we want to be able to make predictions about the future, not just the past. I mean, it's great to be able to predict the past accurately, but it doesn't really do us a lot of good. So we want a model that's, you know, we'll trade a model that's perfect for one that generalizes. So we can put all these steps together in a pipeline that winds up looking a lot like a software development pipeline. We start by codifying the problem and identifying the metrics that we care about. We'll move on to collecting data that's relevant, cleaning that data so that we can process it, and then turning it into future vectors. As we go into the process of actually training the model, we may decide that a different feature engineering approach is going to get us better results. So we may go back to earlier steps in this pipeline at any point based on what we discover in another point. Once we've trained and tuned the way that we're actually generating that model, we'll go on and validate it to make sure that it works well with data that it hasn't already seen. And again, this can sort of cause us to reexamine earlier steps. Finally, we'll put this into production as a service that an application depends on, and we'll continually monitor it to make sure that the data that we trained our model on is still representative of data we're seeing in the real world. For example, if we're targeting bicycle ads, something about the bicycle industry might change, causing a person who's likely to not click on an ad at one point to click on an ad in the future. Maybe cycling becomes more popular because of a popular cyclist, or cycling becomes less popular because that popular cyclist is busted for doping. So every part of this workflow can sort of feedback into an earlier part, but the first few phases are sort of special because they are sort of focused on this experimental really understanding the data and identifying patterns. They're sort of before we put things into production, and data scientists often use a special environment for this kind of work. This is an interactive notebook. Maybe some of you have seen these before. Basically, the idea is that we have a literate programming environment where we can combine pros, explanations with code, and we can execute this code independently. We can run functions and see what they return, and then we can go back and sort of edit these cells, modify the code and rerun it. So it's really an interactive way to experiment with things and see how they work. It can be a really productive environment for the sort of exploratory work. You can also get plotting in these notebooks, which is really handy if you're doing numeric work or you're evaluating experiments because you can put some results and visualize them pretty easily. We'll talk more about notebooks a little bit later. Okay. So now let's talk about containers. We talked about data science, and now we're talking about containers. Who here uses Linux containers in their work? Keep your hand up if you're willing to give me a single sentence definition of a Linux container. Just a couple of people. So I want to get everyone on the same page and just make sure we're all talking about the same stuff. So let's quickly cover what Linux containers do and how they do it. If you run a regular Linux process, like installing a library, the Linux kernel is going to maintain some data structures with information about that process, like the binary that you're running, the command line arguments, the environment, and so on. Each process is going to have a set of identifiers for files it has open. Since different processes can open and access different files, this is going to be special to your process. My file number zero is going to be different from your file number zero. Each process also has a private view of a certain portion of physical memory on your computer. Your CPU and your operating system are going to cooperate to give each process a virtual view of this part of memory. So what this means is that if I have a given address in one program, it's not going to point to the same location in memory as that same address in another program. Does that make sense? All right, so this actually turns out to be really important because it means that processes can't interfere with each other's memory because they have no way to refer to it. If you used microcomputers in the 1980s or if you've been to a computer museum, then you're aware of a world where misbehaving programs could crash your whole computer, right? Because every process could access all the physical memory and overwrite memory that another process depended on. But once virtual memory became ubiquitous, personal computing became a lot safer because of improved isolation between processes. If you have no way to refer to the physical memory that another process is using, you can't read or interfere with that other process's state. So containers sort of take this idea of having your own private view of the inside of the process and extend it to things outside of the process. Some of these kernel resources that we can access are namespaced, which means that we can restrict what processes see. The root file system is what your process sees as the slash directory, right? In a conventional process, that's also the root of the whole system, meaning that if you access the root, you're accessing everything that anyone else on the system can see. The process table controls how your view of other things that are running on the machine. And again, in the conventional process, you can see everything else. Finally, network routes control how you can interact with the outside world. Can you connect to anything? Can you use any network interface? Can you accept connections? And so on. And in the conventional process, you get access to everything. When we run a process in a Linux container, we're essentially saying that we want to do something interesting with those namespaces. We want to restrict how the process can interact with the outside world. Maybe you only want to access things within a certain directory or only see a certain set of processes or access certain network routes. You can also impose limits on processes running in containers saying you can't use more than a certain amount of memory or more than a certain percentage of a CPU. So this is sort of low-level kernel functionality to actually use these things to make applications. We need to have some interface built on top of it. And container platforms provide ways to build container images that package up everything we need to run an application. And so we can put it in a directory that doesn't have anything else in it and run that application. So we start with a base image that provides just some very basic functionality, like a C library and a shell, for example. Then we run some configuration recipes to install their dependencies. And then finally we build the application that we want to deploy in that environment and that goes on top. And all of these things are layers that are immutable and you can refer to each layer by itself, which means you can always replay your application at some point in the future. When we combine these containerized processes into an application, we usually use what's called a microservice architecture, where our components are these lightweight services that only communicate via well-defined interfaces. The cool thing about these microservices is that you can scale them up by running them on the same machine or scale them out by running them on different machines and you don't have to change anything because they only use these interfaces to talk to each other. They don't depend on it running on the same machine or having access to the same files.  So it actually set up the configurations that our applications will use. We'll use what's called a declarative configuration. And this means that instead of telling our container platform how we want to set up an application and what we want to make it do, we're just going to say this is what it's going to look like when it's ready to run. These are how these services connect together. This service needs to be able to talk to that service. That service needs to be talking to another one. And we'll let the container platform do the work. This enables creating an application in one place, like on my laptop, and running it anywhere, like on the data center, on someone else's laptop, in the public cloud, wherever you want. And the cool thing is that you can define connections between these different services that are inside your application, but you can also say I want to publish one of these services to the public internet by exposing a route to it. So if I visit, if I visit a URL, I can say that that's going to go to a certain part of my application. So the last thing that container platforms provide that's really interesting from a data science perspective is continuous integration and continuous deployment. How many people have developed an application and given it to someone and it hasn't worked? So yeah, maybe never. I mean more disciplined than I am. But continuous integration means that basically when I'm working on my application, I commit it to source control and there's an automated build in the controlled environment. If my application builds successfully, then I can create a container image and a deployment specification and automatically put that into production. So it's not a matter of it works for me, but it doesn't work for you. It's a matter of it works for the continuous integration system so we know it will work anywhere. And we have this deployment that we can just install somewhere else. So containers are awesome for applications because they provide these sort of primitives that are really useful, but they're also really useful for data science workflows. And in the rest of the talk, we're going to look at why this is the case. But first I want to talk about what we're not going to do. A lot of material on containers for data science starts with the assumption that the only thing missing from a data scientist's life is a bunch of YAML and Docker files. I think actually most people who work in data science don't want to become infrastructure experts. So we're going to focus on high level tools that make these sorts of workflows easier. And a big problem we're going to look at is the problem of reproducible research. So we talked earlier about how data scientists do a lot of work in notebooks. And notebooks are really great for experimenting, but the notebook itself is not a great artifact for sharing or publishing. I mean imagine if you did software development by sending someone a screenshot of your terminal window as you're editing a program. That's basically what you're getting with a notebook. So what we want is instead of this notebook that I might give to two different people and get two different results, or maybe an error, I want something that I can build once and send to anyone and know that I'm going to get the same results everywhere so that this is actually reproducible. So a really easy way to do this is to use a service called mybinder.org. It's a hosted service. You don't actually need to do anything to use it. You could just give it a URL of a Git repo. It builds a container for you and it starts a deployment on Kubernetes that you can use. And this is an open source so you can use the service that someone is hosting out of the goodness of their heart, or you can run it inside your own company. And just by putting a notebook in a Git repo, you can share it with anyone in the world. Binder is great for that reason, but a source to image build actually provides a framework to make services like binder in a flexible way. So basically the idea is we have a builder image that knows how to turn a Git repository of something, whether that's source code, notebooks, or whatever, into a container image that we can run on Kubernetes. And in this example, I'm going to use a builder image that knows how to take a repository of notebooks and turn them into a notebook server container image. So by using this image, I can package up notebooks and make them reproducible without having to become a packaging expert. So let's see that's in action. Do those builds, and here's what it looks like. That's the source image tool. So I can use this S2I command line tool to build images no matter what Kubernetes distribution I'm using. I'm going to specify a source repository, a builder image, optionally a tag, and then I can actually push that, like I would have pushed that image to Quay. But because I'm using OpenShift here on my laptop, what I'm going to do instead is I'm going to use a source to image build strategy, and that'll save us some steps in a quick demo. So I'm going to start this up, and then I'm going to watch the logs as they run by, because I want to make this notebook reproducible. I'm going to start by downloading the entire universe and putting in a container image, and this is going to run for a while, and then we're going to have a notebook that we can run anywhere. Now the cool thing about source to image is that it's not just a way to do something like this. It's a general framework for making things that work like this. So we'll see ways to be more general than just serving a notebook. The other thing I want to call out about this is that the builder image I've used here was developed by Graham Dumpleton, who is a red hatter.  Anyway, so this gets sort of results of publishing a notebook. So we just saw how I could use source to image to create an environment that has similar functionality to a binder service, publish a container, publish a notebook in a container on my laptop and not have to write any code or YAML or Docker files or anything to do it. But source to image is more general than that because we can do anything that involves going from a Git repo to a container or to a service. So a couple of examples I want to call out. The Selden project is an open source project that has an S2I builder that takes serialized Python objects and just turns them into S services. So you can operationalize a model by having an API on top of it. But let's look at a builder that doesn't think a little more involved. Let's say we want to operate on a notebook that trains a model, loading a Git repository containing that notebook, post-processing the notebook to see what libraries it depends on, post-processing the notebooks more to generate a script that trains the model, then running that script in an environment that will have the application ultimately running in, saving the model, and then creating a rest endpoint to serve the model. It's kind of a lot, but I don't want you to get lost during the demo. Everyone following? Okay. So here I'm just going to fire off a build again using the build strategy and this is an S2I builder that you can download from Quay. Basically what we're doing is we're getting a notebook that trains a model. While we run this, we'll look at what the notebook looks like. So I'm going to fire off the logs here and then tab over to examine the notebook. So this is a notebook that you can view on GitHub. Basically the idea is we're going to specify our requirements in a designated variable so that we don't have to do any really exhaustive analysis of the notebook. We just sort of say these are the libraries I'm going to depend on. We have some basic code that trains the model. This is a K-means clustering model. You don't need to know what that is. You just need to know that it takes a two dimensional vector and returns a small integer. And then we have two distinguished functions called predictor and validator. And basically the predictor actually uses the model to make a prediction. The validator verifies that the thing that you've passed into the model is the kind of thing that the model expects. So in this case, we have a two dimensional vector. We want to make sure that it's not a one dimensional vector or a string or something else. So I'm going to use the insomnia rest client to interact with this thing that I've just deployed. I'm going to pass in some arguments, pass in a two dimensional vector and get back a small integer. But wait, there's more. I'm going to pass in a different vector and get back a different small integer. Now, what do you suppose happens if I pass in a three dimensional vector? I get a descriptive error message. So you can use this sort of thing to operationalize models and enforce that people are doing error checking and really sort of go from an exploratory environment to production very easily. So frameworks like source to image may get super easy for developers to publish applications, but they aren't just for applications, they're really for a wide range of things. All of the examples we've seen so far today are focused on developing analytic techniques rather than putting them into production. And to talk about production concerns, let's look at these last two steps of the process and what it looks like to put machine learning into production. We've trained and validated a model on preexisting data, now we need to put that into production and continuously monitor its performance to ensure that it's still working as well as we expect. Predictive models are typically deployed as components of larger applications and we need to manage their life cycles at the same time that we've managed the life cycle of the other components of the application. And these applications are built by cross-functional teams who work on different parts of the problem. We've already spent some time talking about what data scientists do to find and exploit patterns in data and that's an important part of the picture but they depend on the work of data engineers who make streaming and batch data accessible and consumable at scale and of application developers who really tie all of this work together to provide a robust scalable application with a great user experience. Some teams also have machine learning engineers who focus on scaling the machine learning parts of the pipeline themselves and making and model management. So the first community I want to talk about is called RadAnalytics.io which is targeted at application developers and provides tooling and a downstream distribution to make it really easy to develop intelligent applications and containers on OpenShift. One of the cool things that RadAnalytics does is it provides actually an S2I builder that takes a specification for an application and will deploy in the same OpenShift project both the application and a Spark cluster that it can use. So you have a compute cluster that's scoped to your application. When your application goes away the compute cluster goes away too. It's not like you're borrowing cluster time you just use OpenShift and manage all of it in one place. Who here went to Vacek and Stevens talked about the OpenData Hub this morning? Okay, so watch the recording if you didn't raise your hand. This is a cool community project that shows how you can build a platform for data intensive AI workloads in a cloud-native environment with a completely open-source stack. And it sits downstream of some projects like Strimsy which makes stream processing manageable on OpenShift, Ceph which provides scalable object storage and Jupyter Hub which provides an environment for collaborative notebook editing. So you can deploy the OpenData Hub on your own cluster or use it in the public cloud. The final community I want to tell you about is Kubeflow and if you went to Marcel and Pete's workshop yesterday you're already excited about Kubeflow. Kubeflow is targeted at machine learning engineers and provides a wide range of libraries and tools for deploying machine learning workloads on Kubernetes including TensorFlow, Jupyter Hub, Selden, and PyTorch. And this is really a great way to make these reproducible environments. Right? So if you want to deploy a machine learning workload on your laptop and reproduce it in the public cloud or across several clouds it makes it very easy to do these things. So I want to wrap up by just quickly reviewing what we talked about today. We reviewed how Linux containers provide isolation between processes and discussed how container application platforms make it possible to develop applications composed of a bunch of containerized microservices. We learned about a data scientist workflow to provide context for how containers can make this easier. We talked about source to image and some high-level tools built on source to image which provides a flexible way to benefit from the work of packaging experts without having to become one yourself. We saw how source to image is general enough to support many use cases including reproducible research, model serving, and compute cluster deployment. Finally, we learned about three great open source communities that are working in this space to enable intelligent applications, data engineering and machine learning workflows all in containers and all completely open source. Right analytics.io the open data hub and Kubeflow. So I hope you're excited to get started with these projects and learn more about this space. If you scan this QR code with your camera phone if I don't know what your camera phone will do but it may or may not take you to my personal website which has some articles related to these topics and you can find me on either Twitter or GitHub as at will be or email me at willbeatredhead.com I think probably have time for questions offline. Right. There is no talk up next but please don't ask me 30 minutes of questions but I'm happy to take questions. That's I mean that so that's just a flask app with I mean it's it's sort of a running joke in the machine learning community that you're just going to put a flask app around your model but that's that's just what that demo did. You could you could do something more scalable or or more resilient definitely in production but just sort of to show that S2I works for that application and that you can do a little more than people might think you can do with S2I. Thanks Zach. Good question. Yeah absolutely. So I think this is this is a really so the question is how do you monitor a model in production and verify that it's working as you expect and I think this is an interesting problem for a few reasons. Right. One I mean the sort of fundamental reason is in a lot of cases you have no way of knowing the truth. Right. Like you have no way of knowing whether you showed an ad to someone who is likely to click on it because most people don't click on ads. Right. You know so so you have to sort of monitor you probably have to monitor something other than comparing results to the truth but there are a lot of things you can monitor. You can monitor the distribution of the data that you're seeing. You can monitor the distribution of the predictions that you make. Right. And you can say well if these things change over time significantly if these diverge from what I expect you know and there there are metrics for measuring whether distributions diverge then I know that the data that I'm seeing in the real world are no longer similar enough to the data I trained the model on so that I have a problem. Now how can we do this concretely? Well a container application platform like like Kubernetes or OpenShift you can use there's this great support for metrics and monitoring metrics and time series so really it's just a matter of exposing those metrics that you care about like whether it's what predictions you're making from a given model service that you can look at them as a time series or look at the distribution over time or whether it's some way to characterize the data that you're seeing. I don't have a demo of that but that's that's sort of how you would go forward with that with things you already have. Yes. Yeah so that's that's a that's a good question so the question is like with with container images you have reproducible research because you always have an immutable image but what if you're depending on data that's coming in from outside that container image so I mean there are a few solutions to this problem I think there's a really cool open source project called Packarderm which basically provides a versioned data store on top of another distributed file system so that you can say you can refer to something by a hash and know that you're getting a particular version of a particular dataset there's a project called Quilt that's that's a sort of for managing datasets and and versioning datasets. I think in those cases it makes sense to rely on something external and have those as metadata in your container but for small data you can certainly bake it into the container image as well. Right so it's not it's it's in general when you're using randomness so the question is how if you if you start with a random model and you want to sort of optimize the weights in the model so that you get to something you can use so typically when you use randomness in a program you're not actually using a source of true randomness right it's not truly random it's it's a function that has a very long period it's a pseudo random number generator so you start from a given value or set of values and then you can replay a stream of numbers that satisfy certain statistical properties and we we call them random colloquially right so so if you use the same seed you'll get the same results basically yeah and so this is this is actually I mean it's come up because we've had we've had notebooks that we distribute with with output right and if there's some randomness in the notebook we've we've always wanted to putting a seed in because otherwise people are very upset when they get a slightly different answer you know that's then that is in the output that was already in the notebook when they rerun it thank you so much


Hi, my name is Jazz Ozery from the Department of State's Office of Science and Technology Cooperation. Welcome to our Guru Guru series today on the start-up checklist. Our entrepreneurship expert is David Rose, founder and CEO of Gust. Gust operates the world's largest online platform and community for entrepreneurs and early-stage investors. David will be sharing insights from his recently published best-selling book, The Start-up Checklist, 25 Steps Through a Scalable High Growth Business. David has been described by the magazine Crane's New York Business as the father of angel investing in New York. He is an Inc. 500 CEO, a serial entrepreneur, an author, and a keynote speaker who is founded and funded over 100 pioneering companies. He is the managing director of Rose Tech Ventures, founder and chairman emeritus of New York Angels, and founding track chair for finance and entrepreneurship at Singularity University, the Google NASA-sponsored postgraduate program for exponential technologies. If you'd like to join us at any time during the conversation, please use the chat roll box on the right-hand side of your screen and go to JustGuru on our Twitter site. David, thanks so much for being with us this morning. It's a pleasure to have you. It's my pleasure. So I figured I would start this morning by talking a little bit about the book that I've just published that you mentioned. It's called The Start-up Checklist and it's all about how to start a business that's designed to be a large business. Now, all businesses by definition start small. You're one person, you start a company, but there's a difference between a business that starts small and stays small, the typical small business, and a business that starts small and is designed to grow very big. We often call those a startup or a high growth startup. And so this book is designed to tell you how the things you need to keep in mind to start a business that ultimately will be really big. And so in order to do that, we first have to take a quick look at the things that differentiate a small business from a startup.  So that's about as small a business as you can get. But that's a very different business from the kind of businesses that you read about in the blogs or that go public or businesses like Uber and Airbnb and Apple and Google and Facebook. And so what is it that sets the first kind of business apart from the second kind of business? And so let's think of the differences. First of all, if you're starting a lemonade stand or a one person business, that's really great. Nothing wrong with that at all. The vast majority businesses in America are those kinds of very small businesses. But if you're starting a business, this is going to be potentially a very large business, you often might have a co-founder. So it wouldn't necessarily be just you, but it'd be you and a co-founder. Also, because you're going to be doing many things, larger things, you might have employees instead of just you operating or selling lemonade. And then if this is going to be a really large, growing, potentially enormous business, those employees who you hire aren't going to be want to pay not just in salary, but they're going to want to often receive equity apart ownership in the company so they can be incentivized and sharing the rewards of doing that. You will likely have investors, you might have investors if you're lucky, who will provide additional capital that will help your business to grow large fast. That means you're thinking big. Investors are only looking to invest in businesses that potentially can get very, very big. And if you're starting a big company, you have to think big from the very beginning. And that means thinking global. I'm here in New York today. We have people around the world on this call. Every business today has the potential to be global. And increasingly, that means mobile. As you probably know, Google now ranks search returns on their site for sites that are mobile higher than sites that are not. And so many people around the world are now first looking at things through mobile, through their phones. Your business is going to be growing. You're looking not to create a static business, a business that will get to a certain size and stop. If you're going to be a really high growth startup, you want to have a business that's going to keep getting bigger and bigger and bigger. And therefore, your focus as the entrepreneur, as the founder, has to be not on a job for yourself as the president or CEO of the company, but rather on the company itself because you are building an entity. You're building a company that can get very, very big and that ultimately that's where the value will be. And because of that, that means you need to think from the very beginning about what your exit is going to be. It doesn't mean you have to try and sell the business from day one. It does mean that you have to think about who might ultimately either want to buy the business or whether you're going to go public and be listed on a public stock exchange because that's where ultimately the value is realized from these startup businesses. So if you have all of these things that are the differences between the small business and the startup, what does that mean in terms of the way you actually start the company? So let's go through and take a look at a few of these. So first of all, it means that instead of working by yourself, just under the name of a company, you actually have to create an entity, a corporation. Every country has some form of limited liability corporate entity, which separates the business from you. And that corporation will have shares of stock, which means you can sell shares of stock, sell part of the company to investors, you can give part of the company to your employees, and so on and so forth. There will likely be different types of stock in the company. Your lawyer will help you set this up. You as the founder get one type of stock. Investors purchase a different type of stock that often has protective provisions involved. You will have an option plan. A stock option plan gives your employees the right to buy some shares of stock, but instead of giving your employees all their stock at once when they join, that wouldn't make a lot of sense because they might just take the stock and then leave the company. Instead, a stock option plan lets your employees earn that stock over time, typically in the US over four years or so, while they're working for the company. Next, as the company is growing, the relationship between you and your employees has to be set out in detail.   The cap table or capitalization table shows who owns exactly how much of the shares of your company and what their rights are, who gets paid out first and second and so on and so forth. As you get to be a more complicated company with more investors and employees and partners and co-founders, all of whom have owned different parts of the company, it's very important to keep that in mind and keep that straight so you know when the company exits who will benefit. As a global company growing big, you're going to be on the internet, some way, somehow, buying, selling, finding your customer's marketing over there.  This is no longer a cash economy. For a growing business, it's an electronic economy. You've got to keep track of your numbers. You can't just put cash in the drawer. You'll typically have an accounting program often online these days that will let you track all the money coming into, going out of your business, so you can understand what the profits are. You will likely require at some point in this process an attorney, a lawyer, to help you get everything set up. You don't need to spend a fortune on lawyers. They don't need to be there all the way through spending lots and lots of money.  So you want to have the best advice you can get. As an international company, you're going to be growing. You may be located in one place. I'm here in New York City, but my company actually has offices in Sao Paulo, Brazil, in Granada, Spain, and London, England. You may or may not have offices around the world, but you will likely have customers around the world, perhaps even employees in other places. So you're going to be thinking internationally. And then as we discussed, you have to separate the job of you as the CEO or president of the company from the company itself. And so that means when you're looking at the ownership of the company and the control of the board of directors, those are things where you have to differentiate between you as a person and you as your role in the company. And then finally, everything you do has to be tracked absolutely cleanly all the time because when anybody is going to invest in your company or possibly buy it, they're going to go back and check every single decision and document that you did to make sure that everything was actually done accurately, appropriately, and legally the first time. So that's a quick overview of the things that we discussed in the book. This is actually the table of contents for the book, which goes through 25 different steps, 25 different chapters about all the specific things that you need to do if you're going to start a company that's going to start small and get big. David, I'd actually like to start by going into your own unique background and your first impressions about the startup world. How did you become interested in entrepreneurship and in the angel investing generally? I am fortunate enough to, A, have been grown up in New York and the United States, which is a very entrepreneurial culture. But even beyond that, I had a role model and my father is an entrepreneur. And his father and uncle were entrepreneurs. So I am a third generation entrepreneur and saw that it could be done. And so having a role model is really useful. But then I also have internally, I'm an entrepreneur spirit. So I started companies, I started starting companies when I was about 10 years old. My first company was Rose Productions, which produced headshots and programs and brochures for my brother, who was a magician. He would do children's parties, magic shows for children's parties. So I did all of his brochures and programs and of course charged him for that. And then I started companies when I was in high school and college, graduate school. So I've started half a dozen companies as an entrepreneur. One company that I started in the 1990s got very, very big. It was in the wireless internet space. We had 125 people and we raised a lot of venture capital. And then along came the dot com crash in 2000 and that wasn't fun. So that company all of a sudden went from very, very big to very, very small and then eventually nonexistent. And so at that point, I semi retired from the entrepreneurial world and became an angel investor and started investing in other startup companies. And so then having done that for a number of years to the point that I've now invested in well over a hundred other companies. I finally, about 10 years ago, went back into the entrepreneurial side seeing that there was a need for a new platform to bring together all these startup entrepreneurs and these startup investors. And so that's when I created gust. So I've been running gust now for 12 years. There's my current entrepreneurial operation. So I am both an investor and an entrepreneur. I'm proud of being both. Well, David, we can clearly see that you've had a lot of steps to success on your way to being an angel investor and having founded and funded these hundred pioneering companies. But what we do also know is that entrepreneurs learn a lot from their failures as well as their successes. Could you perhaps give us a example of a mistake you might have made as a young entrepreneur and then how you were covered from that mistake? There are probably two things that it took me many years to learn.  And the first one is to not take things personally. We're all human beings. We all react to people personally.  But in reality, it's just business. So you really have to sort of divorce the emotion from this.   It's not because they're evil. It's just because for whatever reason, your needs for money don't match their needs for funding. So don't take things personally, number one. And then number two, also took me two or three or four or five business to figure this one out, is that cash is king. So cash is the life, what a business. You're trying to ultimately make money. To make money, it takes money. So you're going to have to invest, whether it's employees or products or whatever. And so if you don't have money, it's lubricating your business, the business just can't operate. And so therefore, whether the money comes from bootstrapping, from your savings, from loans, from investors, cash ultimately is king. So always watch your cash. Don't spend more cash before you have it. Don't scale up before you need to do it, because if you run out of cash, the business stops. Whereas if you have cash, even if things aren't working, you can pivot and then try and do something else with that cash. So cash is king, always pay attention to your cash. We have another online question. What are your recommendations for businesses and regions where resources and infrastructure is minimal? Smaller markets potentially have less access to investors and they also have less access to the talent pool, potentially individuals that make up your team. What do you do in situations like these? Well, there are two things. First of all, there is the local economy. I would posit that entrepreneurship is not something that is special to the United States or Western Europe. I think entrepreneurship in the entrepreneurial gene, as it were, people who are ready to be entrepreneurs, are effectively spread around the world. So you can be an entrepreneur anywhere. There is no respecter of geographic borders. And that means that in every single area, there are markets, there are people to do things.  And then once you get traction, once you get going, you can then expand from your initial beachhead to neighboring regions, neighboring areas and internationally. And that brings us to the second piece, which is that the world has changed so drastically over the last decade or two with the internet. So now, whether you are talking about starting in Finland, whether you're talking about starting in Silicon Valley or in Asia, anywhere you are, you can start a company today and draw on resources of people and employees and partners around the world and produce something that can then be consumed in multiple languages around the world, virtually on the internet. So I would suggest on the one hand, both looking actively in your local area to find people to work with you, to find co-founders and to get your business started there and then keep your eyes open for larger opportunities. International financing today is still on its very earliest stages. There are not a whole lot of investors, certainly individual angel investors, who invest internationally. But it's beginning to change. You're beginning to see it. And I think, you know, five years from now, 10 years from now, it'll be much more common to have no respect for borders, so you'll have people investing anywhere around the world into any kind of company. But for now, start small and keep your eye on the big ball. Great. And that actually leads to a different question, David. You had mentioned teams and I was wondering when you're discussing who you want to be part of your team and you're saying that we're in a globalized network, so you can really look for people across borders. There may be language differences. There may be a better resource, say, in one country than in another country. How do you find the right mentors in these areas? Because you really do need mentors to be able to get your business started. A mentor is a word that is often misused. A real mentor is somebody who works with a protege over a long period of time and serves as their guide and their teacher throughout a whole career. And finding a real mentor is actually very, very rare. Most people anywhere in the world, even in the US, never have any mentor at all. And so the idea that you have to have a mentor day one, that's a check off thing to do, that's incorrect because if you spend your entire time looking for a mentor, it's like looking for love, the harder you look, the harder it is to get it. So crew mentor relationships build naturally. If you are open to seeing what's out there, to helping other people, you will likely find that over time mentors will develop. But that being said, what people are really looking for are advisors. And that's a different kettle of fish. So first of all, in terms of advisors, there are a lot of people who write and produce advice, what I call mass mentoring. So things like my book, for example, so anybody can read my book and get the book. It's available in the hardcover and audio and digital download, as opposed to talking to me personally, which I would love to do for everybody. But that's why I'm doing events like this, which again, I consider scaled mentoring. So rather than try and find a mentor day one, although you definitely want to keep your eyes open, if somebody in your industry or your area or a more experienced entrepreneur offers the help, that's great. But take advantage, first of all, of all the scaled mentoring that's out there, the books and the blogs and the videos and the web sessions like this. These are all great kind of tools that you should do, and they're easily available. Well, yes. And it's really important, as you said, to clarify what the terms that we're using are, because as you said, a lot of times people use the term mentor and use the term advisor, but don't understand how you defined it as a relationship versus, like you said, doing a program like this and giving advice, but you can't really get the one to one and spend a lot of time. So thank you, David. I also, we had a question, a comment coming rather where they said, if the student is ready, the teacher will appear. Absolutely. My father taught me that when I was about five or six years old, and that has been one of my mantras ever since. When the student is ready, the guru appears, and when you are ready to receive information from a mentor, when you really understand that mentoring and a protege, that's a two-way relationship, and the mentor needs to get something back from the protege, the feeling that comes from helping somebody, then often you will find a mentor. Absolutely. And so I think this is a natural tie-in to saying, if asking for a mentor is not necessarily the right question you should be asking, what other questions should entrepreneurs be asking as they embark into their journey? Well, the first thing to do is what not to ask, and the things that are not to ask are things that you can find out on your own. I am often amazed to find that people are asking questions about the simplest things, and if they took two and a half seconds, they would realize that the answers are all right in front of them in books and blogs and webcasts and all these kinds of things. So you have to put something in before you get something out. You have to do some work. Nobody is going to hand you a company on a silver platter. Nobody is going to make your, is out there to make your life easy. That's not the way the world works. So you have to put things in and you have to start. And the first way is do the reading, do the research, and don't go if you actually have the chance to talk to somebody who might be an experienced advisor or potentially a mentor.  Ask them the tougher questions, the questions about them, personally, the kind of advice questions that are not generic advice that you can find on your own. Yes, I think a lot of investors probably appreciate the preparation because you only have so much time and they want to use that time effectively and wisely. So thank you, David. We now have another question from a viewing group in Bermuda, B-E-D-C. And the question is, what do you say to entrepreneurs who want support, both advice and financial support, but don't want their idea to be stolen? And I think this might go back to what you touched upon with employee agreements, but could you expand the idea maybe a bit further? Well, yes and no. Actually, this is one of the most frequent questions you get from, we get as investors, from entrepreneurs, which is how can I share information with investors to get them to fund my idea, but not have them steal my idea? And the answer is that's not the way the world works. And I'll tell you why, because ideas by themselves, which entrepreneurs often think is the most important thing, really aren't. There are many, many, many ideas out there and a great idea, no matter how spectacularly wonderful, doesn't by itself do anything for you. It is the execution of that idea that is most important. Number one, number two, investors are not in the business of stealing ideas, right? If they were to be entrepreneurs to start a company, they would start a company. And there are so many ideas out there, investors don't need to steal ideas. Let me give you an example. Right now on Gus, we have about half a million companies who have profiles on Gus, who are looking for investors in the part of the Gus community. Half a million. Do you think that there are half a million different business ideas out there? No, there are not. Do we think that there are 50,000 different business ideas out there? No, there are not. Do we think that there are 5,000 different ideas out there? Maybe, but I probably don't think so. 500, that's more like it. So what that means is that there are between 1,000 and 10,000 people today already doing exactly what you are doing, who are looking for investors at the same time. So it's not a question of secret ideas. It's all a question of execution. And entrepreneurs should not at all, at all be concerned about protecting the ideas. Now, if you have a secret formula or something that hasn't been patented yet, a trade secret, that's a different kind of thing. You don't need to tell everybody your deepest, darkest secrets day one. But in terms of business ideas, share it widely to everybody. Great. And David, I actually, I think that's a perfect segue into your area of expertise, which is angel investing. We had a question from Turkey, which asks, how much time should you put into a presentation when you go to an angel investor? What is the angel investor looking for? And what is the most effective way for an entrepreneur to communicate their idea to that investor? Good question. And that's actually all stuff that I covered in my book. If you want the details of information in my first book, this is my second book, my first book was called Angel Investing, the Gus Guide to Making Money and Having Fun Investing in Startups. And that was designed for angel investors telling them what to look for. So if you're an entrepreneur who is interested in what the angels are actually looking for, you can get that sort of a secret book, which is their secret playbook, which by the way, has been translated into Turkish and Chinese, and much other languages as well. But that being said, there are two questions here. One, how much time should you put into your presentation, which I'll interpret as how much time should you spend making your presentation? And then how much time should you spend giving the presentation? How long should it be? And the answer is in terms of making your presentation a lot, a lot, a lot of time. If you think about it, if you need cash just for you to grow your business and the investor is the one opportunity to get that cash, then this is the most important thing in your business life, right? You had better spend as much time as it takes to do a really, really solid presentation. I would point for some suggestions to a, I did a TED talk about this several years ago, which has been seen about a million times so far. You can just go online either on TED.com or on YouTube and search for David S. Rose Pitch of BC, and you'll get about a 12-minute video on how to create a presentation for an angel investor or a VC. But essentially what the investors are looking for is they want to understand the idea, first of all, they want to understand the market, that the market itself is big and growing. They want to understand what the pain is in the market. Why do people need what you're doing and then how do you solve that pain? What's your product? What's your solution? And then how does that solution of yours actually make money? What's your business model? And then who are the customers that you're going to sell this to? How do you find those customers and others? How do you get to market when your customers are looking at your solution and other solutions? Why are you better or different? What's your secret sauce? And then when you put all this together, do you have the team to execute and deliver on this? And those are the kind of things in general that investors are looking for in a presentation. And the actual form of a pitch to an angel or VC, you should plan about 15 minutes 20 minutes maximum. After that, they'll ask questions and you can do demonstrations, but the actual core pitch you should spend as long as it takes days or weeks or months to do it and the actual delivery 15 to 20 minutes. In the delivery of a pitch, David, because you had mentioned that we had viewers from around the globe, Asia, Africa, Latin America, everywhere in between. Is there a difference in the type of pitch that one of these individuals would give to someone in their country versus somebody in the US giving a pitch to a US investor? The answer is I haven't done too much pitching internationally myself, but I would say yes. And that has to do with cultural differences. Angel investors in the United States, for example, are now those of us who are serious about angel investing and are semi-professional, see lots and lots and lots of pitches. So I probably see a thousand or more pitches every year. And so on the one hand, we're tuned for just those specific things and we can get to the cut to the chase very quickly, whereas in places where you're dealing with somebody who might be a first-time angel investor who hasn't had the experience of seeing lots of pitches, you might need to take it a little more slowly and walk through it. But one of the thing that's really important is that your present, the most important thing about a presentation is that it have a logical flow because you're teaching somebody about your business and the market from when you start your presentation. And so if it's logical, they should be able to follow you all the way through your presentation. And then in certain cultures, you need to be a little more formal than you do in other cultures. The whole startup culture is increasingly informal. So I mean, I'm wearing a jacket today, which I've worn a jacket. I wore a tie when I was six years old. And so it's for people like me, getting informal is a little trickier. But in some cultures, you're still very much a jacket and tie or business suit kind of culture. So you have to adopt it for your particular region. But the most important thing is to logically and clearly take somebody through the entire process so they can understand what you're doing. Expanding on that idea a little bit, we have a viewer in Moscow who would like to know what advice you would give for foreign entrepreneurs who want to start businesses in the US. So our previous question, David, was foreign entrepreneurs pitching to foreign entrepreneurs in their country or US entrepreneurs pitching to US investors. But now we're combining the two and saying if you have a foreign entrepreneur coming to New York, for example, or to Silicon Valley, how does that dynamic change? So first of all, as I mentioned, international cross border investing is still in its earliest stages. 20 years ago, it would have been incomprehensible. Nobody would have invested outside of their home country. What we're seeing now is economic borders and entrepreneurship borders are dropping pretty rapidly. So I mean, I have personally invested in companies in Peru and in Finland and in Spain. And so it's happening. But again, it's very much the exception rather than the rule. So since most investors are investing locally and since there are many opportunities for investors to invest, if you're coming from Russia, for example, to look for money in the US, you're competing against every American business that is looking for money also from those same investors. So the first thing is today in 2016, most investors are not going to write a check to a company that is located and operates in another country. There are some venture capital funds that are set up to do that, but they're not all that common. So the first place to look is within your country. There are active angel groups in Russia. As a matter of fact, dust is the official platform for the National Federation of Angel Investment Groups in Russia and we're localized into Russian. So you should start by going to Russia, start by going to August in Russian, put up your profile and look for investors there. That being said, if your company has scaled and is successful in your home economy and is now looking to expand in the US, that's a little different story. So if you're doing that, the expectation from American investors would be that you will have an office here. You might even move the headquarters of the company here to the US, in which case you then be a US company with an international heritage, which is fine and we'd be investing in a US company or you'd have a major presence in office here. Investors typically like to be able to meet the CEO, visit the offices of the company and that's difficult if you're outside of your local country. Great and thank you to Moscow for sending us that question. Now let's move to a different area of the world. We have a viewer from Iraq from Basra who asks if you have any advice for startups in a specific sector. So in this case, for example, this individual has a farming business and in our just tech eye competition every year, we often have individual sectors and agriculture is one where we have a lot of finance, sorry, we have a lot of finalists who are coming up with really innovative startups to address agricultural problems in their country. In terms of scalability, what steps do you need to take for this sort of sector versus say if you're using an app or an online ICT startup? So actually agriculture, what we know as known as ag tech here in the US, new companies, new startups and technology supporting agriculture is actually a very versioning area. So the whole field of agriculture and food tech is exploding. I'm actually a mentor for an accelerator program called FoodX here in New York, which is all about food and agriculture related startups. So the trick for a business in any sector, if it's going to be looking for investment and looking to grow is it needs to be scalable.  That's not a scalable business. On the other hand, if you are developing a technology to support agriculture, for example, some type of new fertilization for seeds or processing equipment or the things that improve the overall process that can be applied to many different farms and many different producers around the world, that's an inherently scalable business. And so again, there's a great book called Crossing the Chasm, C-H-A-S-M by Jeffrey Moore, which is all about how do you get over the initial hump from a good idea and your first early adopters to the mainstream people who will purchase your product? And the answer is you have to cross that chasm, cross that big divide by getting some customers somewhere to start doing it. And then you branch out from them. So if you have a new process, for example, that works with one type of crop, after you get that market established and going, then you can expand to a different type of crop, or you can expand to people who do that crop in a different country. And so establish your beachhead and expand from there. Our viewers, I know we're mentioning a lot of resources that are helpful to look at in your preparation. And on our website at the end of the program, www.gistnetwork.org, we'll try and post the books that are being mentioned and also naturally have David's presentation. So if you're taking notes, we'll also have that available as well. So going back to Bermuda, we have an individual named Kalina who asks, if you are starting a business as a small business, is it better off to be bootstrapping or should you look for loans and investors? Bootstrapping, bootstrapping, absolutely. So if there is any way at all that you can bootstrap a business, that's what you should do for all kinds of reasons. First of all, it's faster. You can do it right now. It's not instead of waiting for an investor, which might take years, months or forever or never to get there. Number two, you will have more control. The minute you take investors, you are giving up a piece of your company, ownership of the good things that come from it and part of the control of your company. And so by bootstrapping and doing it yourself, you retain control and ownership of your company. And then number three, bootstrapping. To bootstrap means you have to be really tight in your plan and you really have to be tight with your cash. And that makes for a better company. So if you can possibly bootstrap a business, then you absolutely should do that and only go looking for investors after you've actually demonstrated some traction. And that's another big problem. Many people say, oh, I have an idea, so I need an investor. But that doesn't work. Investors do not invest in ideas. Let me repeat that. It's a very, if there's one thing that you're going to get out of this podcast, investors do not invest in ideas. They invest in companies that have traction, that have shown that somebody has taken this idea and executed on it because investment is all about execution. Remember a little while ago, I said that we had half a million companies on Gus looking for funding. So there are all these ideas and all these people with ideas out there, but the ones that get investments are the ones that have already shown that they can do something and get something done somehow, some way by bootstrapping on their own. And those are the kind of companies that investors want to invest in. Viewer took what you had just said about demonstrating the success of your startup by asking what benchmarks can you hit to know that you are being successful? The best benchmark that investors look for is traction. And traction means that you have, if you are driving a car and the wheels are spinning, if you're on ice, the wheels will spin, spin, spin, spin, spin, and not go anywhere. But if you're on asphalt or pavement, they spin it, they get traction and the car moves. And so when we say traction in the business for looking at startups, that means, first of all, have you found some type of what we call product market fit? In other words, have you found that you are creating a product or a service that fits what the market needs and that somebody is willing to pay for? And the best way to demonstrate that is by having somebody actually buy your product. And so what investors look for is some type of traction that shows the business that you have started can now keep going. And if you're taking our investment that money in, that that money is going to now do bigger, faster, better things, but you've already got the car started. You already have some kind of traction. And so it depends on the different type of business. It can be paying sales. It can be a viral distribution. It can be wonderful reviews and happy customers. The goal is some kind of traction that shows that you have hit that product market that are very close to it and that somebody is going to be willing to pay for what you're selling. In terms of product market fit, there's a viewer from Armenia who asks if there is a method to compare and evaluate different ideas that they might have if they may have two or three similar ideas, but one might be better than the other. If there's an actual way to compare and contrast that is effective for them to go forward with the right idea. It'd be great to say there is some secret website or process that you could put six ideas into and have the winner pop out. Unfortunately, it doesn't work that way. That's what entrepreneurship is all about. There's a great book called The Lean Startup by Eric Ries, which discusses the whole question of how do you find that product market fit? How do you keep your startup very, very lean and bootstrap? I discussed this in chapter three, I think, of the book, but you can get Eric's book and it goes into much more detail. And also a book by Steve Blank called The Startup Owners Checklist, Startup Owners Manual. And in fact, in the first appendix of my book, The Startup Checklist, I have a reading list of a couple of dozen other wonderful books about starting up. But the essence of The Lean Startup methodology is you start something, take the best, what you think is the best idea or the most promising idea of the ones you have and put it out there, do something. Do the leanest, tightest, what we call an MVP, minimum viable product, which is not the product you ultimately want to ship. It's not the full flower of what you're doing. It's the smallest, teeniest starting point that might actually be a benefit to somebody and put that out there and then see what happens and then measure that and then tweak and adjust the process or the product or the service and put it out there again and see if you get more people buying and paying more money or getting more product.  So the answer is pick the best one you have a shot at, go do it and see what happens. We know that one of our viewers has a little bit of trepidation about the whole process because you've been very clear, David, about certain steps you need to take looking at stock, looking at the employee agreement, looking for attorneys. And the viewer is concerned that he just doesn't have enough knowledge. So is there any advice you could give to someone about what seems like a daunting task but to know that perhaps failure is not a bad thing? What would you tell that viewer? Sure. First advice, read the book. Which is why I actually wrote the book, to put all that stuff down on paper. That being said, remember that the reason I wrote this book particularly was these are the the technical things that you need to do to to create a startup that's going to get very big. Now in reality virtually every single thing in that book can be done later. It just costs more and it's more expensive. If you start out doing it right it makes life a lot easier because you can do it easier, faster, cheaper from the beginning. But ultimately the most important thing about a company is to start. And so do not let the fact that I've given you 25 different complex things get in the way of starting. If you have to read the book then forget it, throw it away and just get something started. The biggest problem that people have when they're thinking about it but are prepidacious or scared about starting is they're what we call entrepreneurs. People who want to be an entrepreneur but never actually get themselves together to start. They might read all the books and go to all the meetups and talk to all the people and sit there spouting off on websites and commenting on blogs doing everything in business except actually starting a business. And that's a real problem. So my first suggestion would be if it's not all that complicated, number one, the book is a guide for you and there are plenty whether there are 92,000 books on Amazon about how to start a company. But the most important, most important thing, forget the books is just start the company, do something. And if you have to you can do all that stuff I mentioned in the book after you get the company started. But read the book first so you have an idea as to where you're going. Thank you David and to our viewer I hope that that gives you a little bit of reassurance that as long as you take some action and as long as you start you're on the right path. So we're unfortunately running low on time so I'm going to give you one more question David and this is from Bermuda, B-E-D-C asking what is the difference between an accelerator and an incubator and is it better to be in one or to just continue with your journey? Sure. So historically if you were an entrepreneur starting up a company you were all by yourself and you were starting up and you were seeing something other people didn't see and you were creating it in a vacuum. Over the last 10 or 15 years there have been a whole it's a whole ecosystem of supporters who are out there to help you start your business whether it is webcasts like this whether it is books like the one I've written or whether it is people on the ground successful entrepreneurs and advisors who can help you get going. So the first of these were places called incubators and an incubator like an incubator on a farm for chickens or in a hospital for babies is designed to provide a nice warm comfortable nutrient rich place in which you can start a tiny little fledgling business and get bigger. So incubators are typically offices they're shared offices where you can find an inexpensive place to rent where you can find some equipment some advisors who are available for the legal or accounting or other kinds of business things. You typically will pay rent to be part of an accelerator although some accelerators are part of universities or other not-for-profit organizations and it might be less rent or even free if you are a part of that university. On the other hand an accelerator is a new type of thing. It started about seven or eight years ago. The term was invented by a guy named Paul Graham who started something called Y Combinator which was the first accelerator program. It proved to be remarkably successful and there are now hundreds of them around the world as a matter of fact just actually a couple weeks ago put out a report on all the world's major accelerators which you can download for free and so an accelerator is typically a short intensive three-month usually period highly selected where startup teams that are in their very very earliest stages typically more than an idea but less than product market fit apply to join this program. They're accepted and acceptance rates are typically very low often two, three, four, five percent to this program and during that three months they are given intense real mentoring and advice and hands-on support often legal and accounting and other kind of support. They're often brought together in a co-working space or shared working space for that three months. They'll work with other companies in that cohort and they'll learn from each other and then at the end they will be presented to investors and and potential venture capitalists and angel investors so the goal is to take a really great team and accelerate it to product market fit and to getting funding so they're both kind of they both be very very good you want to make sure you're in a good one so check references from all of them if you have a chance to be in a top tier or good accelerator program by all means look at that because they often come with funding as well at least to stipend to fund you during the process of the accelerator so an incubator is typically a location that you go and spend as much time as you need there an accelerator is typically a three month or so program you apply to to help jumpstart your company. Thank you for that distinction and thanks to Bermuda for sending the question. So David now I have the privilege as the moderator to ask you our final question which is what is the one takeaway you would leave for our viewers today? It's probably something we mentioned a few minutes ago which is that if you are thinking about starting a business start absolutely that's the most important thing you know you'll read the book and I'll tell you that you want to have a plan you want to turn you do a business model canvas to figure out if your idea is worthy of doing those are all great things but if there's one takeaway if you think you might want to start a business start it and that's the most important thing and you'll very quickly figure out having started it whether this is the craziest most insane thing you've ever done in which case it'll run away and won't start it again in which case you know you shouldn't be an entrepreneur or else you will find that it is challenging and exciting and crazy making and you are cut out to be an entrepreneur because there is absolutely no respecter of time or anything else one of my favorite entrepreneurs in the whole world my mentor and role model the guy named Norman Lear who is now 93 years old going 94 my father is 86 and is still working as an entrepreneur there's no respecter of age but you can get to 93 years old and not start a business by just talking about it forget the talking get out there and start your business David that is truly an inspiring way for us to end you've given us practical advice you've given us the strategic thinking about how to get started you've given us resources which again will be on our website and also knowing that we have support I think entrepreneurs a lot of times feel like they're alone but you've told us about this great resource gust which I encourage all of you to look at and we just really thank you for your time David this has been a wonderful presentation it's been my pleasure being here go out and start a business and to our viewers thank you for participating as always and again we invite you to find David's book the 25 steps to a scalable high growth business and we're going to close today my name is Jazz Ozery and as always I encourage you to think innovate and improve our world


It starts from raw data, which are in files, and maybe it's not even in files, it's on hard disks in your drawer. And they may be in repositories, but typically not. And some of the data would be lucky enough that it has been processed and somewhat cleaned up. They end up in collections, and where are the collections? Well, they are in your drawers. There are also some, sometimes, in public repositories or private repositories. And if they're really lucky, they're in some databases. Now, getting closer to publications, there is some data products that people produce by taking some of the data, and they process it through one or more workflows, different analysis. These essentially show up in public repositories, but also in databases, in websites just loosely linked, and sometimes as supplements in a publication. Now, on top of that, of course, are the publications themselves, which mostly do not have data associated with it. If they do, they can be horrendous. For example, I've seen this publication where there is a 146-page supplement which has tables in PDF. Okay, so what can you do with it? Nothing. You have to go back and sort of redo the whole thing of converting it into a form that we can process. Well, other things happen to the data, too. There are people who have developed processes of creating sort of fact collections, if you would. An example would be Drug Bank. So they have information about drugs, their targets, what the drug does, and they have compiled this information from various sources, and they have made it available as a website and also sort of as an internal database. There are other people who have taken the same kind of information, taken the more general things, and formally represented them as knowledge bases or ontologies. And by trying to make them formal, they have also uncovered some very interesting problems with the data, and especially with definitions of things. So Marion would tell you horror stories about trying to put anatomical terms and subcellular terms into an ontology, and the fact that definitions are not sometimes clear. So when we point to a piece of data and say this is about X, well, what exactly is X, and if there is a community debate about it, we have a problem. But the fact that people have tried that exercise makes this, these ontologies, also valuable sources of data. Two other things have happened. People have started annotating data and publications.  You know, Tim's group has this tool called Domeo, and the goal is, okay, if you have a publication, you, the reader, can take this and associate it with other things. You can say, okay, this is an antibody, and that's something we do in life using Domeo and other tools. The other thing that has happened is people are beginning to cross-link publications and data. So there is a thing called a link-out which relates a publication to all the data that might relate to this publication. So these auxiliary things are not really data per se, but they are in the space of data shared. Finally, there are many sources these days that can act as aggregators. So they take other people's data and make a huge catalog, if you would, pointing to various things. Now, in the business of data sharing today, we have all of these elements. So we have a lot of duplication. We have a lot of things that are represented one way in one place and represent another way in another place. But in this world of data sharing today, this is what we have. Now, the thing about the pyramid is that the top part of it is more shared than the bottom part of it. However, if you try to, the utility of the top part is different from the utility of the bottom part. If you're trying to re-analyze data, you really want to get those numbers. So if you are a bioinformatics kind of person or if you're just trying to verify something, you would want to get to the actual data. If you're trying to discover new things, if you're a new postdoc trying to figure out your ways, you would like to get the summarized data, the collections, the aggregates, the papers, and annotations of the papers. That's what you would like to get. So there is a need for a data sharing enterprise to cater to all kinds of needs, the needs of the information or the analyst, as well as the needs of the people who just want to get at their hands onto something and then they would delve deep. The interesting thing here is, did you know how many repositories there are of life science type data? NIF knows about 7.61 so far. I didn't even know that there are so many repositories. And surely they do not cross-reference each other. That's just a factoid. The point I wanted to make is, if you look at all the data that's there, there's a really uneven distribution of the data, the parameters of data. How much data is there? The data volume varies differently across the scales. The weight of data velocity, that means the rate at which data comes in and goes out, varies a lot. The nature of data, it could be binary data because it just comes from machine, or it could be highly processed XML data, or it could be RDF because it has been manufactured, it has been worked on sufficiently that people have formalized it. There's a lot of variability, similar variation in location and availability. Now, let's go back to what Kamran said. Here's the paper that he referred to. And the paper says that it's a pity that we somehow assume that the scientists have done the right thing and they have made the right conclusions from the data, but that's far from reality because we could not produce anything. Well, that's what Kamran already said. If you actually read the paper, they also say a few other things. And these are almost quoted from the paper. They said, well, we need a paper to report all data sets that are relevant to it, all, not selective things, everything. And it's important. And there must be an opportunity to say this is data that didn't work, so if there is negative data at whatever level of the pyramid, they have to be reported. The third thing the paper says is that there should be cross-linking. So if there is a paper that says something, there is another paper that refutes it, the paper that refutes it should really point back to the data where it found a problem. And similarly, if you support something, you should also point. That's what the paper says. So there has to be absolute cross-linking so that we know what comes from where. Well, that's a very reasonable thing to say because we want to produce, you know, reproducible science, reliable science. The things that sort of go with that is we need to get at the data of not, you know, five years from your initial experiments, but as soon as possible. And the more people that see it and the more people that re-analyze it and find errors, if any, you know, people make mistakes. Making a mistake is not a problem. But if you make it available, it's you and other people who would find it. And it's better for science. Now, of course, there is this. This is from a recent report. We said why data should not be shared. I just put it up there for fun. But I don't believe that, of course. But I have a suspicion that many do. But this is only one problem that there is this fear of intellectual property going away.  There are some practical problems. One practical problem is this. This is data that we got from a group when we asked for their raw data. Well, I didn't really know. We didn't know what to do with this. And they were not telling us. So imagine the scientist who produced this data after a lot of work. If this person was asked to clean up the data and make it available to me in a proper way, they have to do a ton of work. And it's not simple. So when this happens, guess what happens to their IT guy? This is what happens. This is an actual email. Which says, please, please clean up the hard list because I'm running out of space. You are not cleaning your data. And I have only so much space. I don't know what to do with your next analysis because I cannot do it. So the problem I wanted to point to is saying that I'm going to clean the data, saying that I'm going to structure the data, I'm going to annotate the data. I'm going to make it available in a proper form. A proper form is not simple. It's a lot of work by not only the scientist, but an entire team that does various parts of it. Now let's go back to the paper. What the paper is essentially saying is the following. If you do an experiment, regardless of whether the data leads to positive results or negative results, store the data. Remember the plight of the guy who wrote that email. You may not have storage to store that data. So if somebody has to do it, there needs to be enough of a shared space where voluminous data can be stored. Annotate the data at least to the extent that we know what the data is about. Unlike the example I showed. Now if you are analyzing, tell us what you did exactly. So not only the results of your analysis, but the process of your analysis. Now if you have an analyzed result, well, tell us what the provenance was. And then if somebody finds an error or you find an error, point to the place where the error occurred and tell us that. And if you have any prior publications with the error, you make a reference to that. That's what this paper is suggesting. Now it's suggesting that for the scientist. For the publishers, it's saying if you have a paper which has a result, which has some future reference, you make sure that all the references are consistent. So plus it's telling the publishers, well, if you have published something with a reagent, I need to know the catalog number of that reagent. And Anita will tell you how painful that business is. So if you have done an analysis with a version of the data, you must at least give us the time stamp of the version on which your analysis was done. Now before you repository the system, this paper is also telling us, well, make sure that you make the data available as soon as possible, not three months after it was submitted. So if you are a repository and you want people to access it, don't just give the ability to download because it might be a truckload of data. If it's two terabytes of data, I cannot download it. So if you are a good repository, you would also give people the opportunity to analyze in place. The data must adhere to the right standards. It must be consistent. And it should be simultaneously analyzable by a lot of people. That's a lot of constraints. So when people say, I am going to... So here is my question. If we do this over all people, all scientists, all institutions, all repositories, public data centers, publishers, whoever is operating on any piece of the data or any derivations thereof, it's actually a whole lot of work. And what it says is that at the back of it, we really need a very scalable and elastic infrastructure. If we do not have that infrastructure, this scientist is not going to be able to afford one petabyte of data stored for all the data that that person has collected through his or her professional life. So we need the infrastructure just to think beyond just the scientist if we want to make data sharing a reality. It's a whole enterprise and a whole lot of sort of back-end infrastructure has to be in place. It also says that, okay, if you give the data to somebody and I'm going to just index it, but that indexing on a lot of data is a lot of computation. So I need to have the ability to compute it. Just in if the size of a Lucene index is over, it's about half a terabyte. It is just because of the volume of data we have. And we do not deal with binary data. No, we deal with structured, semi-structured data. So this business of having the right infrastructure, we should not forget it when we talk about the equation of where is the data? Why is it not being shared? Right. So the other implicit assumption is that we just don't share data. We expect a lot of services to be around. We want to search and query the system so that we can find the data. And here are some analysis that we do, some kinds of analysis that we do in this. People look for facts. People look for, you know, landscape service. People actually ask, you know, what kind of data do you have about this? Just want to know the data holdings. That's one of the questions people ask. So we also want the system to do active analysis. You know, do run this analysis on that data. So here are two examples. And what we are also hearing from this paper, and we do this just a little bit in NIF, is traceback. What we do in NIF is very simple. You have some data and you derive other data out of it. So you write views on top of the data, for example. If you have a chain of views from some original data and the original data gets updated, you need to know that. All the views need to know that. So wherever you have a data repository, where you have derived results, you have to have a traceback facility. Now, this becomes a much larger problem if the data is distributed over multiple repositories, multiple databases. And we still need a system-wide provenance tracking. It's not just provenance tracking of one database. Because remember, everybody's pointing to the right data. And the data that they derive from is not necessarily with them. It's with the originator of the data. So there are many groups today looking at different parts of the problem. We mentioned NIF because, you know, I'm in NIF. But if you look at INCS, it is working on standards. It is working on tools. If you look at this project called Elixir, it's looking at architectures of how a multi-institution system would work in terms of providing data, providing indexes, providing services, and computations. So the opportunity for data exchange looks at what are the requirements of every stakeholder in the data-sharing enterprise if you should go read their reports, and they are actually very insightful. So I have some questions. One is, can we really support all the data that's there today? And the answer is no. We need to think of an architecture, not just individual data-sharing centers, but an architecture which will sort of cover the whole thing. There are two questions that I'm going to focus on in the rest of my talk, how can we also create a monitoring and incentivizing method on top of what exists? Now, going back, I have to hurry up, I suppose, how many of you actually have looked at Facebook properly?  So all data types have their own semantic types and a unique ID. And anytime there is a program running on it, we know exactly what program is running on what piece of data. This is maintained in a log in the Facebook file system. And any action that you do on it, so if you update something and then delete it, the system would know, and it would have to go to all other machines which has replicas of this and have to go and delete it. We believe that this distributed activity model is applicable to a data-sharing enterprise. So if you really want to monitor, you should be able to do things like this. If a data set has been deposited, you should be able to know, because that's an event, that's internal to the system. If there is a resource referencing another resource, and this reference comes in because it's structured, like data-site or DOMI, or some standard, you know that it's a referencing event. So you should be able to characterize the kinds of data events that occur within the whole data-sharing enterprise, and you should be able to monitor it. What happens if you can monitor it? You can count. You can count the frequencies and regularities of the various operations and the agents that cause it. Remember, I said agents, I didn't say people. So the agents include people and databases and software systems, anything that is instrumented enough that I can sense its activities. Now, if you do that, you should be able to create an accountability score of good data citizenship. And we are not talking about the formula, but it has certain properties. If you submit something great once and never submit in five years, your accountability would go down a little. That's expected. Now, if a lot of people are talking about you, there's a lot of buzz about your paper, well, it should increase because you are clearly affecting and influencing the community. Similarly, if you have a publication and if you have more referenceable data than the voltage of the publication should go up, that's not very hard to implement because the technical infrastructure for this exists. And then, regardless of whether it's person or publication or data center or whatever agent you have in the system, you should be able to take this and also create an influence score and an influence class. So you should be able to say a data center is really a specialist in this class of data but not in that class of data. We do that a little bit in it because we say this source is more appropriate when you are doing a physiology query versus when you are doing a pharmacological query. This is not that hard if you have a semantic framework like an ontology in place. And guess what? The social networks already know this. So there are influence engines, that's what they are called. One is called clout, you know, what's your clout? That's a number, my number is 48. Obama's number is 99. Seriously. There is another called peer index. And the third one, it's called cred. Now these engines, what they are doing is they are looking at activities and computing scores. Now whether there's good for the society or bad for the society is a secondary question. However, the fact that they are looking at impact, they are trying to find measures of impact, I think is significant. And I think more of this will happen. Now does this directly apply to science? Not quite. Now if you are a theoretical physicist, we should not worry about your data submissions. The system should know that. The system, these measures are for online activities. So it's expecting that you sort of update every day because they're working off of Facebook and Twitter. Now that's not how science progresses. So parameters need to be adjusted, equations need to be changed. Also, I suppose some of this course will be community-based. In some sub-community, the rate of publication may be higher than another community. That should be factored in. I'm sure there are a lot of people who will object to this. They'll say, what, you are judging me?  This proposition makes no sense. But I believe that A, this would happen. And B, this would happen by through third-party watchers, like very good viewers. And I think if this happens, there is a lot of in-kind incentives that people would get. For example, people would want to know, are their data cited enough? It's good to know that your data has 1,500 derived products or papers. It's good to know that. So if you do this analysis, these analytics would inform the utility of your research. And I think that's important in progressing science. And I'll stop here. Okay, we have time for really a single question before we move to Merseysau. It's a comment to both yourself and Cameron, really. I think the first time I bumped into the problem of submitting supplementary material data sets to a publication was in the 90s. Three months ago, there was zero progress in 20 years. So it's all well to say, people need to submit their data set. But currently, we can't even submit a script or a PDB file or things like that to PLOS, for instance. So I think it's not just the scientists who are the bad guys here. Even the scientists who are the good guys, they just can't play the game. I completely agree with you. I said this is a problem of the entire infrastructure. So there are many players here, and there is a need for a large sort of backend thinking and holistic thinking. So that's why I didn't want to point it only on scientists. Yes, yes, I had to.