All right, welcome, everybody. Our next session is a keynote by Dr. Brahmar Mukherjee, who's the chair of the Division of Biostatistics at the University of Michigan. Dr. Mukherjee has been, I first met her at a JSM conference many years ago when she was focused primarily on statistical genetics. Dr. Mukherjee has since moved on to modeling viral infectious epidemiology, such as the spread of COVID in India. And she'll be speaking to us today about some of her more recent work. Please, everybody, extend a warm welcome to Dr. Mukherjee. Thank you so much. Can you hear me? Can you see me? Can you see my slides? All yes? Check, check, and check. Thank you so much. I was actually very worried after Sandrine and not being able to share her slides. But I just want to thank the organizers of this conference for not only creating a platform like Bioconductor, but also data scientists with different perspectives and different viewpoints together, statisticians, biostatisticians, computer scientists, bioinformaticians, epidemiologists all together. And this is a wonderful platform. So I'm going to talk a little bit about electronic health record research. And I'm not going to talk about my modeling work in India because I think that the electronic health record, just the general premise of how to harmonize data, how to analyze data, how to think about selection bias, that would be of more interest to this broad group of people. And I was finally able to actually combine my methodological research in electronic health records with some of the very tricky problems in COVID that we are all grappling with. So you might think that, why am I not in Seattle today? I'd rather be in Seattle than this rainy morning in Michigan. But I run a summer program. I always dreamt of a data revolution. And this is my army. Over the last seven years, we have trained more than 280 undergraduates. And this is their last week. So I cannot really leave the nest as the mama data scientist here. And as soon as I finish the talk, I'm going to go listen to their rehearsals. They have their final symposium. So we have 39 undergraduates from all over the world learning about the things that we are all interested in in this audience. So I divided my talk into three acts. Personally, I find giving virtual seminars extremely frustrating and intimidating. So I talked to my father, who is an actor. And he said, maybe you should try dividing it up into acts like a play. And each part makes a point. And you should pause and give the audience some chance to reflect on what you have said. So that's what I'm going to take my father's advice and going to divide my talk into three acts. And the first act is really unusual for a statistician. Because most of the time, we are really involved in analyzing data which has been collected by other people. But I was really fortunate to be part of this vision and mission of collecting EHR data and connecting it to other auxiliary data sources in Michigan. And what that taught me as a statistician to be an integral part of data making, I want to share with this audience. And then in act two, it's the meat of my talk, the statistical meat of my talk. How do I, as a old hat statistician with a PhD in 2001, becoming completely anti-Diluvian, think about sampling selection bias misclassification in big data, in this era of big data? Do my skill sets matter? And then in the third act, I'm going to talk about some of the COVID-19 problems. But throughout the lecture, I'm going to try to integrate some of the things that we saw with COVID-19 and expanding our data repository during the time of a pandemic. So I have to really thank, it takes a village to do this kind of work, but I would like to thank three particular individuals. Lauren Beasley is the statistical architect. She's a Feynman post-doctoral fellow at Los Alamos right now. Lars Fritscher is a human geneticist, and Max Salvatore is an epidemiology graduate student. So we needed genetics, we needed epidemiology, we needed biostatistics, we needed our programming, we needed bioinformatics to really pull off this project and medical informatics. And you'll see a snapshot of all of those skill sets coming together. So in Michigan, Precision Health, this initiative started, it was supported by the provost's office. It started about 10 years ago. We are actually celebrating 10 years of Precision Health. And the idea was to create a data infrastructure or an ecosystem so that it can serve as an accelerator and incubator for interdisciplinary health research. And it can be in a hypothesis-driven way so that you can validate your findings or be an agnostic way so that you're interrogating this massive database for some leads and some clues, which happened in the early days of COVID when we did not know much. I really like Michigan Genomics Initiative, that is our cornerstone study, we call it MGI, and it started in 2012 as a part of a building a biorepository at University of Michigan. We do not have Women's Health Initiative. We do not have Nurses Health Study. What can we build upon? And so it started with gathering a patient blood sample prior to surgery. So I highlighted that sentence in red because the selection mechanism, how people were recruited into MGI, this participant recruitment mechanism is extremely key to what we do with the follow-up statistical analysis. So they were recruited in anesthesiology clinic just prior to undergoing surgery. So this is a perioperative population. And this was a very bold consent, one-page consent, which allows us connect two core data elements, electronic health records with genetic data, but also anything else that's linkable through your social security number. And it also allows us to re-contact the participants with any findings or for future studies and data collection. It's a one-page, quite forward-looking consent that was rolled out in 2012. And I really, we have seen biobanks all over the world. In particular, UK Biobank has been game-changing with genetics, imaging, all kinds of omics data are shared with the whole world in such a democratic way that has been transformative for health research. But I really like Michigan Genomics Initiative. It does not compare, it has, you'll see that it has about 100,000 participants right now. It's not half a million. It does not have all the data domains, but it's very close to my heart because I see the potential to translate my work into the patients on the other side of the campus for example, I know this patient and I know this physician. So I said that the two core elements of MGI is definitely genetics and electronic health record. And we have electronic health record which will be characterized as medical phenotype from the electronic health record to go to medical phenotype is a huge stretch and I'm going to talk about that as well. And the medical phenotype, we have longitudinal data. So we have more than 16 million medical encounters and we have genetic data after imputation about 20 million genetic variants across the human genome. We have 92,000 consented samples so far as of June, 2022. But then with this, we all know that an individual's existence cannot be codified through just genetics and EHR. So we collected other forms of data. We rolled out an epidemiology questionnaire and behavior and lifestyle on a subset of population in a collaboration with Apple. We collected smart watch data. We have environmental questionnaire, exposure questionnaire. We have social data, we have family history. So as we started, we also realized the potential of linking auxiliary data sources. This has been really game-changing and those of you who are thinking about these data linkage projects in your medical center, I strongly encourage you to think about the consent which allows us to link publicly available data. And so we have the core elements which I mentioned are the genomics and the epidemiology questionnaires and some surveys that we rolled out. But then gradually we built a database where we connected to patient medical insurance claims, prescription data. We connected to dental records because that stay in another electronic health record system in Michigan. We connected to national death index. We connected to state death index. We connected to cancer registry. And then from geocoded longitudinal residential information we connected to many neighborhood and socioeconomic variables. So this took 10 years to build but we actually really chew on it and nibbled on it bite by bite to get this data ecosystem. So when COVID happened, the advantage of having this was that we already had an engaged patient participant community and then we were able to build cohorts with this digital ecosystem with COVID vaccination cohort, COVID outcomes cohort, now long COVID cohort, as well as roll out surveys. For example, in the early days of COVID, May to June of 2020, more than two years back, we did not know much how our participants were feeling about COVID, what were their sense of community exposure, what precautions they were taking. So my colleague, Kristen Weller, actually rolled out this survey where we could really study our participants and communicate to them and also really build a connection with our catchment area. So recently, once you do these things and then yesterday we published a paper on do you really need, when you do these surveys and you have the survey linked to HR, does self-reported COVID symptoms or outcome data add to the electronic health record or vice versa? Do you need both sources of data? So once you have created this mechanism, it really becomes easy to think about big projects. What no investigator would be able to pull off alone if the institution provides you with such a system, then maybe you are interested in a particular question and you can deep dive into a particular questionnaire or a particular form of omics data because you already have a lot of data available on a lot of people. So what is our summary of this process in the vision moving forward? We envision a rich data set obviously with large N and large B. And so this data set can be exploited and used by various investigators. It could be a social scientist, it could be a political science fellow, it could be somebody in pharmacy, of course, School of Public Health and College of Medicine. It also has questionnaires which makes us compatible to contribute to Biobank meta-analysis and we have been able to participate in many consortium related to COVID or many other disease outcomes because we have common questionnaire with UK Biobank and some of the phenotypes. It's a shared resource for the institution, it's just not for one investigator and that's very empowering and gratifying that you're building a legacy and you create something for the future. And as I told you that it could be really leveraged when some emerging health crisis or emerging challenges take place. So based on this, I was brave enough that electronic health records is one part of the study with two other fantastic women in our school write a cohort development grant of recruiting a population-based study because electronic health record, I'm going to, now I have sold the data set to you and in the next act, I'm going to completely debunk the data set and say that what are the different problems with this kind of data. But I was fortunate enough and bold enough and courageous enough to put forward this cohort development grant called MyCaS or Michigan CaS to study environmental risk factors and cancer. Over the next five years, we are going to recruit 100,000 population-based sample targeting on underrepresented groups and collect data on environmental exposure. This will be a healthy cohort of 25 to 45 or four year olds and we are going to collect all kinds of omics and exposure data. These are cancer-free individuals. It's a proper cohort study. So as a statistician, without being involved in this process of precision health and doing recruitment, doing community engagement studio, connecting with the participants, seeing the joy of that, I'd probably just analyze someone else's collected data for the rest of my life. So I wanted to share that excitement with you that we have this study, we really can once you have control on the data collection, then you can actually share it in different forms, use it for other people who are going to study other end points, except for cancer in addition to cancer. So this has been really game-changing. So with that, I'm going to take a pause because this is my sort of journey as being a part of this data making process or creating the data quilt of MGI or Michigan Precision Health. And I'd like to just pause for a minute for us to think about how the database looks like. And then I'm going to talk about the statistical issues with electronic health records. If there are any burning questions, I'm happy to take. All right. So now that we have thought about what the data architecture looks like, let us think about what are the downsides of this data. So the pros and cons of having a database like this is that it is easy. With probably the effort that we put in to recruit 100,000 participants, it would be very difficult. Participants are already coming for their healthcare to Michigan Medicine. It has large sample size. It has a lot of variables. It also has longitudinal data automatically. The median follow-up time in our cohort is about five years of data. And about 70% of the patients are actually seeking their primary care at University of Michigan as well. And the piece that which excited me the most is this callback potential. That when a crisis happens, we can reach out to this engaged participant community. But then the cons are that we really do not know why are we seeing an observation. We see, first of all, this is a hospital-based cohort, so it's not representative of a general population, unlike UK Biobank, which has the other bias of healthy participant bias. But, and also, we do not know why we see an observation in an electronic health record because that depends on who you are. What kind of access to healthcare do you have? Do you have insurance? Are you a hypochondriac? Do you have health history of diseases? It depends on so many things that we do not know the probability of a sample being included in your analysis. And also, some of the phenotypes and the data we extract are incredibly poor. And so we have to go back to our classic training in epidemiology and sampling, thinking about who is in my study. So we have to think about generalizability. We have to think about representativeness. We have to think about selection bias. Otherwise, we are going to produce an incredibly rich wrong analysis and database. So I see our curriculum since statistics, biostatistics, bioinformatics departments being really, really becoming very heavy on machine learning and computation. And that's absolutely necessary. But I do think that it's also important to think about stat one-on-one questions for big data, who is in my study and what is the target population of inference? And my pet peeve is that I urge every computer science department to reciprocate that we are trying to learn so much of your tools. Please institute a course on sampling and study design. And so those of you from computer science departments, please relay this plea. So now I want to say that what is going wrong? Before I go to electronic health records, I just want to talk about this fundamental concept that the big data paradox, which Zhao Li-Ming has beautifully explained in many of his work, including a recent nature paper on COVID, that big data can often actually hurt you. For the longest time, statisticians complained about not having enough sample size. But now that we are gathering data from multiple sites, we are really thinking about bias, but bias which does not go away with sample size are more than precision. So we have to reorient our toolkit to think about these different sources of bias, sampling bias, selection bias, information bias. So I borrowed this slide from my colleague Rod Little that suppose you have a true parameter that you are trying to estimate. Suppose that you know it's a simulation setting that you have T equal to 0.4. And we have spent a lot of time learning in classical statistics to define inclusion or exclusion criteria and think of a very well-designed study where we are not so concerned about bias. We sort of think that T would be inside the credible or confidence interval, but the length of the confidence interval is wide because it's such a carefully curated study. But now we have this big, big non-probabilistic sample. We can deal with probabilistic sample. If it's like over sampling certain groups, we know how to deal with that as long as we know the sampling probability, the whole field of sample survey has been built on that to deal with those situations. But non-probabilistic sample is where we do not know the probability of a sample being in my observation, being in my analytic dataset. So here we are dealing with a situation where I have tremendous N. So my precision and variance is going to, at a very, very fast level, it's becoming very, very narrow, the credible interval. So I am losing the ability to overcome even tiny little bias. This is incredibly important that this bias is not going to go away with sample size and your variance is almost, your tiny little variance is actually even hurting you to overcome that bias and do plausible, valid, principled inference. So this is the big data paradox where your sample size or your standard error being like 0.0001 order and your bias being of 0.01, 0.02 order is not being able to overcome. So now we probably want to design statistical methods which borrow on both of these tools and both of these principles that you have the true data and you have the well-designed study and also you have non-probabilistic sample, how to combine that. So exciting research is really coming out and statisticians know how to do bias variance trade-off and combine through shrinkage and other principles how to combine these two kinds of inference and two kinds of studies. So coming back to MGI, I'm going to show you why we started thinking about this that when I first looked at MGI, roughly 50% people had a prior cancer diagnosis. So obviously we really do not know, we do not have a population-based sample in the population in Michigan, we do not have 50% cancer incidents in this age group. What is shocking to me and surprising and heartwarming that the people that we approached 75 to 80% were actually willing to share all of this data with the University of Michigan researchers so that we can help people and help human health. So when we see that in the first period of this data gathering, we are so excited that we have so many people with genetics data and electronic health record data, what can we do in terms of medical informatics and genetics? So the first part after the first five, six years, we actually spent on calculating polygenic risk scores for multiple, multiple diseases, in particular my focus was on cancer in this database and really create repositories where people can evaluate and also download different polygenic risk core constructs. But at the time we did not really pay much attention to the selection bias issue. Gonzalo Abecases who started this Michigan Genomics Initiative, I remember I'll be in like really feisty conversations with him that this is garbage in, garbage out, this is not going to give meaningful interpretation or even genetics odds ratios. So lo and behold, we did look at the data and I want to show you something really striking which led to the conversation that I'm going to study mathematically. So what we saw in phenotype after phenotype on the X axis of this left side of the plot, I have plotted log odds ratio from genome wide association study catalogs, genetics is a field where we are very fortunate, germline genetics is a field where we have lots and lots of large studies where we can see that this is probably the gold standard meta-analysis of consortium studies. And on the Y axis, I have plotted much smaller MGI with selection bias. And what we see is a striking reproducibility in terms of not just the P values or the ranking, but point estimates. This really made me feel like a failure as a statistician that I don't understand this. This is so 50% people have cancer, this is a bias sample. How can these be so concordant? And the lens concordant coefficient was around 0.9. And we saw that for many phenotypes. And so it was very clear that in some cases it works and you can ignore the selection bias, but when? Is it always? It's not enough to know it works for this dataset. We have to know as statisticians, it's our responsibility to figure out when will it work and when will it not. So that genetics work was really, I would say that did not pay much attention to these issues of selection bias and outcome misclassification. So what we embarked on this journey is that why? Why is the bias and rich sample giving similar answers as population-based million dollars of studies? And then the other thing which really bothers me is that Michigan is one institution. And you have, you do not have, we do not have like NHS in UK that all the electronic health records, all medical encounters of a participant is recorded in the same system. So it has lots of gaps. So we do not really know what's happening when we are not seeing a participant. So I wanted to really think about with my postdoc, Dr. Beasley, that what happens when the selection and misclassification are at play together? And I want to show you some progress that we made. So the two questions we are trying to understand is that we are not seeing the complete data. Maybe things and diseases are happening in these question mark areas, but we are not capturing them. So in misclassification language, it will say that there is truly a disease, but we are not capturing it or less than perfect sensitivity. It can also happen that the disease is truly being reported but not there. So there could be less than perfect specificity as well. So phenotyping or coming up with who has a particular disease is a very complex process in EHR. And this talk is not about that. We use multiple auxiliary sources, structured and unstructured content of the EHR in order to come up with who has certain disease. Many times just the ICD codes or the disease phenotypes are used, but most people use very complex and sophisticated algorithms to determine which using all parts of the EHR, including treatment data, including prescription data to come up with a phenotype assignment which is much more precise than just using the ICD codes. So still they are often noisy because you are not getting to see certain variables and certain reports because we do not have integrated EHR. And then thinking about selection bias, you really have to think about that question I posed. What is your target population of inference? Am I trying to generalize my results to Michigan population or to the US population? Or is it to just Michigan medicine? So we have data on people we approach for consent, but beyond that, what kind of data do we have? Do we have individual level data? Do we have summary statistics? Because there is no free lunch. We have to have some data on the external population in order to map my internal analytic sample to the external population. So here is, I want to really convince you that when misclassification and selection are at play together, all our classical intuition, even about measurement error is actually off, off the statistical intuition ballpark. So what I'm showing you is a very well known association between gender and biological sex and cancer. Women are less likely to get cancer and cancer as overall phenotype. But when you actually select your sample based on disease and gender, then you can end up in a completely different association from that left-hand side blue. You can go to the right-hand side red where you are saying that women are actually have higher odds of having cancer. So it's not really attenuation towards the null because if selection really starts to depend on what you care about, that variable, then you and also the disease, then you can end up with massive bias and incorrect conclusions. So how do we think about that? As a old hat statistician, as I mentioned, interested in epidemiology and interested in misclassification and sampling, of course you put together models. And you put together models so that you can relate the selection mechanism and this observation or the contamination mechanism with what you really care about. So what our framework is that D denotes the true disease status that we do not observe. D star is the contaminant EHR derived disease status after your pipeline. S is the sampling or selection indicator of being in your study. And then there are different parts of covariates that we are going to talk about. Some covariates, particularly W, is informative about the selection. So we need things which are going to explain selection. Then for the sensitivity and specificity mechanisms, we also need covariates X and Y, which are informative. Remember, we do not have gold standards here. And then what we are really interested in is the relationship between D and Z, which could be the genetic odds ratio that I showed you and motivated this. So now what we have are models and you might question models and we are going to come back and criticize this as well. But we need some thinking framework on studying mathematically how selection and misclassification is actually affecting our inference. So that I can go back to the question that why? Why were those biased sample estimates concordant with the GWAS population based estimates? We need to know that. When would that going to be true? So what we see here is that different components of the misclassification model and selection model. And I don't want to put too many equations here. Different covariates are informing different parts of the model that there could be overlap between these covariates as well. Our ultimate mathematical goal is that the model that you're after which is with the true disease variable D and a covariate set of covariate Z and the model that we fit. The model that we fitted in all of those genetics papers with contaminated variable outcome D star and only on the sample data. We just did this and we sort of blind faith assumed these theta tildes are going to be close to the true thetas without thinking about the selection without thinking about the misclassification. Now I want to relate the two through these models of X and Y. And this is where I think this is if you are like distracted because it's very hard to pay attention to people who are in a virtual in a hybrid conference who are virtual. This is the key slide. And this was sort of beautiful to me as a student of epidemiology and studying case control studies for a long time could actually establish a relationship which ties the model that we fit with D star and S with the model that we are after. And that equation does not look too complicated which sort of like actually starts making sense that where are the participation in terms of specificity and sensitivity are going to play into this equation. And there's an offset term. If you are doing case control sampling and outcome dependent sampling you'll recognize that sometimes we have this offset term but they don't involve the covariate they sort of get involved, absorb in the intercept and that's why we can do estimation in case control studies. But what we see is sort of really a transform logistic type model syncing in with the sampling variable that a mathematical framework to study the effect of sampling which is coming out as an offset term and then the two functions of the misclassification also appear in this equation. Now that we have this we can, it's not, it's still incredibly challenging to estimate those CZ, BZ and RZ it involves a lot of thinking and careful mathematics to go from these models and many times it's actually condition and probability and Bayes theorem. So what we were able to do is to come up with algorithms which really tell us how to estimate to decouple the problem because we have lots and lots of unknowns so maybe we can fix the specificity parameter and try to take care of sensitivity first and then we can probably ignore selection bias at one part and then come back to selection bias. So the good thing was that you'll see that if there was no misclassification this result sort of makes sense because if there was no misclassification then these numbers, the specificity will be one and the sensitivity will be one this will look like the logistic regression and similarly if this sampling did not depend on covariate it's looked like an offset term which is free of Z. So it actually matches and parallels what we believe is the true relationship. So that was kind of interesting and once we had this relationship you can read our papers, I'll share the references in a moment that you can actually come up with a transform logistic function through which you can start estimating the corrected theta from the naive theta and there are different ways of doing it in absence of gold standard data and there's a lot of literature on that. But then you put together this misclassification literature with the selection literature so at the end of the day when you found this relationship we did a lot of simulation studies and we were able to see that if you do the transform logistic link you are actually able to correct that bias in the log odds ratio and odds ratio that I showed you for cancer and gender it actually gives you reduces the bias. But then what about selection? What about selection and again how to deal with the selection bias you have to really think about the selection mechanism how someone is ends up in your analytic sample very, very carefully. And so what we did there is that there is a lot of literature now in combining a probability sample with a non-probability sample. And so if we have the target population if we have some individual level data then we did inverse probability weighting. But if you're not so fortunate that we do not have individual level data on the target population then we had to resort to suboptimal methods like post stratification and raking which only has marginal summary statistics but not individual level data. At the end of the day it was still a very complex estimation algorithm because you have all of these three things unknown and many times they are not identifiable because you do not have enough data. So we had an iterative process of getting the weights with a fixed misclassification probability but along the way lots of mathematics and a lot of computational tools really help us to come up with a solution. I wanted to show you that not all GWASs are free of this like MGI is not great for all GWASs. When we looked at AMD age-related macular degeneration as an outcome not cancer we saw that MGI log odds ratios are far from the consortium odds ratios and the Lin's concordance was 0.55. So the question was can we get to closer to what we had in the gold standard but there is as I said that there is no really free lunch but there is reduced price lunch so we could improve the Lin's concordance with the gold standard after applying our methods to point from 0.55 to 0.7 but it's not completely corrected at all. So I just want to mention that in this world we have to accept that probably we are never going to get to perfect inference so we should really promote the concept of less imperfect inference because less imperfect is all you can do with this data but you should also definitely try. So we developed a software, a package Samba we had many cool names for all the packages that I have heard about since morning and so this is cool one Samba selection and misclassification bias adjustment which has complete inference including standard errors and P values please use them and share your feedback with us. So at the end of act two, what is the takeaway that we actually were able to in the misclassification literature we did a lot of work where the literature is novel because we did not have gold standard data for sensitivity and specificity and then we tagged on the waiting piece with the misclassification literature. The reason why GWASs were largely okay I showed you the AMD example is that when we did the math we realized that selection mechanism is often not related to the value of a single SNP or a particular marker or an omics data they're not related. So you break that association arrow and that's why often selection does not influence the relationship between the genetic the particular SNP and the disease outcome. Misclassification could also be a bigger issue for cancers versus AMD. So we have to really pay attention to these things and in this case I'll argue that two plus two or one plus one is larger than two because the two literature existed but we really needed to study them together. So these are some of our publications on this issue the statistical publications and with that I'm going to move on in the last two or three minutes of my talk I want to show some of the COVID examples where we benefited from having this work done. So the last part about COVID-19 and electronic health record I have been extremely disappointed by the lack of integrated electronic health record data in the United States. If you see the impactful studies that came out in real time about vaccines about variants they have been from UK, Israel, Denmark and all of them have great data integrated data systems where the vaccination data, the testing data the clinical outcomes data, the infection data are all integrated and you can get to that in real time. In US it's pockets of places and systems but not really at a national level. So it has been really, really very difficult to understand what is truth and what is a generalizable finding and all of these studies also show you that basic statistical training in terms of study design, matching, addressing confounding selection bias, meta inference these are all very, very important. So because we had COVID we had done all of this work with cancer and other phenotypes when COVID happened what we did was again we are looking at a disease risk model because our outcome now the D becomes COVID and then Z kept changing sometimes we are interested in genetics sometimes we are interested in race ethnicity sometimes we are interested in socioeconomic status or pre-existing comorbidities. The outcome also kept changing whether it's test positive or infection or is it hospitalization or is it mortality or death so it kept changing and selection was very prominent who is getting tested particularly in the early days of COVID whose data are you seeing? So we had built a lot of models on who is getting repeatedly tested so the probability of selection or probability of selection of the into the sample was being built. So with that in the early days of COVID we were able to study differences in COVID outcomes across race ethnicity. This was a paper which was in 2020 still in early days and what we saw is of course that there was a disparity in terms of COVID outcomes this is very well known but you have to do proper adjustment and proper adjustment in terms of many people could not do it they were just publishing the unadjusted odds ratio and we can debate what is the proper adjustment but what we noticed is that even one year later where COVID outcomes that you can see the height of these bars have gone down dramatically thanks to vaccines and treatments but still the reduction has not been equitable across all communities of color so I think that this mechanism of having this method set up this database set up helped us to delve a little deeper into some of the profound questions and again this is another example where we looked at we did not know who is at high risk of COVID related severe outcomes so we searched agnostically throughout the phenome and we looked at what contrast in terms of the past THR who is at high risk of getting hospitalized and not hospitalized there have been many many many models for risk stratification for COVID outcomes since then but this was very early days and we could study stratified by race and ethnicity do we see particular comorbidities in the past electronic health records showing up across ethnicities so I think just getting data together harmonizing and trying to get better will never be at a population-based level I have accepted that I have reconciled with that truth but still trying our best with correcting all of these errors is incredibly important so there are lots and lots of projects ongoing in my lab particularly long COVID this is the most poorly measured phenotype that you can think about about lots and lots of work going on with repeated testing data how do you think about booster effectiveness as we are getting first and second and now probably an Omicron booster in the fall how do we know who needs it and whether it's giving us additional benefits or not so I think that this has been a fantastic time to work on this data and as a takeaway I want to mention that having the right tools and the right team and all of this infrastructure setup was very crucial to do this work I see a lot of things on Twitter just criticizing academia that industry is offering a lot more energy a lot more money and so on but I have to say I have to defend that I do not know of any other profession where I did not have funding I did not have any resources but overnight I just decided that COVID is very fascinating to me I want to work and study and analyze COVID I think that working with EHR and such massive data set you make a lot of mistakes so it's very frustrating but it's also fascinating I do recognize that I'm a data dreamer but I'm not the only one you have been listening to me for the last 40 minutes so you are part of that dream and I do think that statisticians have a lot to offer to major public health and healthcare issues and I'm so fortunate to be a part of this community so with that I want to thank you all and also thank my funders as well as every participant of Michigan Genomics Initiative and the Precision Health Initiative and the organizers of Bioconductor so thank you so much Thank you for a wonderful talk Dr. Mukherjee Do we have in-person questions for Dr. Mukherjee? The microphone, we have several in the room You've got one question online from Gabriel Odom Can we take a stratified random sample from the EHR or use a virtual twins approach? You're muted Dr. Mukherjee Sorry, that's exactly what I mentioned that there is so much work that can be done with the digital twins and matching approaches and just think about your samples sometimes not using all of it is actually better so you have to really that's why I was really proposing that there is re-institution of study design and sample survey courses in our programs not just go overboard with machine learning we need that too but we need to think about how is data made because if we don't do that then it's going to be perpetual bias and study design need to be taught in computer science departments as well Absolutely right So the survey sampling has always been kind of a dark art even within stats departments how do you propose to address this especially places that might not have as the depth of resources that somewhere like Michigan School of Public Health does? So I think that I'm very lucky to be in Michigan because I took a course from Rod Betel like two years ago where he made us read Namens like initial paper with finite samples calculations it is incredibly hard sample survey is very hard for me but we persevered too and also because we have the Institute of Social Research which is a leading sample survey organization so I'll admit that I can actually knock on the door of Rod Betel, Mike Elliott and say what am I going to do but I do think that thinking about selection bias with big data is incredibly interesting and important area and there are some things which are informative of why you see an observation right for example like you know number of visits that the patient has historically whether they have access to insurance there are predictive covariates so I do think that we need to do some modernization of our sampling and design courses and make it a requirement so Michigan actually in our health data science program that we have sampling is a required course and we teach it jointly and I do think that we need to recruit faculty in those domains yes So not entirely unrelated to the previous the Michigan Department of Health and particularly its nexus with the Michigan School of Public Health is if not always cozy at least fairly friendly we have a question from I believe Sarah Stankiewicz asking she says I'm in Florida how helpful or supportive was the Michigan Department of Health or I guess the DHHS I understand this might be a little bit more difficult to pull off somewhere like Florida but can you comment on how generalizable this is at the state level or the province level? Yes, so I think that what we did is for many of this so we have a paper coming out exactly talking about because you can prove all of this theory you can even have the software but how do you get this external sample data that's really key to be able to do this like you know mapping and so in many cases we have used the breakfast data of ERFSS we have used in Haines we have also worked at Michigan Department of Health Michigan Medicine has very good connection with Michigan Department of Health but even with that we cannot get the individual level data we can get margins in total so we have to be prepared to do some of the methods which can only use joint distributions margin but not individual level data but that's the tricky part that's why I said that we can put forward all of these methods and beautiful identities that's definitely a first step towards understanding but estimation is still 10 steps away and you may not have enough data to correct for all of these biases similar question unsigned but I'm not mistaken but noting that compared to the NHS or a typical European or even Canadian health system there's a lot more fragmentation in the US so there's potentially ascertainment bias at the level of counties or individual systems plus on top of that we have large systems like say spectrum in Michigan where they have their own interests and how the data is used and what it's used for do you see this being insurmountable relative to a national health care system I note that for example ascertainment bias in the UK Biobank appears to be more severe than in all of us so maybe it's not always bad that they were behind the curve in terms of data unification but do you see this as a pros and cons or just pure cons? So I think that I really think that you know some of the studies in COVID particularly have come out of the VA have come out of Kaiser Permanente have come out of a large data system where they have harmonized data so I do like the fact that the UK Biobank is a special example of half a million people but if you think about public health England on NHS data they were able even Israel Israel's data is actually through Clalet Health Systems which is a large health care provider which has very good coverage but they're extremely favorable to statistical research and scientific research and producing things in real time what I saw in US yes we are like right now there are a lot of efforts for example the recovery trial and also different consortiums studying long COVID but we never can get our act together in real time it seems to me and so even with the variants right so even with the Delta variant I am amazed that all the originated in India but the sequencing data and the vaccination data and the clinical outcome data were not linked so the initial papers all came out of the UK so I think that this mapping and also seroprevalent or when as did this random sampling of the whole country this is also possible because of the geography of UK as well but NHS I do think that I'd rather have a national integrated data system where people are willing to partner with scientists and it's not really just protected data for the government you mentioned the VA do you see that as our sort of the closest we're ever going to have to a national health system in this country? But it is a very selective population right they have done a lot of work in terms of the VA studies on COVID vaccine effectiveness have been very carefully done with lots of controls and selection of you know a contemporaneous control as well as historical controls test negative design various variants of that they could do it because they have the system but it is a selective population so we need better data on people who are invisible to us if we train our algorithms on data on visible people we are always going to cater to visible people Can you hear me? Yes, I can hear you Very interesting topic in general very fascinating pertinent one I've been interested in for a number of years so two-part question and when you were drawing out and those key slides and everything what about the causal mechanisms a lot have been written about taking a biased sample and trying to blend these together have you looked at it from a causal perspective in terms of some things you can get an estimate from the data and some things you really need that knowledge and writing out the graph about the causes? So that's what we are working on right now this has nothing to do with causality this has like an observation model we did not do a causal treatment effect but I do think that what we are trying to do now particularly with vaccine effectiveness study to emphasize you know causal inference we have done propensity scores and we have we there is a model-based approach but design is also very very important that we have seen in many works so we are trying to really bring that design-based causal inference flavor into this work and I think that you are absolutely right that's the next step there will be lots of other things in the DAG in terms of collider bias the selection variables and the treatment variables and we have to think about that this sort of points to that because Z is our variable of interest and what you can do is honestly based on how is Z correlated with W or Z the misclassification covariates and the selection covariates and what kind of conditional independence can you assume? So you can decouple some of the biases under those conditional independence assumption and there are some cases they will be still entangled and then there you cannot identify and decouple this is exactly pointing to some of the you know missing data and the causal inference literature exactly the map is exactly similar Yeah and the follow-up on that then it's interesting you were picking genetics as an example does that give you any special ability as an instrument variable with the genetics being nothing causing that although we could talk about population biases and sampling biases being present in different countries but I'm curious does that open up any doors that you're going to leverage that you wouldn't be able to leverage otherwise? To be honest genetics Yeah to be honest the genetics example started because Gonzalo believed that we'll be able to learn like odds ratios and I believe that we cannot so it has started almost as like intellectual bet that no this cannot be mimicking population based U.S. studies and so I was proved wrong in some cases but it is not always blanket truth that was my point we have to learn under which assumptions you can recover these estimates so that's how it started because the core elements of MGI were electronic health records and genetics but now we see the advantage of having that right because the germline susceptibility even having polygenic risk core as a confounder or covariate we can do that because we have automatically stored that in our database probably normally if we don't have that as a covariate we not use that but because we have done all of this work germline susceptibility can figure out in our models but you are very right that we can talk about like there is not too many confounders of that genetic odds ratio that's why you can retrieve it similarly like when I decide to join a study or surgery sometimes is not related to a value of a single nucleotide polymorphisms on my genome that's why that association between Z and W is very weak and that's why you can retrieve it so but if the associations were stronger if we are looking at a like a gene environment association and so on then you are not going to be able to retrieve it depends on which polymorphism or the polygenic risk or you're looking at and what is your selection mechanism in this case they were automatically like they were very weakly correlated and that's why you could retrieve it Thank you Thank you Thank you for that question Do you have time for one last question from Ian Smith or we're running short here but it's a reasonably good one about balancing the cost high quality data and sampling with the with essentially where's the exploration exploitation tradeoff and how do you determine it we all know that garbage data produces garbage output but we also know that no data produces no output so somewhere in between there there's a tradeoff do you have any work upcoming on how to determine that how to determine that that tradeoff and the optimal balance point for it in these types of studies Yeah, so this is a great question and I'll point you to Zhao Liman's paper where he actually tries to compare the information content of a random sample of size hundred versus a bias sample of size million how do you really compare and he has a very nice paradigm of of actually studying it and it all depends on if you have a self selected sample what is your like contamination factor of the what you are trying to study and your selection mechanism but I believe that you know the figure that I showed you that there is a narrow credible narrow credible or confidence interval the bias is not that much obviously there is information I do want to use the HR data I learned a lot during covid there was method in this madness when you look at fee was plots when you are able to design studies carefully you want you are actually able to learn something and one thing I learned is that maybe the absolute numbers of prevalences and things like that are absolutely wrong but there is a relative thing that you can still learn and so this is something that I think that we are writing a piece on really building on Jowley's work because Jowley's work is not in a regression based model that how can you when do you think that this is the information threshold has been so much distorted the bias the selection bias is so much that it's useless that paradigm of thinking about it how to compare the information content I think is very useful and actually he shows in a new recent nature paper that the Facebook Delphi survey gave such an overestimate of vaccination in the United States and when you look at like Sam carefully done sample survey much smaller size how you can retrieve that information better so I think good quality data we need that in order to even understand our mistakes even if it's not that large scale thank you so we're out of time thanks thanks again for our wonderful and detailed talk and some stimulating discussion I guess everybody give a warm adios to Dr. Mukherjee because it's time for lunch. Thank you. Thank you so much. Thank you.


Trying to get back to the basics of great products power comes from sharing information. I try to convince people to slow down Hello everyone, welcome back to the soaked by slush podcast My name is William from that ball and with me in the studio is is a crowd. Hi, sir. Hi, William I almost looked at the camera almost close. Let's go Today we have a an interesting guest once again as we usually tend to have on this show So welcome to the soak by slush podcast, Philippe Botteeri Thank you my pleasure to be here great to great to have you do you want to tell listeners Who you are and what you currently do Sure, so I'm a partner with Excel. So we are a global venture firm So we started about 35 years ago in Silicon Valley. So this is where we have our roots and culture Doing early-stage investments in technology companies in Silicon Valley and over time we realized that Silicon Valley did not did not have the monopoly of innovation. So we opened an office in London 2000 so 21 years ago. So we've been on the ground here You know for more than 20 years and we also have an office in India So today, we are one of the very few early-stage You know venture platform with a truly global Network and coverage and this is is that network that we give to our portfolio company and hopefully give them a strong competitive advantage and You know from a stage 10 point So the core of the business is you know kind of seed series a early-stage investments over time We also have added gross capabilities So we have a global girls fund which can you know invest At a later stage in the business. So anything from you know, 40 50 million dollar up to a hundred hundred and fifty million dollars And the three key areas where we are investing overall is anything around digital consumer anything around fintech and online payments and anything around You know cloud computing sass and analytics and security, which is what we're gonna Talk about today. So so we've evolved into a full-stack platform So we anything from seed to to late-stage and and we do that on the global basis Yes, can I also ask what is your background? What do you do? Yeah? So I joined and that I think ties nicely to do this this discussion So I joined Excel 10 years ago You know before that I spent about a bit less than 10 years in the valleys So I was with another firm called Bessemer Venture Partners And before that I was with you know with McKenzie both in their power to office and I started my career in With them in Europe as you can guess from my accent and from the other side of the channels I was born and raised in France But I think what so the you know, the question is like why did I move back from second body to to London in? you know in 2011 and So I was Kind of lucky to start in venture 15 years ago. So I was in 2006 Basically, which was at the the beginning of cloud and sass like when I was starting in When I started venture and in the valley I think the world what the question was like is sass and cloud the fad or Is that is that really gonna gonna last and I think I wrote my first blog post Which is you know why I disagree with Tony Zingali who was like a you know, well-known CEO It was CEO of Mercury interactive at the time and basically he was doing a keynote at a conference saying that basically sass, you know would disappear And so maybe I wrote my first blog post in one now. I think sass and cloud is actually the future Of software so from 2006 to 2011 You know at Bessemer I helped build, you know, their cloud practice You know investing in genuine sass companies that cornerstone demand a loco intact And in 2010 investing in a company called Criteo Which was kind of the first picked it up play around kind of online advertising around retargeting And it kind of opened my eyes about what was happening in Europe. I saw this great company out of France They were growing from 5 to 15 million In a year and no one was repaying attention to it because no one was really looking at Europe And then a year later, I just got the call from Excel and I was like, wow something seems to be happening in Europe like We're seeing stuff there that that you know shows that it's it's starting to happen I wanted to be on a global platform. So I thought you know You know if you want to build global businesses you need to be global ourselves So that was very important to me and I really like the team there So I made this the you know the move back back my bag Lately in London me 2011 and then at the time I was like well a lot of what I did in Convally was around, you know cloud computing and sass but at the time like 2011 2012-13 and 14 a lot of the action in Europe was around the kind of the first generation marketplaces and consumer services, but not much was happening on On the cloud front. So for the first three years, I invested in companies like, you know Like our and fiber on the marketplace side And I was still looking for these cloud companies and the first one I found was people doc in 2014 and then in 2015 I invested in you know, Algolia and and Dr. Leave and started to see that a new generation of founders was you know building these great cloud companies and then looking back in 2016 like Seven out of ten of our last most recent deal that Excel was actually cloud company I was like wow something's changing now and and we just need to You know, we just need to tell it, you know, let's tell that to the world Let's show that Europe can actually be a greater persistent for for cloud company And that was the burst of the Excel Euro scape. So we did the first version in 2016 Which was kind of this overview of where? You know the state of the cloud for for Europe And you know, we kind of added You know, including the list of what we think where you know kind of the top hundred cloud companies out of Europe at the time and You know when you compare the number of where we were in 2016 and where you are today. It is just night and day, right? I mean and just show how much the ecosystem has accelerated Just I mean just give you a couple of stats. I mean in 2016 there was like around like a billion dollar Invested in the entire cloud ecosystem in Europe for the year, right? in 2021 the largest round which was Silonis was in Europe was a billion dollar So, you know in basically five years went to all the money investing the entire ecosystem to just one deal And the one billion became like 30 billion, you know, you're today in 2021. So, yeah Difference isn't inflation Yeah, I mean it was been a steep inflation All right. Yeah, but you mentioned the Axel Euro scape It's a report or whatever you want to call it like a like an overview of a lot of this sort of data and and and related to these these things and you've just actually released the 2021 version of this Axel Euro scape on the 19th of October. It's it's available online I think we're gonna put the link somewhere in the description so people the listeners can go read it, but Yeah, should we go through it? There's a lot of interesting stuff in in the in the text here Yeah, and maybe just to give some You know some background around the acceleration that we're seeing certainly Around, you know cloud I think in Europe, but also globally and then we'll we'll compare Later in the discussion, you know, how's Europe perform against the US but overall I mean if you think about the market, there is roughly, you know half a trillion dollar or a bit more of software spans and And you could say that, you know, barely get less than probably one-third of that today as Migrated to the cloud So there's still, you know, if you look at the this migration from on-prem software to to cloud we're still at still the very early innings of that of that transformation So this trend is not something where we're saying well, you know We are the top of the S curve and most of the the workload have moved to the cloud No We're still at the very beginning and I think that's why you're seeing this, you know, this very long trend like very deep trend Steel, you know pushing a lot of this public and private company Overall, I think if you look at the the public side, I think what's interesting is that today You have basically 10 software companies that are worse north of a hundred billion dollar in Market cap and out of this 10 company, I mean together There were around four billion slightly more than four billion dollar and more than half of this, you know 2.2 billion just one company. It's Microsoft. Sorry. Do we might do in four trillion? I think he's Yeah, four trillion. Yeah. Yes trillion dollar. Yeah, it's correct And Microsoft is, you know, it's 2.2 so slightly more than half of that in just one company and Microsoft as added in market cap $600 billion in the past year This is more than the nine other companies combined So clearly Microsoft has been the big winner of you know, this Migration to to the cloud and that's what explained their success because in the past You know with when Satya, you know to the help of Microsoft you really refocus the company On Azure and Office 365 to really push all their workload to the cloud and the numbers out there You know to prove that it was clearly right and then on the other end If you look at the only company in this 10 company with market cap as declined in the past year That's ACP and why because I mean you can argue that you know That cloud migration has been totally unclear, right? And they haven't really been able to capitalize on this core trend that we're seeing in the software ecosystem Yeah, exactly. No, I'm one happy Microsoft investor myself, but it's it's very true It seems like since this has been happening quite quite for a long time already And it's easy also to kind of draw parallels between cloud computing and and the whole, you know growth of the internet and and platform economies and everything Related to that It seems like this has been going on for a long time that we should be finished by now There should be no gas left But as you said, there's there's so many companies that still haven't made a transition and there's so many countries Still coming online in the coming years It seems like this will be a massive advantage also for for your for instance now that is According at least to your report It seems to to be catching up quite quite well Actually one question I had Before we dive deeper into the report you had Europe and Israel kind of pooled together as one one area and then comparing that to the to the US. Is there some Some specific reasons for for pooling Europe and Israel into like kind of one one market or what's the logic behind it? Well, I mean we could call it the kind of Mediterranean region so extent I don't know. I mean, I think we're including it because Israel has always been a very active market in terms of venture Overall and and says always for us be part of that our, you know, same ecosystem And there are a lot of ties between the Israeli in European and US ecosystem So that's why I think we're you know, we're including them in in this In the same bucket Yeah, exactly, but yeah, if we if we go back now to to 2016 You mentioned the funding round or the funding size of private money coming in That has grown massively the market caps Seem to be growing. What other stats do you feel like are the most relevant to point out that? Europe is is indeed catching up now to the to the US in in terms of cloud computing Well, I mean, I think when we look at the progress that Europe has made in the past five years I think the main question that you know, I think the ecosystem has earned the right to ask is You know, are Europe and Israel on paths to global dominance? Historically Europe and Israel have always been considered in the shadow of kind of the the US venture ecosystem and you know, if you look back, you know, 10 years It was all about the kind of the Copycat and and and Europe trying to look at what were the successful internet or marketplaces You know in the US and try to copy that in Europe. I mean, that's was a playbook of rocket internet, for example So it was not about innovation. It was really about, you know, let's try to copy and adapt to the local market What someone in the US is done now, and I think today we're very far from that. I think today Europe has really shown that it can create Category leaders, it can really innovate and not just copy what what where what was happening in the US And if you look at some of You know a few facts here, if you take the largest cloud IPO in 2021 That was actually UI path Which was a company company out of Romania So Europe at the level of the global economy So Europe at the largest cloud IPO This year, I mean, this is something like if we have told if we have said like four five years ago in 2016 Like Europe is going to generate in five years from now the largest cloud IPO You know, no one would have believed it like it was just not meant to happen and it is actually happening If you look at, you know, the fastest Company to get to unicorn status The number one is whiz, which is a cloud security company out of Israel The number one is hoping which is a virtual event company out of London. So two of the top three fastest cloud unicorns You know and just coming from our region right in Europe and Israel And if you look at the the top three funding round I mean, it's certainly Europe has always been behind in terms of the size of the round, etc And and today actually, yes, I mean out of the top three us as to And you know, the largest round was in that data breaks at 1.6 billion and the largest in Europe was still nice at 1 billion but Overall like we're really catching up like we're talking about billion dollar round like okay It was 1.6 1 but it's still like it's massive round and when you look at on average Kind of the top rounds happening in the region. It's not that there is like a big a big difference So you start looking at these data and you say wow, you know a lot of very interesting things are Actually happening in in Europe now The question is the quantity, right? So if you look at the the total amount of Of, you know, that has been invested basically. Yes, you're the US is still Still larger than than Europe Europe is, you know around 30 billion and it's about two-third the US about 46 billion So we're still doing a bit less. I think if you look at The number of IPOs as well, I mean there's was 21 IPO in the US 11 in Europe For this year on the platform. So overall, I think in volume the US is still Larger but comparable is not that they're you know, 10x larger which was the case in 2016 now, you know, we're kind of two-third Of the US But what's most important to me is basically the quality of You know of the outcome, right? And so if you look at the number of IPO, there are more IPOs in the US than coming from Europe and Israel, but Actually, that number is going much faster because you know, if you look at You know, last year the 21 is a 30% increase in number of IPO for the US versus 11 is actually close to 100% increase For for Europe and Israel. So, you know, it's growing faster The amount raised in IPO is similar the average market cap at IPO is similar and the average Brought new multiple of this company going public is also similar If you want to listen or if you want to Look at the report at the same time while you're which part should you be looking at right now on the report? Yeah So right now we are, you know, I'm you know, most specifically on the section about Europe and Israel and are they on the past to global? I mean, I'm sorry slide 29 and so I would just you know, just saying that yeah in terms of volume You are still in front but in terms of quality of these public companies that are coming out of Europe They are actually very very similar and and if you look at the You know the private side and the unicorn Basically, the you know, it's the same story So there are 43 new unicorn out of Europe versus 74 in the US But the median value of this unicorn is very similar the median capital raised is similar The number of unicorns that I've raised big rounds of $250 million is the same And the time it takes on average to become a unicorn is similar So thinking what we are saying here is both on the public and the private side things are very Comparable in terms of the quality of the company the access to funding etc. And so to me I think that is the most Important point because at the end of the day, it's all about creating this category leaders That's interesting. Yeah, there's a lot of it's fun because we've had this discussion this conversation On this podcast before about Europe and the ecosystem and everything. So it's nice to have Interesting concrete data here to actually demonstrate the trends that we've been discussing in previous episodes but actually to follow up on what Will William asked about the the choice of sampling still like I think about Europe and Israel now Okay, first of all, I understand that Europe is in a country whereas Israel is a country So there might be some differences there But I guess the interesting follow-up to Williams question would then be like is is Israel making Europe look significantly better Compared to the US or what is how do you think about that? That's that's a good question. So yeah, I mean to me, it's interesting to put Israel in in in context, right because What happened in this ecosystem like if we look at, you know, 2000 when we started in Europe I think a lot of most of our investments were either in the UK, so in London or they were in Israel intelligently so the Israeli ecosystem has been here for You know very long time and I think what we've seen in the past 20 years is that Europe has moved from You know kind of being bipolar around this two ecosystem to now being a combination of You know 20 plus hubs and cities where we're seeing great great innovation I mean again, you know five years ago if you had said that Bucharest would generate the biggest Cloud IPO of 2021 like, you know, people who has erased their hybros, right? But it actually happened and it showed that really innovation can come from anywhere in the region and so basically what happened in the past 10 20 years that you know all these types of evolved in In Europe, but Israel, you know has continued to evolve as well And has been doing very well, right? I think you know, you'll see Basically a slide in the report saying that Israel is emerging as you know, the unicorn cloud factory of the region And it's true that if you look at the roughly 80 unicorn that we have in Europe in Israel, Israel is one third of it And it's also one third of the new unicorn that that we're seeing That we saw in in the past year And and we look at an interesting statistic shot. I don't know how meaningful it is, but it's kind of interesting Which is to look at the unicorns per million people in the country And by far like Israel is like super high at you know kind of 2.9 Versus, you know, most European countries from the Nordic France or Germany are like, you know, 0.1, 0.2, 0.3 So it's kind of an interesting stat to see to show that actually Israel it's kind of And and normally some extent In kind of the unique unicorn creation World but the question is so what what is happening in Israel? Like what what is this? the case To me, this is a combination of different things, right? I think the first one is that Israel as You know some very a lot of great talents and the the source of these talents is one The army and the 8200 units. So they're very strong But if you just like you think about the army as kind of sea, you know lands Uh And air and they have like a fourth branch which is kind of this digital cyber intelligence Etc. So this is a great source of talent. The only thing that happened is that Given that the Israeli ecosystem has been around for a long time has been great at you know, you know Especially in the security space and the software cloud space. There's been a lot of acquisition there So all pretty much all the big US technology company have made acquisitions In israel and therefore these acquisitions have evolved, you know in the past 10 15 years Into kind of full-fledged R&D and product center And that's also a great source of talents So that's I think the first the first thing that I put in is a great pool of talents in israel The second thing is that this talents Have the right skills You know, they have skills which really match The big and the big trends that we're seeing in in tech like all this secular trend around You know cloud computing around security around infrastructure around payments. These are all expertise are very strong in israel So when you combine these talents With uh, you know, there's circular shifts and the great match. I think it's um, you know, it's very powerful Um combination And then the third thing is that, you know, you have a lot of local seed funds Um, you know that have been willing to make like pretty big seeds And so you can see Company out of israel's raising like five 10 20 million dollar seeds So from the get go, they have, you know, very strong team with access to a lot of capital Um, and then of course you have, you know, all the kind of the global funds which are kind of Coming as well to to um to invest the big ground. So all these combinations, um, you know, together I think have led to really, uh Great things for for israel Yeah, exactly. That makes makes sense What about in terms of founders and operators? You you mentioned that there's kind of a new generation building companies Is the is it the same founders still building as in 2016 or Has the morning porridge changed along the way? What's different with with this batch of companies? Well, I mean, I think the first thing is is quantity, right? I think you have a lot more people coming out of um, in europe, it's always add, you know, some very strong, um, universities, especially around, um, you know, Coding i mean computer science, but also around the data and algorithm, which is, you know, Which are great skills in the current world Uh, but I think what we have thing also is a shift of Motiety and and now if you look at the number of students coming out of these schools who are interested into the startup world I mean, it is now becoming the, you know, a dominance, uh, majority, right? Because people You know, they're less excited about working in, you know, in banking or consulting or some of the big industrial group Like they're more excited by, you know, working for, um, you know for startups or actually starting their their own startup Um, so I think that's one thing around the talent pool is that, you know, it's really, uh, it's really been growing exponentially um, and the second thing is that the level of ambition now I think, um Is, uh, you know, it's really unprecedented like all, um all the founders They want to be they want to be global. Um, you know, they want to build category leaders Uh, and so it's, um, you know, I think for me That's been it's been very exciting to be in this ecosystem and be able to meet every day several of these founders and to see their Uh, their energy and commitments through to creating, um, these great companies exactly Do you think it's uh, you mentioned in your report that the The, um, the world is or the world of cloud at least is now, uh, flat So you think we are already at the point that it's You have a similar chance of succeeding regardless of where you're based In terms of being based in in in the us or in in europe or israel Um, correct. I mean, I think the, um Well, one thing which is, uh What is clear is that at some point if you want to be if you're selling into enterprise um software like selling to Enterprise company You um, you probably you know, your your largest market is going to be the u.s. Because for a dollar spending software 50 cents Is actually spent in the u.s. Um, so you need you only at some point to develop your go to market there, but in terms of engineering and products I think europe and israel are very attractive places Because to some extent there is, you know, less competition for talent than in see I mean everything is really right everybody Every entrepreneur speak to will will tell you that hiring is hard But I would say that, you know, look at See from the value in the concentration around san francisco when you look at europe where you have actually 20 different hubs Um, you know, you can you have access to actually a broad uh pool of talent And it's probably easier to build that great product team in europe Today for for all of these reasons So now at some point you need to go and build your go to market in the u.s But from a product and insurance standpoint, it's very interesting and that's why also we're seeing a lot of u.s. Companies, you know making acquisition in You know in europe to actually have access to to this talent pool and then there's uh I guess the final part of your your escape 2021 Maps out six Future trends we don't have to go through all of them if you don't Feel like it right now, but like what would you highlight? Which ones would you highlight from this? this trend A list of trends here that you see in 2022 and onwards I mean, I think to me if um, you know, I wanted to um, you know to flag three of them I would say that you know the thing the the number one and this one we flagged in in last year as well, but it's really the importance of automation for for the enterprise and what we're seeing right now is that With ai becoming more performance the number of use cases that you can automate increases because with ai you can automate a lot, you know workflow that are a lot more sophisticated And so what's been very interesting is that with time you're seeing that the potential To automate processes in the enterprise actually growing and I think this is a trend and it will keep growing You know for a long time. I think because it just feels that we are at the very early innings So you see company, you know, like, you know, ui pass and and silo nis Genesis like all the this new generation of companies which help automate this enterprise Tasks and workflow are becoming more and more efficient and and that is You know, very exciting Now if you look at, you know, the second trend that I wanted to to to flag here, it's security focusing on like cloud security feels like it's a very Long lasting and exciting trends and why is that? I mean, I think Um, historically, uh, all the workloads And a lot of the workload are still on-prem, right? But when when your workload is on-prem, you have a specific set of security parameter, right because you control You know, you have your your hardware and then you install your app and you need to configure it But it is a Some extent more simple environments When you move to cloud, suddenly everything becomes virtual and your infrastructure becomes code, right? Because it's all about how you code the different instances on on aws and and start this application, etc And so as your infrastructure becomes code The level and the number of vulnerabilities and the challenge of securing that infrastructure are very different And and you need to do it with a focus on code, but you need to do it also with a focus on configurations um, and so We're as more and more workloads are migrating to the cloud. We're seeing actually, you know, that's super increasing number um of uh, you know of challenges and You know, leading to the rise of the next generation of Cloud security company like, you know, we mentioned whiz and snake and a few others So that's I think the the third one and and the fourth one Which is also quite Exciting for me and this is something which is new in the urscape this year Is we have added, uh, you know a section in our top hundred around crypto Uh, and why is that because I think this year we have really reached the tipping point Tipping point for crypto where basically we are seeing it being starting to be part of the financial um infrastructure Uh, and we're seeing all the, you know asset custodian or the banks etc getting You know, I mean to provide their customer the ability to, you know, buy and hold and trade these crypto currencies And uh, and they're building so you you have a need for a new infrastructure layer For all this financials institution to be able to operate uh cryptocurrency Uh, and to me that's super interesting because it's like, you know, I described crypto today as probably as I would say the internet uh in the early 90s Uh, so this is just I think the the the beginning Um, and there is a real need for all this, you know core building block for the crypto infrastructure You know chanalysis is one of them on on the the compliance side and kind of you have fire blocks on the custodian side and Nansen on the data side. So It's been very interesting to see, um, this ecosystem developed in in the past year This is music to your ears. Really? Yes, no, but I agree. It's uh, it's definitely a trend to to watch out and it's very very early Uh, I think it's good to compare it to the early internet, but the only thing Uh, the difference is the amount of users at the same point in time, which is where crypto is is way ahead of the The internet at the same stage in the uh, or in the same at the same stage in the adoption curve So if the pace will be will be the same for for the coming years. It's going to be a very very interesting trend to watch for sure Thank you. Yeah, uh, it looks like from your smile that you uh, you've been holding A video for crypto as well. Yes. Yes Yes, but let's not dive into that too much But we maybe we'll have some some crypto episodes in the future as well But it's uh, it's very interesting to hear that many of the vcs We talked to are also looking at that much more than maybe a few three four years ago So it's it's becoming it's becoming more and more mainstream In in that sense. So it's it's interesting to see And also the the automation part is is something we discussed. We've discussed as well. It's The possibilities there to automate basically Anything in terms of content creation also is is going to change. I think many companies Quite a lot in the coming coming years Yeah, no, I mean, it's it's super exciting time to be a cloud investor Uh in europe right now. I mean, I think meaning every day all these founders and Seeing all that energy and seeing all the great products that are I mean Hopefully a lot of them building on the trend are described, but there's so many other things that are happening right now That we didn't have the time to to to talk about So I can't wait to see what's going to happen in in the next year I mean, it's going to be very interesting to see how You know europe is gonna You know catch up in terms of volume Uh, you know compared to to the u.s on the ipo from but also on on the funding side And um, yeah, we'll we'll be interesting to see what's coming next, but I can't be more excited about it Yeah, for sure. It's it's going to be really interesting to to see and Yeah, check out the report. We'll we'll link to to to that in the episode description There's a lot of interesting data points and it's a very positive story for once about About europe and its outlooks in in in terms of entrepreneurship and startup. So it's it's a good read in that sense as well So absolutely. Thank you. Thank you. Accel 2021 euroscape on the path to global dominance I like that little subtitle there a very cinematic subtitle Yes, very good. So thanks so much for taking the time to to run through it with us Well, thank you. Uh, william. Thank you. Zach was a great big great being here today Thanks. Thank you. Thank you so much. And thanks to everyone who tuned in see you in the in the next one And until then stay healthy and safe. See you. Bye. Bye. See you. Bye. Bye


Well, I would work as Paul Carlson here, a small workshop guy. On this wall here, I have a piece of plywood and then I have a lot of different braces that kind of stick out and are angled down so that the clamps can go in between those braces and not fall down and you space those out appropriately. There's 4,000 different videos about how to do those. Well, here's an idea I have for you. You've run out of wall space and I certainly sort of feel like I've done that. But I got more clamps I want to store. So what I did was I took some three-quarter inch pipe from Home Depot and some flanges for the end and I screwed those flanges into the studs. I would make sure you find studs if you're gonna do this. And then put a pipe all the way across the garage. Or in my case, I went part way across and then had a little elbow and then went up to the ceiling with another flange. Now I've got this pipe up here at just the right height. Make sure you put it at the right height so that you can put a lot of 24 inch clamps. I built a lot of cabinets and stuff so these are the perfect size for me. And you can store them up there and still walk under them and not hit your head on them. Now you can do a whole bunch of storage coming out this way. How many do I have here? One, two, three, four, five, six, seven, eight, nine, 10, 11, 12, 13, 14, 15, 16. 16 clamps right there. Always remember, you can't stop the waves but you can learn to serve. I hope you like this series of what the heck is short designed for beginners. Give me a like, give me a comment, give me a subscription and I'll share more little tips with you in the future. Small Workshop Guy, signing off.