Bring it now and we're live. All right, good morning, good afternoon everyone. This is Daria from Wikimedia Foundation's research team and I'm very excited to invite you all to our August edition of the research showcase. Today we have two guest speakers. I'm thrilled to have Sneha Narayan and Andrew Su presenting today. Sneha gonna speak first. She's gonna present her work on the Wikimedia adventure, an attempt to build and study onboarding mechanisms on English Wikipedia. And we're gonna switch in the second part of the showcase to Andrew Su for a presentation on the G-inWiki project. So without further ado, Sneha, the stage is yours. We'll be having a short Q&A at the end of your presentation and invite people to stick around after the entire showcase and there's also an IRC channel where people can follow the live discussion. Thank you. Hello everyone. Thank you so much for that introduction, Daria. Hello everyone. Thank you so much for that introduction. Hang on while I get my slides set up. Excellent. So hopefully I'm shedding my slides right now and assuming that I am, hello. So today I'll be talking about the Wikipedia adventure which is a gamified tutorial that is designed to help onboard new editors on Wikipedia. So this was a system that was designed and built by Jake Orlowitz and Jonathan Morgan from the Wikimedia Foundation. And they did this with the help of a number of Wikipedians. And we, the system was also evaluated by researchers from the Community Data Science Collective which is a research team consisting of folks primarily from Northwestern University and the University of Washington. And that includes Benjamin Macro Hill, Aaron Shaw and of course me. So the broad question that we try to address with this project is how do you mobilize and socialize newcomers in an online volunteer community? And this is a pretty important question because online communities that rely on voluntary contributions need a steady stream of newcomers in order to ensure that they continue to exist in perpetuity. So even when new people show up at an online community they go through an adjustment phase and during this adjustment phase they need to learn how to navigate the existing norms and practices of a community. And we call this adjustment phase the socialization process. And if socialization doesn't occur then newcomers often just get frustrated and leave the community. And this is a vexing problem on Wikipedia in particular. Prior work done by Aaron Halfaker and others has found that the number of active editors on Wikipedia has been in slow decline since 2007, largely due to the poor environment for newcomers on Wikipedia. And since no editor edits forever if newcomers aren't socialized and retained in large enough numbers we can expect this kind of shrinking. And essentially if we don't solve this issue of new editor retention it can put this really important public resource that everyone uses at risk. However, it's actually often pretty hard to be a newcomer on Wikipedia. And that's because over the years Wikipedia has become increasingly formalized and complex as a community. So if you're a new editor on Wikipedia there's lots of skills and even tacit knowledge that you need to pick up in order to be an effective contributor. So for instance, you need to first of all figure out what you wanna work on. And once you've done that how to interact with other editors there are wiki markup conventions that you need to understand and follow. And there's even also community wide philosophy such as the neutral point of view principle. And of course there's also finding reliable sources which as we know is a problem that extends beyond Wikipedia in this image. In addition, what makes this even more complicated on Wikipedia is that new editors are mostly expected to seek out all this knowledge on their own or at least with their own initiative. So this is an example of a welcome message that's sent to a new user on Wikipedia. And each of these links that you see here lead to a long essay which I'm sure many of you are familiar with. And this sort of orientation towards onboarding makes the socialization process quite challenging for many new editors. Just seeing an instruction manual like this soon after you create an account and start editing can be pretty overwhelming and also quite dry. So this like my co-authors and I have to ask would a more structured and engaging orientation help new editors contribute better to Wikipedia? And essentially this was the question that was the impetus behind designing and building the Wikipedia adventure. So for the rest of my talk, I will discuss the system design principles that we used to build the tutorial. I'll also show a few screenshots from the tutorial itself and I'll present the results from two different studies we conducted to evaluate the tutorial. So first we conducted a survey study where we surveyed a number of users who had tried the Wikipedia adventure out and essentially we found that they responded quite positively. They enjoyed the system and learned from it. And then so that prompted us to conduct a field experiment where we measured the effect of playing the tutorial on the aggregate subsequent contribution to Wikipedia. And spoiler alert, it turns out that that wasn't a noticeable effect from this experiment. So essentially that was no measurable effect from playing this tutorial on the aggregate subsequent contribution to Wikipedia. And we thought this was an interesting finding just because these two studies on the same system seem to tell us different things. So I'll follow up an explanation of these studies with a discussion of our results. So to start off with our system design principles, one of the things that we aim to do was create a structured orientation on Wikipedia. And part of our decision to do so was based on findings from organizational theory. Organizational theorists often distinguish between individualized forms of socialization and institutionalized forms of socialization. Individualized socialization is sort of like on the job training. It's often an informal type of socialization and it's initiated by the newcomer themselves typically. There's no set sequence in which the newcomer learns things and the experience of learning about an organization this way leads to a lot of variation between newcomers. And incidentally, a lot of previous socialization efforts on Wikipedia have sort of built on this type of socialization. It sort of expects that people will learn about Wikipedia sort of take their own path and efforts like T-House and Adopter User are sort of there to essentially be spaces where newcomers, like when they have a question, they have like a forum or a mentor that they can direct their questions to and they can sort of learn as they go. However, this individualized socialization tactic is contrasted with institutionalized socialization. And this is sort of more, this looks like more formal orientation. So for instance, like a week long college orientation the beginning of the semester is a form of institutionalized socialization. This is typically initiated by the organization. It's more formal in nature. The tasks that newcomers are exposed to are presented sequentially and the experience among newcomers in a cohort is often more uniform. And what we actually know from meta studies in organizational research is that overall the consensus from this party of work comes down in favor of institutionalized socialization. So a number of studies have shown that overall institutionalized forms of socialization have led to increased self-efficiency among newcomers, more social acceptance from the existing members of the organization they are joining and better retention of newcomers. So we definitely wanted to see what institutionalized socialization would look like in Wikipedia when constructing this system. And as an aside, it's institutionalized socialization isn't new to online communities. It's many citizen science projects use these approaches and as well as MMORPGs such as World of Warcraft that have newcomer only spaces that train people before they join the community at large. But we didn't just want the onboarding process to be more structured, we also wanted to ensure that it was engaging. So in addition, we drew upon the literature on gamification and learning outcomes. And essentially the consensus from this body of work is that when you look at the uses of different gamification elements such as achievements and badges, levels, narration and so on, when deployed in educational environments, these are shown to lead to positive learning and outcomes and increased engagement. So based on these principles, my co-authors Jake Orlowitz and Jonathan Morgan designed and built the Wikipedia Adventure, which is as stated earlier, a gamified tutorial that helps newcomers learn how to edit Wikipedia. They did this in collaboration with other volunteers from the Wikipedia community. And if you haven't heard of it or tried it yet, it's still up and running on the site. So definitely feel free to check it out. In case you haven't played it yet, I'll show you a few screenshots from the system. The tutorial is divided into seven different missions, each of which addresses a different facet of editing Wikipedia. So for instance, this is mission two, which focuses on learning how to communicate with other editors. And each mission asks you then to complete a series of tasks that you're prompted to do via little pop-up messages that you can see on the top right here. So for instance, here you're asked to edit a talk page and then the system responds by teaching you wiki markup and encouraging you to use it in a simulated environment. So once you're done with a mission, it then posts a badge to your user page. And this badge basically forms, it's meant to be encouragement for the new user, but it also serves as an indicator to veteran editors that the user that is being oriented on Wikipedia. It's essentially a way to demonstrate to the community that this is the person who's trying to learn, who wants to contribute to Wikipedia in good faith. So that's basically a little summary of how the system works. After building the system, we first decided to evaluate it through a user survey. And so to that end, we invited a bunch of new users through their talk page, a number of them played the game. And then again, we sent another talk page invitation asking people to take a survey and tell us a little bit about their experience. And we targeted the survey at English Wikipedia users who had created accounts recently, had made two or more edits and hadn't been blocked or given a level four warning, which essentially is sort of like a basic test of good faith editing. And the survey included questions about user confidence, user engagement and overall design satisfaction. So here are some results. We asked people to respond at what facets of editing Wikipedia they felt more comfortable with after using the tutorial and over 85% rated highly the ability of the tutorial to explain concepts such as making edits, viewing page histories, adding wiki links and the policy of neutral point of view. And over 90% agreed or straight as an editor or it helped me understand the other. Looked at user engagement in this, oh, sorry. In this case, we found that 80% of users highly rated statements such as it made me want to edit more or it made me feel welcomed and supported. And the survey also included a spot where people could give us open ended feedback. And then they said one user shared a statement that said the adventure was an easier and better way to learn the basics with UDS is trying to run it and just reading it. And this if you recall validates our decision to design for institutionalized socialization. We finally looked at design satisfaction. Overall survey respondents liked the gamification and design of the system. And one user specifically shared that they enjoyed aspects such as the challenges and badges that made it feel more like an educational tool or game rather than a lecture, as well as the way it recorded their achievement to date. And this as you can see validates our choice to use gamification techniques as a way to engage newcomers. So to summarize quickly, in general, we found that the survey results indicated that participants enjoyed the tutorial and it was overall well received. And also user responses validated the design principles that we used in order to build the system. So at this point, we could have declared victory and just gone ahead and deployed the system on Wikipedia. However, we wanted to know does the Wikipedia adventure affect subsequent participation by newcomers? And that's why we did a follow-up study with a field experiment. So in light of the positive survey results, we wanted to test the tutorial in the wild, that is to say in the environment that it would be used to see whether this intervention had the potential to reverse or slow the newcomer attrition problem that Wikipedia has. So to this end, we conducted an invitation-based field experiment that mimics the way that the Wikipedia adventure would be deployed on Wikipedia and measured the effect that it had on subsequent newcomer contribution. So specifically, we had a few hypotheses about how the Wikipedia adventure could affect new user participation. In particular, we hypothesized that after playing the Wikipedia adventure, a new user would edit more and have essentially contribute more edits to Wikipedia. We also hypothesized that they would increase the number of contributions to talk pages and also that contributions would be of greater average quality. So this is, in terms of the design of the field experiment, we identified 1,967 users to be a part of our study. We used similar inclusion criteria as we did for the survey study, which is essentially new users on English Wikipedia who'd recently created accounts and who had passed a basic test of editing in good faith. Users, we invited 89% via their talk page to play the Wikipedia adventure and the rest formed our control group and they were not sent an invitation but they were still, their behavior was still observed for the study. Out of those who were invited, 386 users actually played the tutorial. So basically the uptake of the tutorial just being invited to play doesn't mean that people actually played. So that was a separate characteristic that we noted. And after that, we tracked the behavior of people in all three groups over a 180 day period. We collected a number of dependent variables for all the users in our study. Specifically in the six months that followed their entry into the study, we tracked the total number of edits that a user made in that time. We tracked the number of edits they made to talk pages and we also calculated the average quality of edits they made during that time period. And specifically for edit quality, we used the content persistent metric developed by half acre and others, which, and the logic behind this is essentially that the longer an edit sticks around, the more likely it is to be of high quality. So we then conducted, we ran a couple of models to analyze the results of our experiment. We first estimated the effect of inviting a newcomer to play the Wikipedia adventure on the subsequent contributions they made, essentially those three dependent variables. And then we also estimated the effect of playing the Wikipedia adventure conditional on being invited on newcomer subsequent contributions. And essentially the distinction between the two is that since it's an experiment, randomization is important and randomization here occurs just at the invitation stage. The set of people who actually went on to play the game cannot be directly compared to the control group because that's not a random subset of the invitation group. So essentially we use a technique commonly used in economics called two-stage least squares regression that helps separate out these effects and identifies the effect of the actual tutorial on subsequent contribution. So then I go to what we actually found our results, which unsurprisingly, since I spoiled it for you in the beginning, showed that there wasn't an aggregate effect visible in either of our models on any of the dependent variables. So these are all insignificant effects that you're seeing here. And we were pretty surprised by this. So we ran a number of models and it turns out that this result is robust to multiple parameter specifications of basically all the models we tried.  So basically this is very much an old result. So then the question is, why do we have an old result that was positive feedback from the first survey study? So why is there a difference in terms of the effect that the system had on subsequent contributions? So one of the first things that people ask about is maybe there was something wrong with the system, but as noted earlier, users responded positively to the system. And even if this was a problem, we don't really know what we would have done differently just from the design perspective. It was based on prior research done in the design space. So we still think that just this explanation of like the system was bad is possibly not the most compelling reason for this result. It could be that gamification itself has some limits. People in the first respondents to our survey said that they enjoyed playing the Wikipedia adventure, but there is a difference between playing the Wikipedia adventure and editing Wikipedia. So the engagement that they felt using the system might not have fully translated into editing Wikipedia as a community member. And that could be an explanation for why we didn't see an effect on aggregate contributions. And then finally, a potential reason why the system might not have worked is that the deployment necessarily required users to opt into playing the game. And perhaps not enough people opted to play the game for it to have an effect on the overall pattern of contribution to Wikipedia. So basically, even though this is a form of institutionalized socialization in that it is structured and that it is, and it is presented to all newcomers, it is or intended to be presented to all newcomers, it still requires the newcomer to engage with it and perhaps not enough newcomers seek it out despite being invited to do so. And as a result, it can only be weekly effective in the aggregate. So I also wanted to clarify for the purpose of this audience, like what, I don't think some implications of these findings are. One, as stated before, is that the system is useless. I don't think that's necessarily an obvious conclusion to me because the null experiments essentially reflects finding from a very specific kind of deployment. So I think what the experiment actually tells us is that if you deploy such a system and expect people to take it up on their own, that alone might not change the aggregate pattern of contribution to Wikipedia. However, especially since there are other environments that people could try this game out, especially since we've had positive feedback. And so for instance, I've used this, I've used the Wikipedia adventure in classrooms. It has potential to be used in hackathons and other such environments as a sort of like gentle overview to all aspects of editing Wikipedia. Informally, the students I've worked with have liked it. More research is needed to see whether such a system has positive effects in these kinds of situations. And I also wanna make sure that just because we have a null effect that it doesn't, trying to onboard newcomers is a futile task. We're still faced with the same major problem of needing to retain and attract new users and new editors to Wikipedia. And even if one intervention does not solve the problem, it means that it's still a very viable and compelling area of research in order to figure out what can work. And in terms of implications for the community, it seems that one of the things that we learned from this experiment is that there's a difference between the self-selecting nature of Wikipedia sort of contrast with the very centralized idea of what the community is. So even though we invite people to contribute and say that anyone can edit, there is, and there are many tasks that we can do, there's still a very sort of centralized and shared notion of what it means to contribute to Wikipedia as a merge. And these ideas with each other. And finally, this idea that there are many efforts that have been used to redesign the onboarding process. What can newcomers do in order to better Wikipedia? But what can, in what ways can the community change or the system change on Wikipedia in order to be more accessible to newcomers? For instance, redesign user talk, the message walls, and the work that can be done to see what effect those kinds of system-wide changes have on newcomers as well as current community members. Visual Editor is another example of this. Of that kind of change. And then finally, coming back to testing in the wild, no matter what interventions we try, it is extremely important that we absolutely try to see the effect they have on the users in the field, essentially, and we can only know how effective these are when we conduct experiments of the I presented. And so that is essentially it for my talk. Thank you all very much. Thank you, Sneha. Thanks for the great talk. I think we may have room for one question and then save the other ones for later after this talk. So Jonathan, is there anything you want to relay from the channels? Yeah, we have one question. Anonymous Wikipedia asked on YouTube, is mentoring a solution for users to stay on Wikipedia? Yeah, that's definitely one aspect of newcomer attention that has been a lot of research that's been done that are other systems like T-House and Adopt-A-Use been used in the community. But the reason we tried this approach is that not everyone on Wikipedia can find a, not every new user can find a mentor. There are thousands of new users that come into Wikipedia every day and not every, not every veteran community member necessarily makes a good mentor. So matching people to mentors is definitely an important part of continuing building the community, but this was very much an exploration of how can we scale up this process? How can we make it more uniform? How do we make sure that people don't fall between the cracks? So I do think there are many avenues to explore how we can respond well to newcomers, but this was definitely one of the things that we tried and that absolutely should be other, and people have explored other ways as well. Cool, thank you. Yeah. Thanks, Nia. So it looks like we have maybe a few more questions, but we're gonna save them for later. So I'm gonna switch to Andrew's presentation at his point. And yeah, Andrew, the stage is yours. Again, about 25, 30 minutes and we'll have questions at the end. Fantastic. I'm assuming everybody can see my screen. If not, holler at me. So yeah, your audio is not really clean at the moment. So try again, let's see if it gets any better. Fantastic. Can you hear me okay? It's not great.  You sound better now, Andrew. I think it's much better. Yes, thank you. Okay, good. Fantastic. So thanks, everyone, for the invitation to present. Let me close that. Okay, so I'm gonna be speaking about the Gene Wiki project. And I'm gonna start with a slide that acknowledges the great team that we have here working on this project. I lead an academic team here in San Diego at the Scripps Research Institutes that works on this. We have collaborators at UBC, Washington, Maryland, Belgium. Most of this work is generously funded by the NIH. Of course, we wanna thank all the Wikipedia and Wiki data contributors. And lastly, we are recruiting. This link goes to add for a postdoctoral fellow, but anybody who's interested in applying the principles of open data to biomedical research, we'd love to hear from you. Okay, so I'm gonna start with a few slides that really introduce the motivation for our project. And it is summarized by this slide. And the biomedical literature truly is massive. This is a good problem to have. But the rate of biomedical literature and publication is exponentially growing. There are now over 1 million new articles published every year, averaged over the whole year. That's roughly one new article every 29 seconds. And so it's clear that no scientist is reading all the literature. So how do we think about accessing the knowledge that is represented in this corpus of literature? Well, if you think about, here's one example, right? If you're thinking about one topic, say, fibronectin, which is a human protein. If you search for fibronectin in PubMed, which is the primary biomedical literature search engine, you can get up to tens of thousands of different documents about or articles about that. And that reflects sort of the document-centric view of the world. So typically, what we as a scientific community rely on are experts in the field to summarize the current state of knowledge into review articles. So instead of trying to digest 30,000 articles, we sort of see a snapshot in time of what one expert or a small group of experts thinks is most important. Of course, review articles, you know, they sound a lot like encyclopedia articles. And so the idea is, can we essentially apply the same dynamics of Wikipedia to summarizing gene function? And that was the birth of the Gene Wiki project. So in that neighborhood of 2007 to 2008, we created the Gene Wiki project. And the goal of this was to essentially create a collaboratively written, a community reviewed, and a continuously updated review article for every human gene. And so we created about 10,000 of these articles, and the basic structure looks like this. On the right hand side is an info box where we imported data from structured biomedical databases and we reformatted them and put them into these info boxes. And the hope is that that would encourage in the free text side of things, contributions from the community to summarize current knowledge. So again, this was 2007, 2008. At this point, we have again, about 10,000 articles. They get somewhere in the neighborhood about five million views per month in aggregate, and they get roughly 1,000 edits per month. So just to show you what one particular page looks like, this is a real in, this is a human protein involved in neurobiology. It's a really great article. Most of it was done actually prior to our involvement. So what we did is we added this info box on the right hand side that systematically displays content from structured databases. And it goes on and on and has lots of great information. And just to highlight one particular nugget, I'm gonna come back to later. Here's a sentence that really hits some of the real biomedical significance of this protein. It talks about how the expression of relin has been found to be significantly lower in schizophrenia and psychotic bipolar disorder. So the article is filled with great information and for humans to read, it's actually fantastic to get a summary of the current state of knowledge. So the basic model at this point looks something like this. We have biomedical databases in the field that as scientists we know and love and use. Again, we reform out of that, put it into Wikipedia. And the hope is that we have this two-way interaction that the community both benefits from what we've synthesized as well as contributes back to Wikipedia. So I'm gonna summarize very, very quickly because it's all published work. A few applications that we did that were based on the GeneWiki collection of pages within Wikipedia. So as one example, we did text mining within the GeneWiki pages, these Wikipedia articles to extract structured biological annotations. That's in this paper down here. We also developed partnerships with journals, in particular the journal Gene, where we do a dual publication model for invited review articles. Where in addition to submitting a peer reviewed article that would be published using traditional means, authors would also contribute a Wikipedia article. This is a way to align incentives for people to contribute to Wikipedia. And the third one I'll highlight is this somewhat ambitious, but in retrospect, misguided approach to embed structured data into the Wikipedia article using Wikipedia templates. And we call these semantic wiki links. So to illustrate what was going on here, I'm gonna come back to this little snippet about RELIN, okay? So RELIN has lower expression in schizophrenia. That's great for humans now, but that's actually difficult for computers to access. There's no way, for example, using just that free text for a user to query, you know, tell me all proteins that are significantly lower in schizophrenia, right? That's not something that Wikipedia was designed to do. So in an effort to make Wikipedia articles and sort of facts within Wikipedia articles, a bit more structured and something that in theory could be easily mined out and queried, we created these semantic wiki links. And you'll see, you know, this is the wiki text. Instead of just having a wiki link to schizophrenia, we would embed that in this SWL for semantic wiki template. The target would still go to schizophrenia. So it's displayed as a link as normal. And the type was essentially the nature of the relationship. In this case, RELIN has decreased expression in schizophrenia. Okay, so now in retrospect, right, this is a pretty convoluted way to represent structured data, but, and this was published in sort of what, late 2011. So thankfully, in 2012, we were rescued from this treacherous path we were on by this fantastic application called wiki data. I probably don't need to introduce it very much, but the basic idea is that Wikipedia is to text, what wiki data is to data. The mission statement as articulated by Denny Vendrichek, you know, provided database of the world's knowledge that anyone can edit. So this would be sort of a more proper way to represent structured data in a crowdsourced and community infrastructure environment. And then our gene wiki project really expanded to really make wiki data a primary repository or biomedical knowledge. So, and that's the transition to the second part of my talk here. So now if you, if we think about how we represent knowledge within wiki data, again, I showed you the Wikipedia page for Reland. Now let's look at what the wiki data page for Reland looks like. It's shown here and it's obviously small to read. So let me sort of summarize to the high level. The Reland page for wiki data is broken down into a series of statements. The subject of the statement is always Reland. And so Reland is a subclass of or a type of protein. Reland physically interacts with these other two proteins here. Reland regulates neural development. And down here, the ones we highlighted, Reland has decreased expression in schizophrenia. Because wiki data is fundamentally a database and not free text, all of these concepts from Reland to the targets that it's related to, to the types of the relationships, these all have unambiguous identifiers. The items have these Q identifiers, the properties have these P identifiers. These are sort of a semi-country of the types of ways genes can be related to other concepts. So of course, again, I'm showing here a screenshot of the web interface, but Reland is really, sorry, wiki data is more about being accessible to computers for computers to access. So in addition to the web interface, there is a, for example, a computer readable JSON version of this, at this URL up here, where this is all more tightly structured. And this structured data is available in a number of other formats as well. Importantly, I'll just note to that, again, this fact of Reland has decreased expression in schizophrenia. It also includes qualifiers. So we can see how, in this case, the qualifier is how that expression was determined, as well as the references. So we can see what paper, what scientific paper exactly, is cited to support that assertion. Okay, so this is the basic wiki data model. And so we were very attracted by this. And so the GeneWiki project then expanded a little bit, instead of taking biomedical database information from biomedical databases and putting into Wikipedia, we put it into wiki data instead, because wiki data has tight integration with Wikipedia. Of course, we could use that data to populate infoboxes. We hope that we would have still a two-way communication with the community. And here we'd be accessing and appealing to a different community where we have information scientists and data providers and things like that. And then we can, again, assemble biomedical data in wiki data. So that was the model. And this is sort of a snapshot of where things are right now. This is a sort of a class level view of all the data that sort of the neighborhood of the data that we have loaded into wiki data. Let me zoom in. And you can see in the middle here, we have the genes. So genes have different properties like their gene symbol, various identifiers, and links to other databases. And this says that we have roughly 505,000 of these gene objects that we've loaded or created or maintained within wiki data. So genes are related to proteins by the properties encoded and encoded by, or encodes and encoded by. And proteins then can be linked to things like structural motifs, binding sites, and protein families. If we come back to genes, genes can be related to different diseases. We've loaded something like 8,000 diseases here. And those links are from this database GWAS Central. Diseases can be related to chemical compounds. Chemical compounds can be related to pharmaceutical products. And chemical compounds can also be related all the way back to the proteins that they act upon. So everything in orange here are sort of classes of items that our team has taken a special interest in. And for the primary role in trying to organize information. In addition, you see some of these in purple where these are other teams in the biomedical research sphere that we've loosely coordinated with to load information on things like sequence variants, genetic variants, and biological pathways. So you have in green sort of efforts that were completely, we were completely unaware of until they started showing up. This was an effort by the CDC to load information on chemical hazards. This was the wiki site initiative to load information on bibliographic records that then can be cited. But together sort of really expand the ecosystem in wiki data that is very useful for biomedical researchers. And you can see that again, I'm just snapshotting but there are links out to various other entities within wiki data that also contributes to the richness of this knowledge network in wiki data. Okay, oh, so I should say that in total we're touching or maintaining about a million wiki data items at this point and trying to get the relationships between them as best we can. So in the same way as I did before, I wanna highlight a little bit here in terms of the applications. How do we demonstrate the utility of all the information we've loaded into wiki data? And the first one is actually just using the Sparkle endpoint to demonstrate some integrative biomedical queries. And let me walk you through a series of examples in this space. So suppose I just wanna do an example, that's just relatively simple data retrieval. Suppose I wanna find all genes with some genetic association with asthma. To make, to perform that query, we use the Sparkle query language, the details of which here I'm not gonna go into at all except to note that this is just a query language and you reference in those QIDs, so this is the one for asthma, this is the one for gene and the PIDs which were the properties. But in this five lines of Sparkle queries, we're able to execute this query and then you get this output table here through the wiki data Sparkle endpoint, which tells you very quickly at the snapshot in time, we have 39 genes that satisfy that. So suppose we wanna start refining this query. We wanna get all genes with a genetic association with asthma, where also the gene product or the protein is localized to the cell membrane. To do that, we add essentially these three lines to our Sparkle query that references the QIDs for membrane and then we can get those 39 genes down to these 22 genes. Suppose we want instead of specifically isolating on asthma we wanna generalize this to any respiratory disease. This is where we take advantage of the disease ontology which we loaded, which describes the relationships between diseases. Again, we touch these three lines of the Sparkle query and we get back up to 31 genes that correspond to eight related diseases. So again, asthma, we also see genes related to these other respiratory diseases as well. And finally, suppose now if we wanted to look at all those diseases and show what were the causative chemical hazards that were loaded? And this touches the data that was loaded by the CDC. Again, nothing that we had a direct involvement in loading. We add these two lines of code and we're able to get four diseases related to six chemical hazards. The take home message here is simply that you can start chaining together increasingly complex queries that span multiple data resources in a pretty concise and expressive query language. So all of that can be just encoded in 17 lines of Sparkle query. Stuff to look out, but once you get accustomed to it, Sparkle is a reasonably straightforward query language. So that I think is one way in which we are trying to use WikiData and demonstrate its utility to the biomedical research community. The second thing that we're really working on is building domain specific web applications. So why is that? So if you can, you know, if we summarize what we've done so far, we have loaded our sorts of data from biomedical databases into WikiData and we've encouraged other people in the field to also load their data into WikiData. But this alone, we haven't fundamentally changed anything enabled any new biomedical analysis, right? I as an individual researcher would be more painful, but I could have done that data integration in my local database. Where WikiData I think really shines is we can start to access data that hasn't previously been structured. And that is the knowledge that is contained in the brains of our domain experts, right? So domain experts might know that GeneX is related to disease Y that's not in WikiData, right? And the question is, so clearly individual users can add those individual statements to WikiData using the web interface, but we've thought a lot about how do we facilitate this in a more streamlined manner in a way that directly engages domain experts. So as one example of what we're pursuing here, we built this application called ClamBase at this URL. It's a portal specifically for the Chlamydia research community, okay? And so you can imagine there's dozens to hundreds of labs across the world that deal in Chlamydia and they need efficient ways to share knowledge between each other, between their labs. And so we've built ClamBase to try to organize all sorts of Chlamydia-specific information in a portal that's very specific and tailored to their needs. And we hope that Chlamydia researchers, for example, will find this useful. Importantly, right? Down below, I haven't shown it, but there are edit buttons. So you can say, oh, well, yeah, this knowledge here is great, but I know something in addition. And so if you click that link here, this is sort of where somebody would add something about a molecular function for a protein and people can fill out the exact ontology term for that, how to determine the reference and so on and so forth and behind the scenes, then that is writing directly to WikiData. And so we think this idea of domain-specific interfaces is a powerful way to engage domain experts to contribute data and really structured data and knowledge that had not previously been structured before. Okay, so that wraps up everything that we've done. And I just wanted to end very quickly on just some naval gazing and pontifications that we've had. I'm gonna go through this quickly because mostly it's conversation starters for later and for offline, but I wanna give a sense to people on what we're thinking about for the future. First is we think a lot about data incentives for biomedical data owners and how to convert them into data contributors. The way typically this works, right? NIH spends money and gives money to different resources to produce databases. These databases, if they're valuable and useful, get used. Siege actually then comes back in terms of citation and attribution and that's what the NIH uses to decide, oh, okay, well, I should give these guys more funding. In the WikiData sense is that the CC0 license removes this requirement for citation and attribution. And so we're thinking about, and so then obviously data providers are very worried that this cycle breaks and they lose their funding. So we're thinking about direct measures of usage. This ties into ideas like mining sparkle query logs. This is the topic that Marcus Kreuch mentioned a couple of months ago in his talk, thinking about metrics of network interconnectedness and we're definitely open to other ideas as well. We think a lot about how do we push WikiData content to Wikipedia and I think there's some recognition that we need better and tighter integration of the Wikipedia and WikiData edit histories, including sort of statement level filtering. So not just the corresponding, everything from the corresponding WikiData item, but just specifically the facts that get transcluded into the Wikipedia and then sourced across all the linked WikiData items, not just this references arbitrary access feature. There's a really illuminating discussion I think that goes into some of these issues that better integration would address in this link down here with the WikiProject medicine community. The last one I'll say is that we've been thinking a lot about more expressive data modeling and reporting. The biomedical data models get pretty complex pretty quickly and we need to figure out ways to share and as a community decide what those data models would look like. We put a lot of, or we're investigating quite a bit this modeling language called Shex, our shape expressions, this is worked by Andra Bogmeister and this is sort of an example way of expressing for example, genes must be an instance of a gene and this is one and only one, it must not be a subclass instance of a protein so on and so forth. So it's a way of expressing constraints that is complementary I think to the constraint system that already exists within WikiData. We think about how do we visualize and disseminate these models and it's a topic I think Dario has thought quite a bit about and then also about how we do reporting and violations so that they can easily fixed or addressed by the community. So that covers essentially what we've done and what we're thinking about for the future. Again, I will end on this slide because it is, I get to speak for the great work of many other people. With that, I'm happy to take any questions if there's time. Thanks so much, Andrew, fascinating talk and great overview of everything that the Zoom Wiki team has accomplished. So I wanna ask Jonathan, do you have questions that you want to relay from IRC? And I guess after this we can open up the discussion also about SNHL stock. No other questions about Andrew's talk from IRC so you are good to go, Dartar. Okay, so I'm gonna start directly with mine. Yeah, Andrew, I think that I have I guess one comment and a question like the point you're making about attribution in the problem with licensing. Licensing I think is a very important one. I guess we're still struggling to figure out ways of engaging expert communities in WikiData. And I think as someone who is involved in WikiData, obviously the benefit of CC0 licenses like vastly exceed that the issues of attribution under the assumption that attribution can be set by norms in the field. They don't need to be enshrined in the license level. However, I know that this is a serious issue every time you talk to lawyers from the IP unit of university will tell you, no, no, no, this is the reason why we don't do CC0. And this is probably one of the reasons also why database rights exist to try and protect the curated version of a knowledge base. So, yeah, I think it's a fascinating question. The comment is that I would like to try and explore better ways of addressing this concern. I think you're completely right. That is a big issue. The more specific question I have for you is to what extent this can be generalized to other fields. I know Gene Wiki folks put some thinking into how this model could be adopted by different communities. Can you speak a bit to what's specific to the biomedical community versus what's generalizable? Sure, so first let me just say my comment on attribution was in no way, I 100% am backing the CC0 license. I think it's great. So there's a lot of just education and outreach that we need to do. And I think us having engaged a lot of different data providers, I think we've learned at least some of the issues that are important to data owners in this field. In terms of what generalizes, right? I mean, it's a good question. I mean, we've tried to generalize our infrastructure as much as possible. For example, we developed this wiki data integrator, Python library that helps bought owners to load their data. Our automation, we have an automation and reporting system. For example, that keeps source databases in sync with their representation within wiki data, also an important issue. That one, we haven't really, it's not quite as mature yet, but that's something definitely we think is generalizable. And some of the discussions around data modeling, right? Other areas also have needs of more expressive data models. So these are all things we are happy and excited to work with people in other domains on as well. Thank you. Can I pass it on to Jonathan? So we had one, we have a question from two questions actually for you, Andrew, that have just popped up. The first one I just put in the chat sidebar. How does Gene wiki handle the volatility in life science results? My experience with personality genetics associated between personality and SNPs is that some claims in scientific articles are later revised in meta analysis. The current scheme with lists of properties seems to be grounded on the frequentist and p-value focus thought. Do you think of a more flexible numerical method? So I think wiki data, right? I think made a very important design choice early on. And maybe this applies to Wikipedia as well, but it's not to represent truth necessarily because truth is not always certain. It's to represent claims, right? And assertions on truth. So that I think is important so that we can represent conflicting information. We don't have to reduce it down to a binary yes or no. The latter part of the question, do we think a more numerical system where we're actually storing raw data? I don't know. I think there's potential value in that, but I do think wiki data right now, there's also value at that level of assertions and claims between that sort of a nice layer of abstraction where you're not dealing directly with the raw data. I really like that as a way to facilitate some of the integrative queries that I demonstrated. Awesome. We have another question for Andrew and then we have a question as well for Sneha. The second question for Andrew is, besides search, could one mind this network of concepts to discover new facts about genes and other entities? Yes. This idea of latent relationships in the network is key, right? And this is where you can imagine all sorts of machine learning algorithms to identify those latent relationships from existing drugs being used to treat new diseases, to finding new genes involved in diseases and things like that. Active area of research, but the thing is everybody has their own silo of knowledge on which they're using to train their data. And I think the argument is that if we do that in the context of wiki data, we collaborate on that piece and then everybody competes essentially on the algorithms, which is where the real innovation happens. Awesome. So I think this is probably our last question. This one is for Sneha and this is from Giovanni Cempaglia on IRC. He asked, did you consider other measures of retention? Like survival analysis is, I'm assuming that's, we basically, we did have a few other variables that we didn't report. I think we also looked at contributions to articles and so on. We didn't report them because they told very much the same story as the current results that we, that I did share with you. I do not believe we did a survival analysis, but it's also, the reason we did that is because it's given that this is an intervention that occurred sort of like at the beginning of a newcomer's tenure on Wikipedia, the longer you sort of go out from that point, the less attributable the effect is to this thing that they did like months ago. So, I mean, we represented this, we did, we chose to do six months just because we obviously tested the results for a number of different timescales after, and I mean, and again, it said the same story basically and we picked six months as just one, as one time scale that seemed to represent the results that we got. So yes, we did do some other stuff. We didn't do survival necessarily because the longer the time scale going out from that, it seemed not attributable to the Wikipedia adventure. Awesome, thank you. And that I believe is it and I think is all the questions we had, so. All right, thanks so much, Jonathan. I want to close by thanking again our speakers, Nair Narayan from the Northwestern University and Andrew Sue from the Scripps Research Institute. It was awesome. Thanks a lot for joining today. Thanks to everyone who was on the channel for your question and we're gonna see you again in a month from now with our next showcase edition. Thanks and have a good day. Bye-bye.


Four things narcissists do when you discard them when a person decides to discard the narcissist It's because they've had enough They've had enough of the constant insults and put downs They've had enough of the narcissist using them for money or sex The narcissist ran amok on you They behaved in an uncontrollable way They were very disruptive They were very inconsiderate and they had no intentions of changing their behavior Because they held you responsible for everything they did They made you believe that you were the one who was supposed to change or do better When you finally decided to discard the narcissist It's because you've woken up You've realized the truth of what you're dealing with You understand that if there's anything you need to change You need to stop giving so much of yourself to someone who doesn't appreciate it If there's anything you need to do better It's enforcing your boundaries And the more you try to do this The more it should confirm to you that you need to get rid of them Because they're never going to change They're never going to respect you A major part of their disorder is that they think they don't need to change They think everything is your fault And when you realize this You accept that your only option is to discard them That's the only way that you can deal with these types of people Because they're never going to hear anything you say They're never going to try to understand where you're coming from They're just going to see it as though you're trying to manipulate them Because that's what they did to you And they assume that everyone is just as fake as they are But they already know that the situation is not good for you Which is why they have to target your self-esteem to keep you down To make you lose confidence To keep you thinking you're not good enough at anything else So that you don't have the strength to leave them When you finally decide to discard the narcissist It's because you realize that it's not your fault You're not responsible for the way that they're behaving You're not responsible to fix anything Because you realize that you're dealing with someone who has a mental illness The reason why it was so difficult for you to leave Is because you had hope that they could change And you held yourself responsible for making them change As though you just had to do better or do more And then finally they would be okay But nothing you did was ever good enough for them Because they have these insatiable desires that can never be fulfilled They couldn't even make themselves happy Yet they held you to that expectation Of something that they couldn't even fulfill for themselves They are very delusional entitled people And when you fail to fulfill their impossible expectations They will become very resentful towards you They will run revenge Which is why I decided to create this video So that you can prepare yourself For what the narcissist might do when you discard them One, they will stalk and harass you When the narcissist does this It's because they're refusing to accept that you want nothing to do with them Or they just don't care about what you want Because they're only concerned with their own feelings and needs They are self-absorbed and they lack empathy So it's not like a normal relationship Where the other person understands that you're not happy The narcissist is only concerned with what they want And they expect you to remain around them regardless of how you feel Your feelings and needs are not important to the narcissist They always have to be the priority Because they believe that they're special They believe they're superior to you So you're just supposed to submit to them and meet their demands Regardless of what you want The narcissist sees everything in black and white You're either a winner or a loser And if you discard them They see it as though you have won And they see it as though they have lost And they are very sore losers They cannot deal with losing They set everything up in an attempt to avoid ever having to deal with losing When you discard them They see it as though you have got one over on them As though you have abandoned them without help or support It causes a narcissistic injury It bruises their fragile egos And in their minds you have no right to do that Because there's nothing wrong with them They haven't done anything wrong In their minds they're perfect Which is why they cannot accept it when you try to move on This is why they stalk and harass you This is why they cannot leave you alone Because in their minds you're at fault You're to blame for everything 2. Triangulation When you discard the narcissist They will bring a third person into the relationship So that they can remain in control The communication between you and the third person Will be through the narcissist Or if you refuse to talk to the narcissist The third person will communicate with you directly They will use this third person to bring you under their control again The third person may also be the narcissist's new relationship partner The narcissist may act like they are happy with this new person It's designed to make you question yourself To make you wonder if you made the wrong decision They will try to make you jealous They will try to make you want to return to them When there's really nothing to return to It's just another part of the manipulation They're deceiving you But if they know that you're no longer susceptible to their manipulation They will only triangulate you with one of their flying monkeys With someone who they have manipulated into thinking that something is wrong with you This person will be used to manipulate you into returning back to the narcissist Whether they are aware of it or not 3.  Smear campaign If you don't fall for their attempts to get you back They will talk to people about you And portray it as though you were the narcissist As though you did all of these horrible things to them And then you just left them without any help or support Because they want to get people on their side And they want to destroy your reputation So that you're left without any support Because they don't want to see you moving on with anything If you leave them They want to see you in a worse situation They don't want to do anything They don't want you to feel like you're better off without them Because then that makes them feel small It makes them feel like they're not good for anything So they have to make it seem like you've made a poor decision And now you're paying for the consequences of that decision They can't let you think that you did the right thing Because of how that would reflect on them So they play the victim And they try to make people think you've done something wrong So that they can gain people's sympathy And they don't want your reputation It gives them supply It supports the illusion This is what happens when you discard the narcissist They will stalk and harass you They will triangulate you with a third person They will play the victim And they will start a smear campaign against you I hope that this video can help you to protect yourself If you do decide to discard the narcissist Thank you for watching Please like, comment, share and subscribe If you would like to donate My PayPal link It's in the video description Coach's inquiries You can email me at coaching.nr.surviver.uk Thank you for watching and I'll talk to you soon


on how to download install and use Google Earth Pro on desktop Google Earth recently released a very nice and sleek browser interface for for their platform but their desktop software is extremely powerful it's known as a GIS geographic information software and it's probably one of the easiest types of the software to use and to create professional looking maps with so we're going to introduce it and talk about it if you don't already have it installed on your computer you go to this website here and download it and install and once you do that you can open it up it looks like a gray Earth says start Google Earth Pro and that will that will bring up software and it usually comes with some level of hints and in different ways you can use the software effectively and you have to have an internet connection still even though this isn't browser based because they are feeding a lot of the aerial photography and elevation all the other types of data through the internet it's not all on stored on your machine so here is the view that comes up and you can zoom in and out of the area using your mouse cursor wheel and if you hold down your mouse button and then drag your mouse around it will drag you plan it around into different views and so you can also control those zoom features with these little buttons on the side here and they'll change your perspective so it's just playing around with those and if you ever need to get back to north orientation you can just double click on this little compass here on the north part and it will reorient you in the right direction so one of the most powerful parts about a GIS is these layers and you can add many different layers of information to to get an idea what's going on in an area in a mapped area the default layers that Google provides are borders and labels and places the photos layers changing now but they're still providing some some level of photos and then roads then you can turn 3d buildings on and off and this feature is available for some of the larger cities or it will show you the buildings and then you can also turn terrain on off and definitely suggest you have terrain on at all times because turning it off basically flattens out your view and I'll show you here so that's what you get when you turn terrain off and then you can add your own places so I already have some of these places filled in here and any kind of KML or KMZ dot KML drop KMZ file can be loaded into this map and I I have a module some KMZ files from my online geology class I can turn those on and off if you double click on a feature a layer in that it will zoom you to the extent of that layer and you can view it it doesn't look like it's wanting to show up really well right now that's what the big giant red X is but that is the different places and then you can search for new places their example has an address you can also search for latitude and longitude that's a little picky and how you do that if you know random latitude and longitude and it will bring you right to it so searches for both place names and and coordinates as long as you type the coordinates in right you can also add your own features to the map if you want to add that so I can turn layers on and off just by clicking these little check boxes here and I can expand them and see more information about those layers and I want to make my own I can use these buttons up here and there's all kinds of interesting things you can add I will just do place marks from for now and here you can add a little point in the default is this little push pin you could make your own label and change the colors and do all kinds of the fancy things and then you can see that there and you can even save it as a KML file so and get directions all kinds of other features so that this is a very feature rich relatively intuitive software right here another really nice feature about Google Earth Pro is this this time slider if you click on this it'll bring up this little bar here and notice has dates on it and here's the most current imagery but you can drag it around to different imagery dates that will show you historical imagery for that area if it's available so that's a really useful tool to look at older scenes if you want to see how things change over time or something like that and then another tool that I use very frequently is this little ruler-shaped tool here it's a measuring tool you can measure distances between roads you can also measure you know how long a path might be and measure out the path many different line segments you can also create a polygon which is a just like a shape measure out the area of that polygon there's all kinds of nice measurement features that you can use so that is it for a very brief introduction to Google Earth Pro I definitely recommend you using it it's very nice free software especially if you are interested in in making maps or just examining the terrain doing some GIS type measurements and things like that so thank you very much


Hello, everyone. Can you hear me all right? Thanks for coming. I'm Montana. I'm the co-founder, CEO of Postgres ML. And I'm here to talk to you today about doing machine learning inside the database. So some context on why I believe that the database is the right place to push this computation, why you should add GPUs to an already overloaded instance, potentially the hottest, most resource contentious place in all of your infrastructure. I'll tell you the story about what I've been doing for the last 20 years in software engineering, machine learning, natural language processing, and company building. I open sourced Instacart's open source platform called LORE about six years ago. We were running deep learning models in production on TensorFlow 0.4. Maybe not the best idea, but we learned a lot. And a lot of those learnings from those early days have informed how I build machine learning infrastructure now. A lot of the scalability issues that we hit when we went into COVID in 2020. And we were already a multi-billion dollar revenue company. And we started doubling on a weekly basis in terms of traffic. You can imagine that every infrastructure resource we had started straining. And our machine learning infrastructure, which I had helped build out a lot of, was one of the worst offenders. So a lot of, ideally, naturally, I'm a perfectionist. I like systems to be optimal. I think that's why machine learning and software engineering appeal to me personally. But I've also learned a healthy dose of pragmatism and that you need to ship solutions as quickly as possible with the least amount of effort as possible to get close to that ideal end state. And you can iterate over time. Software engineering and machine learning, especially, these things are never done. These are evolving systems. They're living systems. And maintainability, manageability of these systems is critical for success. Just to give you the meat of the presentation upfront on, are we totally crazy for doing machine learning inside the database? Is Postgres fast? Is a relational database capable of any of these workloads? If you actually compare the end-to-end performance of a Postgres ML system against what I would call to get a Python microservices architecture or a TypeScript microservices architecture, just calling out to open AI to generate your embeddings for your RAB application during query time when you're going to generate that chat prompt, it can be 10 times faster if you do that inside the database. With an open source model that you control, you don't have to wait for your quota limits on open AI. You don't have to give up control over your prompt quality. And these open source models are actually higher quality. They're beating out open AI's Aida embed 002. And they have been for the last six months if you look at all of the rankings for quality in these models. Similarly, you look at industry leaders like Huggingface and Pinecone for a vector DB.  And if you're doing that over and over in your application basis, it's significantly slower. There are other, it's funny to me when people put DB in the name of their product and they're not actually a database. We've seen several of these lately. And they are sort of thin wrappers around databases. But overall, Python microservices even built on top of highly specialized databases, whether that's Redis, which is an incredibly respectable key value store. Like I have a lot of love for Redis and what they do. I have a lot of love for Cassandra and Memcache and all of these other databases. And we were running all of these other databases as various feature stores at Instacart when COVID hit and our growth started exploding. And inevitably, what we found is that most people can just deploy Redis. But few people know whether or not they're running Redis in a persistent safe failover mode or not. And what's going to happen when their Redis cluster actually hits 100% CPU usage or 100% RAM utilization. The short answer is it will crash. You will lose all of your data. You will have to backfill your data from your primary Postgres instance that was actually authoritative backing store or Druid or Snowflake or whatever. And that service will then be down for however many hours or potentially days it takes you to refetch and reload that data. So one thing we learned is that just because you can stand up a database does not mean you can operate a database, especially under duress. And so that's one of the important things I think about these days when building large scale systems is do we actually have the engineering resources on our team to manage more than one database? Or do we have the resources to manage any databases at all? This is a pretty common thing to outsource. There's a reason Amazon RDS is what it is a lot. It's very common to do this. But for machine learning applications in particular, I think there's a few considerations we need to make when we choose a database. Obviously you want high client concurrency, which really just means you're going to need horizontal scalability at the outset. Cassandra is a stand-up example of horizontal scalability for everybody at this conference, which means ultimately you do need to consider sharding your data. A lot of people think sharding is hard. It's really not, especially if you understand your data model. Most people have a user ID in their applications. You can generally shard by user ID, and that's the only consideration. Like that's the end of the discussion. It's not always true. There are follow-up on considerations on what do you do with cross-shard joins, for example. How do you replicate data in your cluster? Those are thorny problems that you can solve at scale once you're making a billion dollars in revenue and can hire the engineering team to solve those problems. They're not necessarily things you need to consider upfront. But you do want millisecond read times for any interactive application. Data warehouses are off the table because of this snowflake, redshift. All of those systems generally can't support high concurrency. They generally can't support low latency. And finally, if you really want to do online machine learning, you need session level data streaming to your primary feature store. And again, this takes data warehouses off the table for the most part because those suffer really bad performance under incremental rights. So almost certainly you're out of the OLTP world into the OLAP world. No, I just said that backwards. Sorry. You want a transactional database, not an analytical database, which is backwards for a lot of organizations. A lot of organizations originally institute machine learning under the data science team. The data science team is the one that has created this analytical database, this data warehouse setup. And then they're tasked with, hey, can we bring ML online into the application? And a lot of KS and Mayhem and Suze as they try to actually make those things fast. Because it's just not what they're designed to do. If you're doing machine learning and somebody ever asks you to build a feature store, all they're asking for is a database. A lot of them, all they need is key value access. So Memcache is fine, Redis is fine, as long as you know what happens in a failure scenario. But more recently, people want these hybrid search apps so that they can do keyword search and they can do vector search. People like JSON document filtering metadata access. So having a document type with some extended capabilities is nice. That will make your applications developers life a lot easier and it'll make your applications a lot more efficient. The more data manipulation, transformation, filtering that people can do in the data layer without having to pull all those documentations back to the application layer. And finally, it is actually pretty nice if you can generate embeddings in the database. It is really nice if you can run LLMs in the database. There's only a couple databases I know that can do this so far. Postgres and Elasticsearch. And I'll talk about that a little bit later. That's really the whole point of Postgres ML is adding these capabilities to Postgres. Finally, I think I've touched on what it means to be a proven database. There are a lot of brand new vector databases, much newer than Redis, much newer than Cassandra, much newer than Postgres. I think people are currently learning a lot of operational lessons about data loss for those kinds of databases. And it's gonna be very interesting to see how that ecosystem evolves as a single index type is sort of added to every other database that has already been proven out at scale in the engineering community. So I've made a quick grid here. All of these grids that you'll ever see are kind of rigged by the creator or the presenter. You can ignore the right hand side of this graph. This graph, that's your vector ML embedding and LLM operations. We've specifically added those operations embedded inside Postgres, so it's taken the lead there. But we could have added those to Elasticsearch. We could have added those to Cassandra. These are all open source databases. I'm a big proponent of open source. And I'll get to the reason why I chose Postgres specifically to add this functionality to because I believed it was the best platform for machine learning database. The answer to that really is the two left hand sides. And if you're a database person at a database conference, I think that these are the most important consideration for all of you long term, much more so than vectors. Like, does your database do vector search? Every database is gonna do vector search in the next year.  But I even think that most people will eventually do embedded ML. Most people will do embedded embeddings. Most people will do embedded LLMs. It makes a lot of sense when we get down into it. But who, which databases support sharding very well? Which databases support joins? These two questions become really important for machine learning workloads. To explain that a little bit, I'll create a toy example. And this is an example that's actually at the heart of Instacart's business model. Is it the heart of Amazon's business model? It's at the heart of Shopify's business model.   And we call that an offer. So an offer just relates to a product. It relates to a retailer. It has a specific price. And it has some other metadata like whether or how many are in stock or not at that particular retailer. And you keep account there and you decrement one every time you sell and then your market is out of stock, et cetera, et cetera. This is a normal form of a relational data model. It's an interesting non-trivial data model. You'll notice that the product has some nested JSON document as it's machine learning features. And those are actually the problem. Those become a really big problem. And I'll get into why in a bit. But we can compare that normalized form to this denormalized form. This is the NoSQL form of the same schema. What you do is you copy the product and you copy the retailer into the offer. And that way you don't need to join when you're reading the data. You can just select the offer by ID. You can get the retailer information from that offer. You can get the product information from that offer. You can get the machine learning features from that offer. All of this nested data is right there. You don't have to worry about the cost of joins anymore. And that's great because A, it saves you compute at read. But B, it makes it easier to shard. Joins make sharding hard. So this is actually really cool. And this is why NoSQL is better than SQL. But there's one problem here, which is if you'll notice what those machine learning features are. Machine learning features are typically statistics that are recomputed periodically, perhaps after every single purchase of a product. I need to recompute what search terms converted for that product. In this case, the product is Coca-Cola and maybe somebody searched for Coke and then they clicked on this product and bought it. And so now the conversion rate for the keyword Coke on this product has gone up some fraction of a percent. If I have to actually go update the global conversion rate and the term conversion rates for all of the retailers that sell Coke. Now remember, this isn't any more one document in a normal form. This is every single offer of Coke. So at Instacart, for example, there may be 100,000 different retailers selling Coca-Cola with this same offer. And if that one checkout happens, that one purchase happens, we have to go update 100,000 search terms for Coca-Cola because it's been denormalized into our Cassandra or our Elasticsearch clusters where we were originally keeping data like this. Those technologies are amazing. They scale incredibly well. They scale incredibly far. Up to the point where your CFO eventually notices or your board or in the case of Instacart's S1 IPO prospectus, all of the public markets, notice how much you're spending on IT infrastructure. That's not a good thing. So if you wanna do real-time machine learning like this, you have to normalize your machine learning data. You have to pull it out of this nested form and you have to join to it at read time. It's really the only sustainable, scalable way to keep this up to date.  We're not gonna update statistics in real-time. We're gonna have a weekly batch job and we're just gonna recompute the statistics once a week and you're just gonna deal with stale data. And then they're gonna say, well, every time we add a new product to the shelves, it takes a week for us to get any statistical information on it and you say, well, okay, we just won't sell that product for the first week. And then some product manager is like, well, this is on special. It's only offered for one week a year. It's now out of season. There are lots of compromises that get made to keep machine learning models running in production over and over again and all of those inefficiencies really become business inefficiencies and cost you either top-line opportunity costs or they impact your bottom-line revenue through data infrastructure costs. And those conversations are actually really unpleasant to have when you're either the data scientist advocating for your model and saying, I need fresh data. I can increase the business value of this data to the company if you can just get me this data sooner. I've worked hard doing all this research. That's really frustrating. At the same time as if you're the machine learning engineer and you're like, oh, but our Cassandra cluster is costing $100,000 a month and we have a directive from the CFO to eliminate usage and you are the biggest consumer. So that makes nobody happy. So finding ways around that is really important. If you think I'm exaggerating and if you think that the world is gonna get better because Moore's law is eventually gonna save us, I have really bad news for you. These models are getting a lot more data hungry with this vector data. The models themselves are getting bigger. The amount of data that they want and need access to and want to consider. You show a machine learning engineer, I want to consider some vectors and they're like, can I rank 10,000 of them? And this is 10 kilobytes of UTF-8 data for one single vector right here. And then all of a sudden they want 10 kilobytes times 10,000 pulled into their Python-Panda's data frame, which is, I think that's 100 megabytes of data and then Python-Panda's is gonna double that because they do an inefficient copy. So you need like 200 megabytes in your Python microservice process that you're gonna have to load. And that's gonna be really slow and they're gonna say, oh, I've just blown my latency budget. How about 100? Okay, maybe 10. I just need 10 vectors now. And again, it's the same conversation, it's the same problem over and over again that you actually cut the efficiency of these systems at multiple levels. So we ended up going a different route at Instacart. We have, I think, several really innovative pieces of technology that have come out of our research there. One of them is PGCAT created by my co-founder.  If you accept that you're gonna do more work in the database, you absolutely have to be able to scale the database horizontally. And what that means is you put a proxy in front and then sharding gets really easy. And you can actually shard, you can replicate and you can vertically scale these machines. Commodity hardware now offers 256 cores and nearly a terabyte of RAM in a single machine. This is actually a pretty enormous piece of hardware. So vertical scalability for a lot of companies. There are multiple reports of companies getting to billions of dollars in revenue on a single Postgres instance. But long term when you're a multi-billion dollar company, you will want multiple Postgres instances. You will want sharding with your application. So PGCAT makes this transparent.  You can start adding shards and replicas behind it and your application engineers no longer have to worry about that big mess. Similarly, the way Postgres works internally for any single one of these instances is that every connection to Postgres is its own fork of the Postgres primary. This forking to establish a new connection is relatively expensive off operation. And so PGCAT, by sitting in front of the database when a new connection comes in, it can reuse existing connections. And so it can actually alleviate a lot of load on your primary by managing the connections for it and keeping long-lived connections open in the background and just reusing them for short-lived front-end clients. This is even more important when you get into this serverless world of JavaScript applications where they're constantly spinning up new Lambda processes and shutting them down. You don't wanna have to create a new backend Postgres connection every single time. Finally, what this allows us to do is this allows us to keep a GPU cache long-lived on the Postgres server where the models and features that we're storing in Postgres can actually be moved to a shared pool by the single postmaster where you actually have hundreds of clients able to access these things concurrently. So you have a mature, robust, serving infrastructure that's been battle-tested by hundreds of organizations. It's an efficient protocol. It's an efficient memory layout. Everything we do in Postgres ML is written in Rust, including our extension for Postgres. We do call into Python for the latest hugging-face transformers where the research papers have only released Python code that make these models work, but by and large, our data operations and manipulations are as efficient as possible in memory. We never have to go over the wire in Postgres ML. Everything can happen on a single server, on a single shard, in a single memory space, including the GPU inference. So I wanna talk about, when we say end-to-end machine learning, like what does that actually mean? This is a diagram put together by Andreessen Horowitz. They're a famous venture capital group here in Silicon Valley. And so each one of these boxes represents a function or operation or server that you need to be running and consider when you're actually doing end-to-end machine learning in real time in production. There, you'll recognize lots of names in lots of these boxes, but there are some obvious ones, like a feature store and a feature server. There's nuance here. Not all of these terms are entirely well-defined. Some people at conferences speak about them as if they are, but when you actually get into any particular organization, you'll realize that there is some ambiguity. But it is nice to have a roadmap laid out like this. The only takeaway I have from this slide is that Postgres ML needs to do all of these things and does all of these things. So while many solutions are point solutions, our goal at Postgres ML is to take raw data from data sources, store it in a single database or a sharded data cluster, and then be able to connect clients. And the great thing about connecting clients is we don't have to support every single language and every single connector because it's Postgres. It's already done. When you have a 35-year-old database, we don't have to worry about integrating with every piece of technology out there because it's already been integrated. And there's a whole host of other solutions that operate on top of Postgres that provide good, solid workflows. If you want data pipeline management that we don't do that, well, there's great ones for Postgres that do. Like you can use Airflow, you can use DBT, you can use any of the solutions out there that I generally prefer open source ones, but lots of proprietary vendors also support and connect to Postgres. If I haven't convinced you that some of the capabilities of Postgres are interesting and powerful, from a strictly database perspective, I think there's an interesting machine learning perspective here is that all machine learning models are a generalization of the underlying data. They should ultimately be a compression artifact of all of the data that they're trained on. So useful models that serve some purpose in the world. They should be small relative to the data that they were trained on and they should be modified infrequently compared to the data, which means that the data itself is bigger and it's more dynamic. And when you have something that's bigger and more dynamic, you want it to be, there's this concept called data gravity. You don't wanna be moving the ever-changing mountain of data to your application layer. You wanna move that small little ML application and these small little ML applications, they are really just a few function calls, for the most part. Now, what's going on at Meta? What's going on inside Google? What's going on inside some of these fang companies or those people producing foundation models? Yes, there are research teams and yes, they have several hundred lines of code that they publish with their paper for their innovation. But those several hundred lines of codes are wrapped up and encapsulated behind a single entry point function call that you pass your natural language string and out comes an answer. And so it's much easier to take that model that comes out, now they're coming out once a week with a new foundation model, but take that once a week and put it in your database rather than take all of the data that's constantly changing and bring it up to your model on every user request. So I wanna rag as a hot topic here today. A lot of people are talking about how to do rag. Most of the ways to do rag, whether you're using LangChain or Lama Index or any of the other homegrown rate ways. Now you start with a user query that's coming in from the left-hand side of this graph. You'll wanna do both a SQL query against your data store. I have Mongo in this slide deck because it's popular, but Cassandra is good too. You'll also wanna send that natural language incoming query to HuggingFace, have it generate your embedding so that you can send your embedding vector to your PineCone vector database. You'll do your nearest neighbor lookup, get those back in LangChain.  Then you take that, you generate your prompt using your PyTorch pruning model because prompts only can be a certain amount of context.  You send your LangChain over to OpenAI to generate your final response. This takes a second or four to run this whole mess. We support all of those same equivalent open source models and functionality inside Postgres ML. You can do all of this in a single query in Postgres ML and it's much faster, it's much simpler. It's a lot less infrastructure to manage. It's a lot less surface area and networking calls for things to go wrong. So the way we do this, Postgres ML provides three functions that cover classical machine learning. There's a training function, there's a deploy function, there's a prediction function. I'm gonna zip through this because we've only got a few minutes left and I wanna leave time for questions. But there's lots of documentation on these functions. These functions take a lot of arguments. You can train with over 50 different algorithms. Deployment helps you manage the life cycle of as you train new models and your data changes, you need to be able to deploy which model is active for any particular project and predicting is how you use a model. I do wanna, I'll cover these. These are the hot new LLM functions. Anything that you would call Huggingface Transformers for, you can call inside of your database. When you call pgml transform inside of your database, you pass it a Huggingface Transformer model string. We will go, your database will go download that from Huggingface, it will cache it in RAM, it will move it into the GPU if you have a GPU in your database. It will then pass any arguments from any tables or any other inputs in your query to that model and give you that model back. You can also do fine-tuning in the database and of course, generate embeddings in the database for your RAG applications. I'm gonna hop over real quick. I just wanna show you what this looks like in practice. This is our homepage. This is a little SQL query. Let's see if I can blow this up, so I'll look. This is a little SQL query. In this case, we're using the transform stream version of the call. It's a modification on transform. The web has had web sockets to let us do these dynamic updates for the past five years or so. Postgres has had cursors for the past 20 years or so. They do basically the same thing. As you can see, we're using a model from the bloke if you're familiar with the current LLM ecosystem. This is one of the new fun, mistral models. We can pass it an input with just a phrase of AI is going to, we can tell it how many tokens we want back. We can hit run and it's pretty quick that mistral can generate a bunch of output here. The output to me seems pretty reasonable. Obviously we can change the string. These are just inputs. I like open source AI better than I like AI. I like that it's response to open source AI is more positive and even cooler. So this is really fun. We have a bunch of these different generation tasks that you can do. Again, we support pretty much everything HuggingSpace supports out of the box as a wrapper on top of that or an integration point. Another thing that we do, if I sign in real quick, let's see how it might actually be signed in another tab. Yeah, so I've got a database running here. I'll show you real quick some embedding stuff, some RAG application stuff. What I did before this talk is I downloaded all the Amazon reviews. There's like 5 million reviews from Amazon with movies.  You'll notice that our notebooks are SQL. Oh man, with this exploded font. These rows are not looking good but I'll try and find one here. Here we go. This is an example of the Amazon data. I'll scroll on past that. Get down to one of the more interesting things. This is an example called to the pgEmbed query. You can see that we can pass it a transformer model to actually perform the embedding with. We can pass it some input text. We can run this and we can get back an embedding for that from the database. So in this case, you don't have to worry about your inference service. You don't have to worry about your model store. You don't have to worry about any of those other components that I showed you in that Andreessen Horowitz box. They're all right there inside your Postgres database. So you can access them all with a simple query. And once you can generate embeddings inside the database like this, if you're familiar with Postgres and you're familiar, well you can see how big these embeddings are. But if you're familiar with common table expressions and the real power behind composable queries, then you can start to build a full program in a single SQL statement. So we'll take that embedding that we had generated in the previous example. We're not actually going to return that embedding because that's 10 kilobytes of data that's useless to a human being. We'll leave that in the database as a common table expression. But then we can do a nearest neighbor lookup against five million other embeddings that we've already computed in the database for all the other Amazon reviews. And what we want is we want the Amazon movie reviews that match closest to best 1980s sci-fi movie. So we can run this query and you can see it takes about 70 milliseconds to return the top five movie reviews for out of five million Amazon movie reviews. And you can see the review body is best 80s sci-fi movie. Best 80s sci-fi horror movie is the blob. So you can actually see that rag applications are pretty simple to build with just a SQL query and a database if you just add two or three function calls to them. It's super fast, it's super efficient. It's all yours. This is 100% open source. You can take it and do whatever you want with it. It's MIT licensed. We also have hosted versions if you're not as comfortable hosting your own database. But I think I'm at time now, so I'll stop there. But if anybody has any questions, happy to step over to the site and answer them after this talk.


Join me. We're in Austria. And what's in Austria? KTM's HQ. The simple thing was that we were invited by WP Suspensions to spend a couple of days here and essentially to learn a bit more about their suspensions. We're gonna have the chance to discuss and spend some time with their R&D department, which is honestly amazing. I'll be able to actually discuss some details with the people that actually do the suspensions, so can't have a better talk than having an engineer discussing engineering things, which is amazing. And also to learn how to better take care and handle the suspensions. All of this obviously to not only to become a better professional, but also to then giving you some juice on it. You'll see all about it on the next Offroad Engineered videos, which I'll be stoked to be doing on the next couple of months. And so just really looking forward to these couple of days, which honestly look tipped off. See you in a bit. So, first stop of our journey. Still a touristical bit, but if you've never seen or you don't know this building behind me, it's easy. It's the KTM Auto Hall. It's essentially a place where you have the whole history of KTM. Join me. I'm gonna share everything that I can with you and sit tight. The KTM Auto Hall opened in 2019 and it's a museum for the whole history of KTM. But not only that. On the first floor you can check how a bike is made, piece by piece, from frame, engine, suspensions, every little bit.  Let's explore. So in this part you have an interactive board that gives you a good understanding of how a bike is built from the first moment. Every bike starts with a design and then it's converted to a physical shape that can be worked on. And it's really interesting because it gives you this interactive model, gives you all the small details that starts with the designer and then goes over to the engineering part and the testing part. So it's really interesting. So in this part you can see all the different frames that shoot different kinds of bikes. And it's really important to understand how all of these small changes on the frames and sometimes big changes affect the purpose of the bike. And if you really want to know the small details on how frames are made the way they do, have a video just for that. So this is a really interesting interactive guide where according to the specs that you insert on this chart, type of riding, what kind of terrain you're riding and your ability of riding will output some suggestions of settings that you should have on your bike. It's really good to understand that there is no perfect setting for every terrain or for every type of riding. So it's something that KTM gives you from this guide and it's really insightful. So in here in these top three options you have mud, sand and hard pack. Let's choose sand and ready to race, driving style, obviously being a bit courageous for my part. And now it's really interesting because on this bike specifically they give you the low speed compression clickers on the shock, the high speed compression adjustment, the rebound clickers and also on the air fork and the air pressure that you have to have. And furthermore you can edit your setting and it shows you how you should do it and what kind of variables that you have to take into account to adjust your bike. So it's really interesting to understand that there is no perfect solution. It's always a compromise between type of terrain, riding style, it's a compromise. That's the definition of suspensions but it's really cool. In here we have the E-XE which is KTM's electric off-road model. And we have to give some credit to KTM because way before these new players are entering the market on the electric off-road part. KTM is already making this bike for quite some time and we have to get some credit to the innovation that they're bringing in. And it's really interesting to get such a thorough explanation on how these bikes are built, whether it's the batteries, the engines or the ergonomics that go into making an electric bike and it's really interesting. And maybe we're going to hear a lot more from these bikes in the near future since the transition is clearly going electric so let's see. On the second floor you can see kind of a timeline of the company where you have all the different models and bikes the brand has been making since the foundation of the company. And in the middle you have a really special room. And now let's move over to the special room, the room of champions. So what you just saw was pretty much an immersive experience in this room where it gathers all of the most relevant and pretty much every KTM winner and champion that made KTM what it is today. And it's really interesting to see that from all of the champions you hear their beliefs, their opinions, their way of life that make them become so successful. So it's quite motivational. After leaving the champions room we are faced with KTM's latest off-road champions right before heading into the adventure range. On the last floor we have access to Motohall's workshop as well as an area dedicated to the workshop of the future. How cool is this for a workplace? This is where they usually do the renovation and the restoration of some models but then go on to the exhibition and it's an actual workplace. I think I'm bleeding orange. So just finished the Motohall experience and gotta say if you're a motorsport enthusiast this is a really nice place. Almost heaven because you really manage to understand everything that has been the history of KTM from their early developments until the most recent wins. So it's really cool. It's a really cool place to understand how our sport has progressed so far and really recommend.