bulk editing activities, it's one of the main projects for MOPI teams in that release. And in this spring, three, four, four bulk actions has been added to the core and they are ready for integration review, all of them and two are ready for push today. So one is the defining the availability for both sections and activities which is that kind of model. You can see in the top right part. So now you can select several activities or several sections, you cannot mix them up but you can select one of those and then you can click on availability and just change the visibility and make it stealth and whatever you can do with it, just with a single model. The same happens with delete activities. Now it's bulk based and MOP sections is already for integration review. So we are just remaining the MOP actions for the compromise we get on UI. So it's quite a nice project nowadays and hopefully we can do a demo on the next demo.


Welcome to the Paul Canabines show. I'm Joe Wamba joined now with Paul Canabine head coach of the Men's Lacrosse team. Coach, let's take it back to that Nazareth game that you guys were able to host on Wednesday. Pretty even with face offs.  And able to win the ball back to have that explosive offensive showing. Yeah, I thought we did a good job in that game. We played really well as a team overall and put a lot of pressure on them. I thought we shot the ball incredibly well early and that really helped us get out of them. And I think Ross played really well stopping their shots. But facing off, he's one of the better kids in the country at 73% as a junior and Justin our freshman after losing a few did a great job coming back. And I think he won half of the face offs he took against the kids so that's really good and against one of the better kids in the country. But Justin is one of the better kids in the country and he's really done a great job all year facing off. But our defense led by Dom and Dylan Harris, they did a great job putting pressure on Naz all day and we were able to get the ball up and go. And they got a very talented attack. They really share the ball and do a good job of that. So how we played was great. You guys traveled up to our IT on Saturday. You guys had played them to open up this season this year. Two different teams on both sides of the ball there. Just talk a little bit about that performance. Well, we just started out a little bit slow. They hit us in the face a little bit there earlier in eight, three, first quarter. After that, I think we really settled down and did a great job. We adjusted how we were playing them. And Ross played really well going the rest of the way. We just didn't quite make the plays you needed. They were just a little bit tougher than us. And I thought when we had good shots, their goalie played very well and made the shots. They made the saves to say they needed it. And they really did a great job clearing the ball and then their 10 man really made some really poor choices. They give them some easy layups, unfortunately. And that's what happens. You miss a layup on one end. You get a layup on the other end. And that's usually what happens in the game. But you know, I took my hat at the RIT. They were the tougher team. They played well. And they're really playing a great brand across. All their seniors, if they play so many of them, their seniors really played very, very well. Let's talk about our seniors a little bit. You graduate a whole bunch of them. But they're going to do well in their careers that they move away from. Yeah, absolutely. Kyle Downifree had one of the better careers here. He should be a first or second team all-American. He did a tremendous, tremendous job all year. Our Mitty Dees or Jake and Parrish did a great job all year. And Wade, he did a great job up front as a senior doing a good job there. So we had some really good seniors did some really great things. And Ross in the net obviously had a great end of the year, which is really great for him. He held us in a lot of games and being real steady there to help us out. But we'll miss a few guys. But we got a lot of good guys coming back. And we're a pretty young team. And a lot of our freshmen coming up are going to be real contributors next year. And the new freshman class coming in some transfers we got, we should be all right. It's nice to return the defensive player of the year as well. Yeah, and Dom had a great year. He'll definitely be our first team, first, second, third team all-American. He had a great year. He guarded the ball. He was a great, most of all what Dom did was a great leader for us. And that really showed what he did off the field for our guys. And that's what we feel real confident coming back defensively next year that we should be a great defensive team. Well, Coach, I appreciate your time every week coming up and talking to us in post-game and pre-game. Well. And it's been a pleasure. It's our pleasure, Joe. Thanks a lot for your help. He's Paul Canabine. I'm Joe Waman. This has been the Paul Canabine Show.


GX4 XL. Well, people like the longer barrel for velocity. They like the longer barrel for Accuracy and you're gonna get that with a longer barrel across the board But if you go with this it has a 13 round mag and 11 round mag additional backstrap But they did a nice job and they kept the same trigger now. I measure this trigger right at Five and a quarter pound flat trigger Nice crisp break Five and a quarter pounds with a reset that I put several rounds through this and I had no problems whatsoever


Okay, so for the rest of the afternoon, we have a number of reports and presentations. First up, we've invited Rex Chisholm to be with us. Periodically, we like to give council updates on some of our research programs. It's been a while since you've heard anything about the electronic medical records and genomics or eMERGE network. Rex is a member, he's one of the PIs on eMERGE. He's also a former council member, so he knew his way down to this room and he knows the drill. It'd be interesting seeing him on the other side of the podium for a change. eMERGE, I believe, has 11 PIs, if I'm right. 12 or 13 now. Well, I'm going to get to them in a second. Dan, okay, it's a point of debate apparently, but there are at least 11 and Dan is one of them. There's also a coordinating center and two sequencing and genotyping centralized centers. So we've given Rex the near impossible task of speaking on behalf of all of these investigators. And he's going to do that for us now as he gives us an update about eMERGE. As he's getting to the podium, I just want to say that Rex also has been the chair of the steering committee since the very, very beginning. So it hasn't been fun yet. Well, that may come soon. It's great to be here and it is interesting to be on the other side. I think I realized it was six years since I've cycled off council. So it's been quite some time. So it's my pleasure to be able to talk to you a little bit about eMERGE. And there are several people sitting around the table here and in the room that could equally well have given this presentation today. So they'll correct me, I'm sure, if I screwed up too badly. So today, you can see eMERGE consists of about nine clinical sites, two sequencing centers and a coordinating center. And most of the work of eMERGE gets done by work groups. You can see them listed here, clinical annotation, genomics, pharmacogenomics, return of results in LC, EHR integration, outcomes, and phenotyping. And one of the things I think that in my view has led to the real success of eMERGE is, although it's hard on all of our investigators, is we actually meet together as a steering committee three times a year, which is unusual for I think most consortia like this to meet as frequently. And we make sure we devote a large percentage of the time at these meetings to the work group so that people can actually have face-to-face time together rather than just sort of the anonymous phone conferences that are also part of this process. There are, as you can see, six subgroups that eMERGE focuses on. Familiar implications of return of results, because that's an important thing. Once you know something about you, then obviously there's big implications for your family. Think about HLA, return of results, legal considerations, info button, and I'll come back to this in a moment, participant survey and phenotype variables. And then we've been lucky recently in receiving some additional support in the form of supplements, one for geocoding. And this has turned out to be a really powerful approach that eMERGE has been able to apply. Another for phenotyping, applying the OMOP model to the eMERGE sites. That's very new, and so we're just beginning that. And then a healthcare provider survey to help understand what it is that we're doing to the healthcare providers as we start to think about rolling out genomic medicine. So for those of you that aren't familiar with eMERGE, it's currently in its third funding cycle, eMERGE 1 started in 2007. And I think it's fair to say that there was in 2007 a fair amount of skepticism about whether one could actually use data from electronic health records for research purposes. In fact, I was often told that I was crazy to even think about it. And it always offended me that I couldn't use it for research, but yet it was used for something I think actually even way more important, which is to take care of my health. So I think one of the good points of this is that during the course of eMERGE 1, we were actually able to demonstrate through a variety of projects that, in fact, you could data mine electronic health records, create a cohort, and then do genomics discovery on that. And in some cases where we were replicating known variants, you could actually identify genetic variants that had been observed previously in completely purpose-built cohorts. One of the other things we learned in eMERGE 1 that was really important is that this was a really, could be a really efficient process, because once you developed the cohort and did GWAS genotyping on them, you could actually completely repurpose that sequence with the depth of electronic health records and completely reuse that data over and over again. And I think as you'll see in a minute, we've used that now probably close to 40 times to look at different phenotypes. In eMERGE 2, which started in 2011, we began to ask the question, well, can genomic findings be applied to clinical care and how? And so we did a few clinical implementation pilots, and it was actually during eMERGE 2 that we also began to focus on pharmacogenomics, and I think that was one of the most exciting outcomes of eMERGE 2. Then in eMERGE 3, we've moved on to think about sequencing technologies, how do we use that to both improve discovery and then how we use it to do implementation? So the aims of eMERGE 3 today are to sequence and access clinically relevant genes, presumed effect gene function, in 25,000 individuals. We're about halfway through that process now, I'll show you in a minute. Assess the phenotypic implications of the variants that we observe, then go the next step of integrating those genetic variants back into electronic health record and then use them for clinical care and ask, A, what are the process issues that are related to that? For example, how much burden do we put on the providers that have to deal with this? And then ultimately, of course, can we demonstrate any improved health care outcomes as a consequence of that? And then, obviously, one of the things that we're always very interested in is, can we produce community resources that can be shared broadly in the community and help broaden the area of impact of eMERGE? So to think from the standpoint of impact, one of the things that we have today then is 110,000 participant genomic data set. And what's special about this genomic data set is that all 110,000 of these individuals are people for whom we have access in a longitudinal way to their electronic health records. That means that we can look not only on temporal changes that happen, but we also have the ability to look at what happens in repeat measures, to look at trends, and to look across these 110,000 people in a fairly detailed way, reusing that information over and over again as we redo analysis based on phenotype from the electronic health record. Underlying all of that is some technology, which is the eMERGE record counter. There's an open-facing version of the eMERGE record counter, which anybody can go to and ask in the eMERGE collection how many type 2 diabetics are there that have a BMI over 40 and get a number. We have a thing called Rodin's Rule that says whatever number you get, it's only going to be half that number at the end of the day, but nonetheless it gives you a sense of what is possible in that 110K data set. And then we have Sphinx, and I'll talk more about Sphinx in a minute, but Sphinx is about how do you actually do the integration of phenotypes and actually deploy those more broadly. So one deliverable, then, that comes out of this collection of participants, deeply phenotyped, if you will, based on their electronic health records, is to actually look at about 83,000 of these that we have comparable GWAS level data on. And what you can see here is a nice example of, I don't know if you can, yeah, you can see that. This is just a principal components analysis that shows the ancestry of that 83,000 person cohort. And what you can see is it's actually quite a diverse collection of individuals, and it's, I think, one of the values of having this 83,000 person cohort and the ability to use that. So right now this is the basis of a lot of the work that's ongoing in Emerge 3 as we generate new phenotype algorithms. We can apply this to this data set and actually ask questions about new discovery of associations that we have not previously seen before. The next deliverable is the development of the EmergeSeq platform. I said one of our goals was to identify 25,000 people, actually sequence a panel of genes, and then think about how do we use this in returning results to the individuals, think about their families, think about the providers, and then obviously think about long-term outcomes in terms of health. So we went through a process where we took the ACMG, then 56, now 59, but actually we were recovered because it turned out we had all of them on the panel anyway. Each site then was able to produce its top six. So those were dependent on either a particular implementation strategy they were interested in or a particular discovery project or a mix of them. And we ended up with a total of 109 genes on the EmergeSeq panel. After careful analysis by our clinical action ability workgroup, we came to a consensus that 68 of those were actually clinically actionable in one way or another. And we can go through a whole series of SNVs, but at the end of the day where we are now with this 109 gene panel is that we have 68 genes and 14 SNPs that are returnable, that we believe are clinically actionable and actually can be returned to, will be returned by all sites. I should take, as a brief aside though, say that some of the things that might not be clinically actionable actually are actionable in some interesting kinds of way. So I'll just use an example of some of the early sequence from our site has identified a few individuals that have variants in genes that have been previously associated with obesity. And when we return these data to those participants, it's interesting, there's an incredible sense of relief because there's some explanation for their condition and it's not just a lack of willpower. And it also gives them the ability to think about action ability for their children where there might actually be some ability to intervene earlier and have an impact. So I think one of the things that we're learning is action ability may be a much broader concept that we actually need to be exploring in more detail. To give you a sense of how this is looking, we're about halfway through the sequencing of our 25,000, about half of them are being sequenced at Partners Brode and interpreted in the laboratory of molecular medicine at the partners. And our results are broken up into two categories. They're indication-based results, so people who were sequenced because there was an indication like, for example, colon cancer is one of the indications that was used for sequencing. And then there are other sites where the indication for sequencing, there was no indication for sequencing. It was just sort of a random probability that somebody would actually have a condition. And it's interesting to see the results. They're pretty consistent. In the indication-based return, we're seeing about one and a half or so percent that are positive. In the non-indication-based return of results, we're seeing in the broad case about 6% positive. And in the case of Baylor, we see pretty similar results. The positives are about 3.2 for indication-based and about 3.1 for non-indication-based. And I don't have time to go into some of the variations here, but they turn out to be interesting. And then what kinds of conditions are we seeing? Well, you can see here, for example, we have about 40 cases in the Baylor cohort that represent some cancer condition. We have about 20 that represent some cardiomyopathy. We have about 10 that represent a long QT. So there's a fairly heavy cardiovascular implication here. And then a few others in terms of malignant hypothermia and hemochromatosis. Another area where it's obviously very valuable to be able to take that information from an electronic health record is to actually develop a process for how do you do electronic phenotyping on data extracted from a health record? And so we've spent a lot of time in a merge developing multiple electronic health record-based phenotypes and to understand the process that is valuable in terms of making those phenotypes as robust as possible. And then we deposit all of those into a phenotype knowledge base called FeeKB, which currently has about 37 finalized and publicly available phenotypes. So if you're interested in understanding how we did those 37 phenotypes, you can actually see the algorithm on FeeKB. To give you a sense of how the workflow works, the way it typically works is we begin with one site who has an idea about how to create a phenotype algorithm. And then after they've done it, they pass that algorithm off to a second site who usually is able to actually improve the algorithm significantly because they either view things in a different way, they have a different collection of data at their site, or there's some usage of clinical values in a way that's slightly different between sites. So that validation step turns out to be really important. And then once it's been validated, and this usually involves some iteration back and forth, we then go to a sharing phase where all of the sites in Emerge then have the opportunity to run that algorithm and contribute to a synthetic cohort that then can be used for discovery purposes or eventually for implementation purposes. And then obviously the ultimate goal of that is to publish. And this has led us to develop quite a few tools in the area of phenotyping, using KNIME as a way to actually package these up and share between sites sort of a computable package that can be deployed at a new site. LMAP, which helps us provide specific mappings between different phenotype elements. I've already mentioned the Emerge record counter and FKB. And so all of these are important tools that are available publicly on the Emerge website and are also very valuable in terms of doing this kind of phenotypic development. So in phase one of Emerge, we did a total of about 14 phenotypes. Each of those then went on to be the source of a GWAS study. In phase two, we did another 15 phenotypes that got us up to about 30. And then we added 27 phenotypes in phase three that'll get us ultimately by the end of next summer to about 70 phenotypes that we will be able to deploy across the network, across either 83, or in some cases, even 130,000 person cohort to actually begin to look at discovery. So another important impact that came out of Emerge that's been, I think, very widely used to this state. And I think I have to credit Vanderbilt is taking a lead on this. And what they did was they developed this tool. Hopefully, everybody's familiar with GWAS at this point. But it's sort of a GWAS flipped on its head. So instead of taking a condition and asking what genetic variants are associated with that condition across the genome, it takes a genetic variant and asks what conditions are associated with that genetic variant across all of what we call GWAS space. So it's a process by which basically ICDs have been clustered into groups and used as a basis to do a FIWA study. Just as an example of that, one of the things that Josh Denny led was a FIWA of the entire NHGRI GWAS catalog of SNPs. And it started with a discovery arm that asked how many SNPs could we apply across this FIWA space and actually then discover new associations. And that was also based on a replication arm which said across all of GWAS catalog space, how well do we replicate what's already been identified? And the answer was there was a remarkable concurrence with this across the entire GWAS catalog. So this is really a really highly powerful approach to think about how to look at genotypes in their association with multiple phenotypes. And I have to just mention one that was a lot of fun that the Emerge Network participated in was to actually look at variants from the Neanderthal population that existed in our collection and ask what conditions were associated with the presence or absence of those Neanderthal variants. So it turned out to be quite interesting to be able to think about how you could look in today's current health record and think about some of the evolutionary aspects of the variants that are different between Neanderthal and the current population. So the final area that I wanted to talk about was Emerge Pharmacogenomics. We did a panel that was developed actually by the PGRN Network. And it was an 84 gene panel that allowed us to sequence across 9,000 participants, 82 of those genes. And then as there were clinical implementation guidelines for pharmacogenomics, put those back in electronic health records and use those predictively when a person was going to be prescribed a particular medication. So we have this 9,000 person cohort. We have both things where there's a clinical implementation guidelines. And most of the sites have now got clinical decision support that supports that at the time of prescription of a medication. But then as you can imagine, in Sphinx we also have a lot of additional variants in those genes that have never been seen before. And so it creates yet another opportunity for additional discovery. And we continue to collect utilization and outcomes data.  And this, again, shows you the principal components analysis. Not quite as diverse as, of course, the 83,000, just because it's smaller, but still pretty good ancestry diversity in this group as well. And then, of course, we've focused on how do we put this information back into electronic health records in a way that can be used by clinicians as they either prescribe a medication or as they talk about risks in families as people have information that says that their family might be at risk for something. And this is very heavily dependent on a project called the Infobutton Project, which uses reusable tools that are available to most electronic health record system. And we've also put together a clinical decision support knowledge base that provides a basis for doing further clinical decision support for people that might be interested in doing that. And then we're in the midst of experimenting right now with the use of common tool called DNAnexus, where all of the emerged data from the EmergeSeq platform, from the 83,000 GWAS study, is available in this DNAnexus platform, where we can build common pipelines and common tools that anyone in the Emerge network is able to use. And we think this is a particularly powerful approach to think about how do we improve the consistency and reproducibility between the different sites. To give you a sense of the impact of Emerge based on publications, what you can see is that we've got a pretty strong record of publications. We've been a little slow in the last couple of years because we're just in the process of sequencing all of these genes. But there are 119 manuscript concept sheets that are waiting and in the works right now for us to use that data as it becomes available. And we're very proud of the citation history for the Emerge network, cumulatively between Emerge I and March of this year, accounted for about 17,000 citations. You can see about half of them in the area of genomics, but also ranged across phenotyping, return of results, and electronic health records implementation. And then, finally, is another way of thinking about use of this data is all of this is, of course, deposited into DbGaP. And you can see that the data is getting increasingly utilized as approved authorized downloads from DbGaP so that people can reuse that information as well. I've mentioned most of these tools already, so I won't be laboring them. But if you go on to the Emerge website, which is a link, is present in one of the things. I can't remember it off the top of my head. You can get access to all of these tool sets for use. And then where we are right now is future deliverables. So DbGaP submissions for Emerge 3, this 25,000-person 109 gene panel. We're going to do about half of that, an interim submission that will be happening momentarily, and then the full submission that will be submitted later. We've already submitted the GWAS E1 through E3 imputed set, or are about to. We have developed IT support at each of the sites to put clinical implementation back in the electronic health record. And while I can say that in one or two sentences, it's non-trivial. And it requires a lot of cooperation between the clinical side and the research side. And I could talk for hours about that, but I won't bore you with that. We're focused on outcomes, both in terms of the implication on families and participants, but also on the health care provider group. And then, as I've already alluded to, creating and deploying an additional 27 phenotypes and then using those phenotypes for discovery. So I know it was a whirlwind tour, and I already went over my time a little bit, but happy to answer any questions any of you might have and talk with you further about what Emerge is up to. And I don't know if any of my Emerge colleagues want to say anything at this point as well. Questions for Rock? Right. No, it's just a fantastic resource. Thanks for everything. Thanks for letting us do it. I have a really mundane question about DNA nexus. How does that interact with DBGaP? So you have to presumably first get DBGaP approval. So you can use DNA nexus, or is there some special status there, because you're not actually downloading the data, but you're logging into your system? Well, right now, the nexus is available only to Emerge investigators who have appropriate data use agreements in place. But so it's just for Emerge investigators at this point in time to see if they're using it. So in that case, are there plans to open that to a broader community? It's a great question. It's not cheap. So I think the question will be thinking about how do we manage the costs of that. But I think if we can demonstrate that it's, and now here I'm just my own opinion, if we can demonstrate the utility of it for the Emerge network, I don't know why we wouldn't want to think about a resource like that that would be broadly available. We have a question related to that, because DNA nexus is a company, right? Yes, it is. So what is the line of thinking around choosing to go with a commercial solution like this that actually gets built and paid and developed as part of a research project like Emerge, but later on looks basically the community to using that specific product? Is that a concern? Is that something that, you know, for sequencing we do it all the time without, I think, much concern? But for software, I don't think that has been generally the solution. I think this is an experiment. It's to actually ask the question, how well does it work for a consortium like this? And the answer is just unknown at this point. It may work very well, in which case people may find it compelling, even though it is supporting a company. But I also appreciate there's a strong desire to have an open source community that can actually support these as well. So I want to raise two issues related to that. One that I think is on people's minds, which is financial, as I said, it locks you into a particular solution. But the other that has to go with open source, I think often when people think about open source, they think, well, it's for free and I can do whatever I want with it. That's one aspect of it. But for methods people, the reason that they're so gung-ho about open source is that then you actually know what methods are being applied to your data. If the source is not open, you actually don't know. And that becomes increasingly a concern these days. And I think even when a commercial partner in this area is introduced, since they can still sell their product in certain ways, there should be a lot of consideration to that aspect, which goes beyond the fact that they built a data platform or not. I agree completely. It's probably worth saying though, one of the features of DNA Nexus is that anybody can actually build a tool and deposit it into DNA Nexus. So it's on top of their platform, but there is the ability to do methods development for analysis that sits on top of that. But your point's well taken. First thing, it's good to move the conversation back to a merge. First, it's very impressive. Could you talk a little bit about the success of putting genomic data into the electronic medical record? But not the technical aspects, but in particular it seems the value of that is downstream. How often, and it's probably too new, but how often is that used by the healthcare entity going forward? Yeah, so that's a great question. So I'll talk about what I know best, and then I don't know if Dan might want to say anything about their experience at Vanderbilt, but we, for example, for the Pharmacogenomics Project, have built clinical decision support that fires at the time a provider writes or attempts to write a prescription for a medication. And if there is a relevant variant in the record that would affect metabolism of the drug that they're about to prescribe, clinical decision support fires and says, this individual is a poor metabolizer. You might consider an alternative medication if there is one, and that's all built into the system. Today at Northwestern, that is firing only for the people that were recruited into the Emerge Pharmacogenomics Project, but because of the way we've built it at Northwestern, it's a flip of the switch to actually expose that to the entire health system, which now consists of about 7 million people. So we're in actually discussions with our healthcare system about when might you actually turn that on. Now in the case of PGX, there's another prerequisite, which is you have to have the sequence of those genes in order to pre-populate the electronic health record. There's some costs associated with that, so one needs to think about that. But the IT infrastructure is already in place and literally could be turned on by the flip of a switch. I think as we start to think about risk scores for complex conditions, then it gets a little more complicated and their economic benefit of that, I think probably is too far in the future for us to talk about. But things like this infobutton project gives resources so that a physician can at least ask a question and get to underlying literature and get to additional sites that might provide information relevant to the particular condition they're interested in. And it's great to hear that there's an economic analysis that's involved and I assume that's part of Emerge that someone's looking at the cost effectiveness of this. So one of the outcome measures that the outcomes workgroup is focused on. You know, that's great, my compliments. Carol? Yeah, this is awfully interesting and it was a whirlwind tour and I found myself wanting to go, but wait, and also, is there a publication on that? And so if we can have your slides, but also there's hundreds of publications, I'm sure. I'm most interested in the return of results and how you actually have made that happen. But my question is whether, I mean, this is such an impressive effort at body of work, it's gone on for a long time and you've really refined it in a number of institutions now with these sort of moving toward all of the things one could think about as being maybe possible, hopefully possible within the all of us study. So what, it just really strikes me, you don't wanna invent the wheel a hundred times. Is there, are there efforts to kind of harmonize what you're doing with the kinds of projects that are going on with all of us? The answer to that is yes. There's actually quite a bit of cross fertilization between people that are sites for a merge and people that are sites for the all of us project. Josh Denny, who I've already mentioned from Vanderbilt is the PI of the all of us, at least for the Consent Project. And then there are several other people involved either in the genomics group at all of us or in the new caps group, but it's about privacy and security for that that also includes several emerge participants. In a lot of ways, I think it's fair to say it may be overstating from a little bit of a point of pride, but I think emerge has been fairly influential in informing how all of us moves forward. But I think all of us, and I'm a very active participant in the all of us project from our site and through committees, it's a huge operation and it's an aircraft carrier. And I think emerge is much less of an aircraft carrier and emerge is actually a much more nimble organization where we can test out some of these things before trying to deploy it in the aircraft carrier state and think about how, what works, what doesn't work and really go for the best of strategy as we think about transforming what we've learned from emerge to the all of us project. So Rex, can I just jump in before we go to Val and just as I was gonna ask a similar question and I guess what I'd like to hear your impression or maybe Dan could weigh in because I know there's a couple of points of joint involvement is what you just described makes total sense, but is that coming from the people that are involved in both or are you getting traction within the broader all of us consortium to embrace emerge as that high lit small boat out in front of whatever metaphor you wanna use? I think the answer to that is it's early days. I mean, so I think early on as the all of us project was being conceptualized, emerge got looked to quite regularly as what can we learn from what emerge did? I think it's fair and now my own opinion has a active participant in the all of us project. All of us is so focused on building the platform for participant recruitment that we've not spent as much time as I, for example, would have liked to have seen us spent thinking about downstream uses of the all of us platform. It's sort of if you build it, they will come and that's the stage that we're in now. And I think when we get to the stage of actually doing all of these things, then they're gonna need to look back to things like Emerge and Caesar and some of the other activities NHGRI has funded as a way to learn about how to do this better. Dan, do you share that view or you wanna add anything?  Val, I think you're up next. Rex, very nice, thanks. My question goes to this anecdotal, or maybe it wasn't anecdotal, that's my question. You talked about how someone was made to feel less guilty because they had some obesity predisposing variants. Was that based on anecdotal data or do you have a study? No, it's anecdotal. I mean, it's like, you saw, so we're doing 3,000 people at Northwestern, at our site. Each of the sites is doing 3,000 people. And out of that, it's like four people. So I call it anecdote, not a study, but I think it does create the opportunity for us to understand the impact on them and on their families in a storytelling kind of way, and that's where it starts. And then once we've heard that, I think we can talk about in the future how might you think about a study to do it, but absolutely anecdotal, no statistics, they're too small to even think about statistics. Yeah, so the flip side of that is someone might just plain give up, right? Because they say, oh, it's in my genes. And so a genetic determination is needed to balance that out. But the other question was with regards to returning information to the patients with regards to clinical action ability, where's it going with regards to heterozygotes for things, recurrence risks? Yeah, so that's clearly something that we need to be thinking about as a next step. But we've got, like I said, we're about halfway through the sequencing of, so that's like five to 6,000, it's about 11,000, I guess, at this point. We don't yet have much experience with returning those results. You're saying we're getting, call it 3%, people that have a returnable results. And I would say at most sites, we were lucky we were one of the first sites to get our sequence back, and we've only returned probably to a dozen people at this point. So it's just too early to tell, but that's exactly the kind of question I think we're well positioned to answer. Thank you, that was very helpful. So one's more general question, one's a specific question. So I'll just, the specific question is it look like based on patients' phenotype, the proportions were no different. If I understood what you're, based on indication, it seemed like it was still about 3%, which is what people have reported across for incidental findings. I was curious if you meant patients with cancer, for example, had no more than people without. That was the more general thing since this is part of the open session. The answer to that one is yes. I mean, if you were selected for colon cancer, the other findings, I obviously knew your colon cancer, the other findings were no different. Oh, I see, okay, so I misunderstood that. But since I think we're still in open session, you spent a lot of time talking about all the work on phenotypes, but I think for a lot of people, it may not be clear what you mean. Like, so you just talked about obesity. So this whole long list of steps is what it takes to really look across electronic health records and agree on how to measure obesity. Is that what those series of steps is about? Precisely, so let me use the example of type two diabetes. So that was actually one of the first algorithms we did in Emerge One. And surprisingly, how complicated it is to produce an algorithm that actually has a high positive predictive value and a low negative predictive value of type two diabetes. And so it involves several steps. It involves several if-then, yes-no branches. It involves not only codes, but medications and a variety of aspects. So even something as relatively simple, notice the air quotes, but relatively simple as type two diabetes took a fairly complex algorithm. And then they just get more and more complicated. And one of the things that we're finding now in Emerge Three is almost every single one of the algorithms that we're doing in Emerge Three is requiring some significant natural language processing to be able to look at free text fields that are in the electronic health record. It's exceptionally powerful when you do that, but it requires some infrastructure. I do want to amplify one thing that Rick said, and that's sort of in response to Val's question as well. So if there are four out of 3,000 people with a funny genotype, funny genotype, and then you go back and look at their phenotype at Northwestern, that means there'll be 35 or 40 once the data set is finished. And then that's a number that I think starts to get interesting. If it's one out of 40,000, no one's gonna, then it's all about families and traditional genetics, but if it's 40 out of 25,000, then you can start to look at the electronic record itself and say, how penetrant does this vary, and what are the phenotypes, and what are the responses? So I think that's one of the things that we're on the edge of learning, and of course, once you get to a million, then all bets are off or not, but I think that's one of the real appeals of this going forward. Any last comments? Not? Thank you very much. Thank you, Rex. Extremely helpful.


Under Florida law a breath test is presumed to be accurate. Now whether in fact it is accurate That's another story. I can tell you that Florida law considers them to be accurate. These are machines that can Have errors like any other computer, you know, I have had instances in which someone hires me I pull the documentation on the machine There are issues there that either one could get the breath test thrown out because of them not maintaining The instrument properly Finding numerous errors with different people that cast doubt on the accuracy and reliability of that Intoxicizer, so are they accurate? They're presumed to be but do we often find issues? That lead us to believe that their results are not accurate in certain specific cases. Absolutely


TEFL jobs in France France is a great destination for the qualified TEFL teacher. Most of the major cities in France have TEFL jobs available, with the great majority of job openings being in Paris. The language schools in Paris offer a lot of business-oriented English courses, so if you have a certificate in teaching business English or previous experience in business your chances of getting a TEFL job are significantly increased. France is one of the easier countries among the EU members to get a TEFL job if you don't hold an EU passport. Business experience and qualifications are held in higher regard than TEFL certificates. Basically, if you have a BA and are comfortable in a business setting, you have an excellent chance of finding work. This is even more the case if you can speak French reasonably well. Finding a TEFL job in France Getting a TEFL job in France will be easy for you if have previous experience teaching English as a foreign language, have an EU passport and can speak French. It will also help if you can be flexible and give yourself some time to find a job. Finding a TEFL job is fairly easy to do without being there. Many jobs are posted on the internet on various TEFL websites so you can arrange things from your home, from where you may be currently teaching English or taking a TEFL course. Most teaching jobs are with language schools but many teachers like to earn extra money by offering private lessons. You can charge between 15 and 20 euro per session for this. The French Foreign Ministry offers an English teaching assistantship program for citizens of the US. BEFL in France Sign up for our top quality TEFL course in France at one of our two exciting locations. Upon completion, you will be proud to hold your internationally accredited TEFL certificate in hand. From there on, you are able to find jobs teaching English all throughout France, Europe and virtually anywhere in the world. The demand for qualified English language teachers is high and you will sure not regret your decision in coming to France and taking your course. Come to France and spend a month training to become a professional teacher. France has long been one of the most visited countries in the world due to its beautiful landscapes, culture and fashion. French cuisine is also renowned all over the world to be one of the most exquisite. France is divided into 13 metropolitan regions with unique local cultures and dialects. Some of the famous provinces are the Provence with its endless lavender fields and the Côte d'Azur with its splendid beaches in the south, the Brittany with the famous Mont Saint-Michel Monastery in the north and German-influenced Alsace in the east. TEFL France The need for qualified ESL teachers is high in France, which means that you will have little trouble securing a suitable position after you complete your training course. Many of these positions can be found in the bigger cities, such as Paris, Marseille, Lyon, Toulouse or Nice. Our two training centers are located in Paris and Brittany, where there are 20 course start dates every single year. On top of our four-week intensive course program, we also offer the combined course option at both locations.  TEFL course in Brittany France Brittany is a unique province on the Atlantic coast. Here, times goes by a little slower. It's green hills and small local villages lure in visitors from around the world. Our training center is located in a charming town where you will get the real northern French experience. Due to the convenient public transport system, you are able to explore the greater region without troubles. You will be getting to other provinces and bigger cities such as Paris, Nantes or Bordeaux or within easy reach. TEFL course in Paris France The capital of France is one of the most visited cities in Europe. People from all over the world are coming to Paris and are drawn in by the magic of the city. Enjoy the finest pastries and wines here and get inspired by the fashion of the people of Paris. This is the UNESCO World Heritage Site, due to the abundance of historical landmarks and exquisite architecture throughout the city. Be sure to go on an extensive exploration tour in Paris to see the Louvre, Arc de Triomphe, Eiffel Tower and the Notre Dame Cathedral.


Bookabus is a new, flexible, on-demand ride-share service for rural areas in West Sussex. Why not Bookabus to visit friends and family, attend a medical appointment, explore local destinations or connect with major bus and rail services, travel within the zones and get picked up where and when it suits you. The bright green buses are easy to spot, are fully accessible, with low floor access and designated wheelchair spaces. To Bookabus, use the Ride Pingo app or Phonus, pay by card on the app or the bus. Enjoy your ride with Bookabus.


Now, since you have studied databases or you have been working in the domain of databases, so whenever somebody says databases, what comes to your mind is relational database. That is fine, because relational databases have been around for maybe three decades or maybe longer. But as we have been discussing and we will discuss further that for the new challenging problems we need to have new solutions. One of those new solutions is offered through the NoSQL databases. So in this module, I will describe about the NoSQL database paradigm. So I have an ambitious module coverage, which I show you right now. So we will looking at database construction, structure, records, association, organization, replication, technology, and of course, search tools. Why search tools? Because as I mentioned earlier, I will mention here. And in the next module also, that there is a lot of similarity between the search engine and NoSQL databases. So let's go into more details. So what is a database?  Single logical unit. What is a data form? Data form is that I have clusters of servers on which the data has been distributed and there are many benefits of doing this. And then we do a partition. When we use the cluster, of course, I have to do a partition also. Partition separates places, distributes the data over those clusters. And I have this collection and of course the schema. The schema is the structure of the database. The schema is defined before creating the database. But remember that in the context of NoSQL, there is no requirement of schema. Just I mentioned it over here for the sake of comparison. So let's look into more details. Records. So what is a record? A record is a single atomic unit, right? And you have no these things from the context of relational model. But what I'll be discussing over here is a table. A single class of record in big table for the context of, you remember I discussed, they're also called tables in a triple store. They may be called subject RDF types or named graphs, depending on the context. In this document store, they may be collections, okay? So we have these over here. I was talking about this in the context of the big table, which you remember, which introduced by which company. Associations. Primary key, you know, a key, and other than that concept of RDVMS, RDVMS concept, a key in a key value store URI. URI in a document store or IRI, IRI in a triple graph store. Foreign key, you know, a relationship, the link or an edge in graph theory that indicates two records have a semantic link. That relationship can be between records in our same tables. Of course, that relationship for a triple, for a triple, this is different. And for RDVMS, VMS, this is, this is basically a foreign key, primary key relationship. These are different things. Just for the sake of understanding, I have mentioned them over here. Let's look at more details. Now storage organization, we have this server over here, user one. This is original data. And this is the full replication, okay? This is the database replication over here. Of course, we can have this replication also. But the point is more interesting over here is flexible replication. Provides application, controlled application of data between databases in different clusters. Updates may not arrive in the same order they were applied to the first database. Typically involves some custom processing. So you can see over here that this replication is through the replica server into the replica database. So these are some of the differences. And how the replication technology works in the context of disk replication, database replication and flexible replication. And these are true in the context of NoSQL. Some of those things which I've been explaining work for the relational model which are described over here just for the sake of understanding. Search tools. Search tools indexing is there. Now what is the benefit of indexing that the indexing takes me directly to what I want without searching all of the data, all of the documents. And of course, there is the reverse indexing also, which points back towards what I'm looking for. Now there is one difference between what we get through a query and what we get through a search engine result. The query gives an exact match. But over here in the context of search, in the context of NoSQL, where there is NoSQL structured query language, we get the results which are close to what we are looking for. And then we differentiate, then we go into more details, then we perform further search and that's how we work. And this is one of the similarity between the NoSQL solutions and the search engine in terms of architecture, in terms of functionality, in terms of search. So for the sake of understanding, I have described all of this. Thank you for your time. That's all I have.


On page 43, we have more songs in the key of F major, but a new rhythm, a dotted quarter note. Listen as I play these two songs and pay attention to the rhythm of the dotted quarter.


Right good morning. Thanks everyone for coming. Just to be clear if you were looking for a conversation about usability You're in the wrong place. That's next door. We're talking about opening this and learning analytics John's role at the opening initiative is to focus on poor software. I need to focus on initiatives and partnerships and Because we think that the analytics space is one where we definitely can't go it alone I felt like this was a good place for both of us to be talking. I do sorry But we dive in one of the things that's been pretty interesting to us about the past day and a half It's just how much is going on in the analytics space It's going to give you some warning that we basically rearranged our slide deck over the past half hour to try to capture a lot of this stuff That we've seen happening And we really feel like given the people that are in this room Maybe the worst way that we can use this time is for the two of us to stand up here and pontificate What we'd really like to do is move through the slide deck reasonably quickly and then move on to some discussion It really pushes us forward to talk a little bit about how we as a community can build better analytics tools and build better resources You can plug into those tools We take our scientific approach to course design what that means is we recognize that every course or every learning bar We build is really hypothesis about what things going to work and we collect data to actually evaluate the experience and drive continuous improvement process And we do this through communities of use and evaluation So really this is really sort of showing what our biases are in the beginning The scientifically based approach to course development is the water that we swim in and it has a huge impact on how we perceive the analytics space Um, obviously that's not going to be the same way that all of you perceive it And that's why the conversation becomes interesting So what we like to walk out of here with um How many of you read our abstract? Sounded pretty ambitious, right? We thought so do that we've reread it so What we'd like to do though is try to deliver a little bit on that promise and dig in a little bit on what some of the tensions are in building reasonable learning analytics, especially around the dimensions of what's happening in use driven design context of OER How it is that variety and adaptability seem to be creating some conflicts for us in building effective analytics platforms What tools can we be building what tools already exist? And uh, what drives it all? How do we get the data? What are the needs and challenges there from there? Let's talk about what a community-based plan for better analytics would look like and hopefully commit to some action What are some effective next steps and really building this stuff? There's been a lot of chatter about analytics over the past day and a half We feel like this is a good opportunity to submit some plans So, um, I'm sure a lot of you've seen our feedback loop slide Um, the point that we come from is that one of the greatest opportunities of technology based instruction Is this ability to invent assessment throughout the experience and capture data to drive these powerful feedback loops to Just better support the learner to help instructors understand where's tips are in that learning process and intervene earlier To better inform the science of learning so that we can refine our approach And to affect changes in the course design experience because we view this as a huge huge opportunity In order to do that to realize this we need assessment and data And the reason that it's such an opportunity is that the huge amount of OER that are out there and the enormous amount of students That are using it now not to mention all the potential students that are using it in the future Can be building and creating huge piles of data that can be used to drive these analytics systems No longer is the information that we're capturing no longer does the information that's being generated need to be confined to the individual classroom and confined to simply one faculty members analysis and interpretation The size of the data here gives us the opportunity to do things that we think can't happen in context outside of OER So we think OER learning analytics and especially the science of learning have a potential relationship that couldn't have existed prior to OER's cooperation so making that in this space it's it's hard to make some sort of data because there's so much variety and Josh took some of the numbers that we we also pulled out because I don't know too much idea But you look at 19,310 modules like new connections. We have Something like 2,000 courses in MIT's open course square 2,600 videos of Khan Academy and all these different things are capturing different types of data in different formats of different systems Or maybe they're not collecting data at all We have all sorts of spaces 600 free courses at the EOU and much much more out of the EOU comments And so, um, you'll question me. What's driving this? What's driving this cooperation? We've done all these great resources We we put them out there releasing access to them. Um, but why do we have so many? Why do we have so many? And a huge part of A huge part of what's driving this comes out of OER's original mission to expand access, right? Very early on the assumption is if we have more stuff out there Then we've expanded access the more things that are out there The more students are going to be able to take advantage of it the more fact would it be able to take advantage of it but it's It's been expanded. I think this this urge towards creating more and more stuff and contributing your stuff has actually also been pushed forward By the philosophy and definition that we have for what constitutes an OER this need for reuse redistributing Revising and remixing the content has led to us putting more and more stuff out into the space And we'd like to think that it's possible to continue with this philosophy But avoid the fifth art which is continuing to recreate things that already exist Instead what we'd like to do is continue to think about things in this context But add this notion of evaluating resources so that we can take what's work working Continue to build upon it rather than continuing to recreate the same set of statistics courses the same modules and biologics It's a recreation becomes a barrier to reuse and if we create opportunities to coalesce and To improve through evaluation. We're actually getting back to our original intent Which is to revise and remix and this proliferation isn't just happening in the OER space It's happening in your institutions, too So, you know, it's it's just happening on a different scale It's something that happens at Carnegie Mellon is that every department feels like it needs its students to learn how to program So they send them off to take an intro to cs course We go through these cycles where that course would be embedded in the computer science college And all the students will travel there to take their course and then the departments will decide this isn't beating their needs And so everyone builds their own intro to programming course We have one in the business school one in the it program We'll have another course popping up even within the college of computer science This and this is this is crazy, right? It gets away from the entire notion of what a university ought to be this notion that we have a center of expertise And uh, we're going to take advantage of it. This isn't just a cnu Everywhere that you go in statistics folks are feeling like they need to build these specialized statistics courses Beyond the core statistics that exist in the mathematics and stats department People feel that they need business statistics research statistics statistics for nurses medical statistics So this isn't just a problem with the oer But the low cost of oer pushing things out into the space for the world to use Actually exacerbates this pre-existing problem or at least our pre-existing sense of wanting to create an awful lot of stuff And we don't want to we want to acknowledge that context is important that proper motivating examples are important in these environments And this is across these areas, but there's also a foundational core that comes across all these different types of statistics And so what we're not talking about is the the one statistics course What we're talking about actually engaging across all these different varieties to find commonality Um, and so just quickly what's driving change in these settings? Why is it that I want to have? 15 different computer science courses or 20 different statistics courses And it turns out that it's a lot of different things in some cases folks are being motivated by data Simply looking at what student performance is happening within their college within their department I think market demand tends to be a reasonable driver for folks that are looking at what skills students need become employable But it's also acknowledged that there's an awful lot of intuition I've got a feeling my course isn't working or simply preference That an ongoing debate with my college roommate who teaches english, uh, the smaller boards full out in Pennsylvania who will call me about once a semester and say hey Completely changing my intro to film class because I'm bored teaching these films. Well something about the old films not work I don't know. I'm just tired of teaching it Um, so there's real space in changing things simply based on preference and not based on what we believe does work Or what could work better And that creates all sorts of problems Um, and use that quality is not the variable that we're duplicating a lot of effort It's hard for someone to consume resources if they don't have a basis of choice They don't know what the pedagogical intent is what context of use is targeted for and what I didn't work at all And you really hit this proliferation You I think it's a very good option if you can't just sign on the thing that you actually need It's hard because that it's hard to evaluate because we're not building pools of data that are deep enough because they're They're just this is fused across a large number of resources Uh, and that makes it hard to improve and hard to scale up things that are successful And all of this is to say really bad effective. This is a miss And we're not the only people that are asking this question. If you were yesterday's keynote, um That's our sorry you trade off I can push the button if you don't understand You heard dr. Cantor asked exactly the came sent question. How do we know what's working out there? How do we evaluate this stuff? So from our perspective at OLI we would say that an OER is effective if it can demonstrably support students in meeting Articulated measurable learning outcomes in a given set of contexts um We're going to risk throwing off the rest of our presentation right now and ask is everybody's comfortable talking about effectiveness in this way for the rest of the Presentation do we miss something that you think is important? Wow. Yes, uh, sorry. I think you So the only thing I would add to that is, you know, I think that some of the research that you've done on funder statistics Of course that says that that the student are able to actually achieve the learning outcomes more quickly Especially when we're looking at the challenges around development or general education lower division courses That the piece at which learning outcomes are achieved Is the one other piece of that that's critical So that's what we intended by the given set of contexts that we have to acknowledge that context is important when we're measuring effectiveness So we have we agree Um, can you define measurable learning outcomes for me and um, maybe I'll explain where we're Qualitative and kind of new pedagogical ways of thinking come in there But the way that when I whenever I hear learning analytics or whenever I've seen it I've seen it in terms of here's multiple choice questions and here's, you know Your way that you're going through the module and it will tell you where you are and where you need to go but a lot of us have to work in you know, program-based education and Kind of Social participation in you know diversity of spaces that doesn't really apply So I'm wondering guys, you know are thinking about that and what you know how your model connects to that. Yeah, absolutely So that's a big question. Yeah, it's a big question. So this has been a few seconds on it So I can actually give an example. So then we've been we've been working on We're working on a number of social learning tools one and working with this called knowledge form It's a collaboration we're doing with the University of Toronto And in that it's a basically a discourse space for Folks to engage in knowledge feelings. It's the idea of developing 20 percent of skills by authentically Engaging in discourse around set of materials. And so we're not doing multiple choice questions But one of our we can still have goals for students to come out of that one of the goals is in statistics For example, we want students to we say authentically participate in the discourse of the domain So when they leave the course they can speak like a statistician they can understand something in the york times and so How can we assess that we can look at we can develop rubrics to measure their engagement in these environments We can look at their peer interactions. We can look at the growth and spread of ideas and communities We can we can look for new forms of assessment that go beyond multiple choice questions So multiple choice questions are one vehicle But rest gives this project the hd catalyst initiative that's looking at this question of new forms of assessments that global consortium project that we're We're involved we're involved in So it's not that we don't want to we don't want to teach a millennial likes to just Tutors and cognitive tutors or questions. There are other ways that we can use data even as their context But we talk more I'm glad to talk to more more about like your perspective in great depth In this definition, I'm not really sure the phrase demonstrated demonstrably support students What that means it seems like that's a little fuzzy And I'm not sure how you would actually evaluate that looking at a correlation between student aid completed this learning module and did better on this Assessment, is that what you did? So I'm not that I'm not real sure that's pretty fuzzy for me in terms of the definition So I think that in that context what we're you know what that teaches on is this How it is that we're defining and measuring our learning outcomes If our learning outcomes are actually measurable then we are able to see that a student that has taken advantage of or made use of a specific OER has Perform better on these outcomes has achieved the outcome in a better way than students That's great that sets us up for where we're going with the talk so if we can hold that question Come back Let's go back to your question to get at the end and see if we've answered it as we get a little further And so so if we can move forward with this definition of effectiveness the question comes out is so why are we Take a look at this now. Why are we getting on this now? and Well, it's hard. It's hard to even define. I mean this discussion here kind of gets up at some of the some of the challenges there It's costly. It's not something that individual faculty members. Do you can do alone? Simply because the students are giving them the time to do it But also because there's a whole range of expertise that's needed around assessment and evaluation and learning science to do this We don't always have the mechanisms to support them in that way It can be threatening. I mean It's we have if we know something doesn't work. We have to do something about it So that can be threatening and we have disparate systems with standards different ways of collecting data And then this question how do we measure effectiveness is And so the just all comes down to say is that to make this happen. We need better enabling processes and systems That's right Even when the data is there to actually drive these feedback loops requires tools for individuals Educators administrators to be able to look at and actually interpret what's happening in the data So this is where this continuing demand and need I think for learning analytics platforms is coming from And one of the reasons that this has been uh, which feels like a pretty hot topic of conversation at this conference Is people are starting to recognize that we've got lots of information, but we don't know what to do with it Yeah, which is really What do we do though? Okay, we've had a big discussion about the need for learning analytics But what does it really mean in practice like what what is learning analytics and how do we create and use them? And we think that this conversation ends up being difficult to have because often tools that whether it's us that are building or other organizations that are building them end up attempting to build proxies to measure learning rather than Attempting to actually authentically engage on the assessment and evaluate whether things are working or not working And this is natural, right? I mean a lot of this comes down to what we do in the academy when we give grades Grades in the end are intended to be a proxy for whether or not a student is achieving their outcomes Similarly though when we're talking about learning analytics We're also looking at not just the immediate actions of a student against a learning outcome But how that students existing in the larger context of a class how they're existing in the larger context of a university And so we're using this phrase learning analytics to try to capture Lots of different things ranging from this authentic assessment of learning outcomes all the way up into how's this student performing and engaging in my class How are they existing in their university or college career? And so this kind of comes to the definition of what we're looking for for what We consider the learning analytics and that's to say that data collection Is it really data isn't released unless it's used and that's the sense making of data can actually be a lot harder Than necessarily collecting sometimes it's easy to collect data sometimes really hard to make sense of it And so we're defining learning analytics You know Reporting and reporting that allows you to make decisions Intervene and and be actionable and so I give the example that you could so there's it's informative to know that there's a correlation between a given set of student behaviors and success. Let's say let's say that's Let's say that's logging into a blackboard or registering for a course or Signing up for an OER by a certain date. We've actually actually heard this before so that's great It's really informative so you can predict average students But how do you intervene because that predictive power goes away if you make the intervention make everyone sign up early That makes the power goes away Or you can look at we also often get an instructor to ask us about things like participation data Which again very useful to know if your students are participating who's participating what they're doing how much But is making every student do everything really going to give us a better picture of What where the learning state is Because that's what we're talking about. We want to make sure that we're collecting data that can lead to Interventions to improve outcomes And part of this then ends up trying to ends up being about Explicitly what kind of analytic tools we're trying to develop is making the distinction between educational and academic management analytics Things that are going to help you predict average students across the whole of your institution They're figuring out what kinds of behaviors there are useful intensive supporting average students Classroom management analytics really understanding and predicting across learning outcomes how students are engaging with the material Where they're being successful, but then down in the weeds how are students achieving individual learning outcomes? Where are they failing and what do you need to do differently in your classroom to try to serve those students? All this leads into the problem of data collection, right in order to drive these systems What we need are common mechanisms for data collection and this has been Made really apparent to me recently in participation with an nglc grant David Wiley and I were actually just having a conversation about this that when you end up having lots and lots of Different kinds of data at varying degrees of richness being collected It's almost impossible to apply a standard set of tools and to give information back to the faculty that are That are actionable So one of the first steps in getting to the ability to build a standard set of tools and platforms Is starting to get at data collection These however are technical problems and the technical problems aren't the hard ones to solve It's the second grade And these are the issues that are actually that if we can solve these taking care of the technical problem the data collection is easy So where we want to go at this point the ideal situation would look like well We have some common data standards and comparable metrics that we can all take advantage of We have analytics Enable know we are we make these these platforms and systems Just available as you create it makes you it's there you have the power of data collection analytics We develop some commonly accepted ownership privacy approaches and we all have a community-based commitment to measuring effectiveness through assessment So how do we how do we approach this? Well, let's let's start by bringing together what already works What already works on data collection what already works on gathering evidence and what works for analysis tools There's already a lot of great stuff out there and that's again That's been what an awful lot of this conference has been about They at least from my perspective is finding out just how much stuff is out there that either isn't They're well publicized or that we just haven't been talking to Um, and we can we can run through these. Okay, so you know within the olie we have our learning dashboard It's uh instructors. It gives a learning outcomes center view of student Student performance. We work with the beginner science and learning center. They have a large data repository called data shop Which is designed for learning scientists and educational data miners The old evidence hub is a great place to crawl to that is collecting Evidence about what makes so we are effective. What makes useful? We are effective We have projects like learning registry which we're trying to tackle this these data collection distribution problems We're looking at forming schemas where we talk for a lot of us to talk about data in common ways um, and there's also some words that we're doing in our project about media and evidence And yes, this is what the current cc olie project has really been about it's been about uh Explicitly building courses that are going to address the gatekeeper problem But doing so in a way that's not building a single olie or we are that's intended to solve our problem To instead creating a community that's going to both help develop evaluate and improve that oar and use it So that once you have something that works you're able to improve it in specific directions In some of the more individual contexts that you need And so we have some existing things, but we also need to build new things in each of the same areas And so a lot of this work is driven by different types of data So we need to define what these different types of data are we have metadata peridata contextual data the context of the use Behaviors in behaviors logging into a system viewing a page watching a video Interaction level data. Did I respond correct or incorrect? What are the what for what are the practice opportunities that are available to students with the skills that go with them? What sort of support they know they receive from an automated tutor system? And then there's this even raw data that provides this opportunity for finer-grained education research to better the science going things like mouse clicks and individual interactions in the system And we think that one of the other things that we need to build are better mechanisms for making sure that the data is shared Right now all of the data that's out there ends up being wrapped up in individual lms's when we are lucky We can get some common agreement on okay, you know I'm participating in this grant and part of this means that i'm going to feedback the data to you But what we'd like to see is a world where taking advantage of the oer means explicitly Agreeing to share the data so you know that this mock-up of the creative commons license here Thought cable was going to be in here. We're going to harass him about Lost my opportunity for that this sense that if i'm contributing a piece of If i'm contributing a resource into the commons one of the things that I expect to get back from that contribution Is that folks using that are going to put the data back into the world so that we can take advantage of it and continue To improve the resource absent these kinds of mechanisms and policies the data will continue to Hide on individual servers and really the only way to make this happen is through a community-based approach And this is the way you know our project versus purchase force development We still have to go of course We're actually bringing together different areas of expertise and these interdisciplinary teams that draw on the expertise of credit email and then also Other institutions that we work with On the previous line, what was the sd? Oh shared data. Oh shared data. So you're proposing a new license That would be share alike, but also I don't you know, it's the top of the slide. We actually argued a bit about whether it should be uh Whether it should be shared data or evaluate alike actually, but that's something that we can get into conversation And so, um, you know real benefits to any grace approach is we feel is this allows us to build alignment In ways that help to break that reusability paradox because we can bring different perspectives on a domain together Find that common core of things Build alignment around a common set of student outcomes. Yeah, we have to adapt for business statistics We need to add some different examples there. Sure. We need to make sure the context is right, but um, We can actually if we can build pedagogical alignment in a community Then we create greater opportunities or reuse that that paradox starts to break down And from our perspective, that's one of the only ways that we're actually able to build better tools So in this way, I think this is one of the ways to break this concern between the infinite variety of courses And folks that are concerned that we're trying to build the one single course that is communities coalesce around a resource It starts to be improved and developed in in specific contexts Uh, so we bring us together to have a full spectrum and by this we mean that full stack of data So we really capture a complete picture about What's happening in in the system in the use of data using that to inform improvement And trying to look up data comprehensive approach to our view on things Uh, what this eventually gets us is you know, even the current analytics tools that we're talking about that we've shown you End up being the tip of the iceberg We end up being able to build incredibly effective learning intelligence systems That are not just going to improve the OER but are in the end going to improve student experience And and there is a consequence here that's worth acknowledging Which is that we are giving something up in this approach and that is um, you know, we're have to open our minds Are allowed to be changed by evidence, you know, it's we have to take a scientific view To construction and to construct any resources and when things don't work We have to try to say, okay, you know, we might have to Improve things abandon things change things modify things to really find What does work and what are the relevant contextual variables that make that scalable? So next steps, um, yes, we kind of outlined a little bit general approach here We were talking about standards a lot yesterday to the problems that space and we kind of feel that innovation has to come first So we want to talk about you know by innovating standardizing and scalability and stop signal So we'll kind of jump to our conclusions Uh, which is this the ways to hit this are you know, not just a shared community commitment to assessment and evaluation But building this definition of analytics enabled OER so that as we continue to create and improve AR We can OER we can do it towards this definition a common approach to data and then building these shared and private analytics platforms It's recognizing that some of these platforms are going to need to exist in the comments But this doesn't need to preclude folks from building their own platforms Which are going to exist whether commercially or inside individual institutions. It's speaking the same language And then these platforms need to be the enabling infrastructure that allows technology to get out of the way of Folks building these resources so you can build whatever your OER is and not have to think too hard About all the technology all the data formats all your operability concerns that go with analytics based approach So we've completely destroyed our intent to have a good discussion by talking too much over our slides, but the questions or comments So I think the things that we're currently using most extensively are the learning dashboard tool that we show Which is intended for instructors We're now building that out to make it a tool to be better used for course developers And the tools that are available from pslc's data shop Which are publicly available and further which we've taken some of our data and already made publicly available We plug into so both the tool and the data has been made available Mostly from our stats course, but we're looking at other courses. Are you using data mining? Yeah, we actually we we are We have models We do we we actually have a competition going on some of the models. So we have a researcher who created a Knowledge construction module for statistics. We actually have a lot. We actually engage in the community of data miners Learning learning science data mining community through the bitter science learning center We actually have complete resources to try to find better models. So I'll do something that we're It's actually a the attempt is to beat the researcher. Can you build an automated model that will do better than the researcher can do? Sure, yeah So what's the what's the process in the earliest date to get a prototype or a reference implementation of this to share the platform idea that could federate Experiences from lots of different sources and places on top of these So it's something where we're moving aggressively on now Talking with some of the learning registered folks here and other folks here about what opportunities We've been bringing systems together and what we want to do is we're working with uh, the tax grantees in the Department of Labor Consortium Planning services these folks we want to mention that platform involves to providing some of these basic analytic functions so that folks who are building those resources can Boots drop their way these cycles Months not here. So I think it's the short answer And so that's actually a call everyone here is to come talk to us Give us your thoughts on this approach and uh, let's figure out how we can work together to do the two innovation That's standardized in scale Thanks. Sorry to have