Okay. Thank you, Ilya. So, multi-center mutation call or calling, what is that? A somewhat enigmatic topic and possibly mundane, were it not for the fact that the mutation calls that we generate underpin so much of what we do in TCGA. So, that makes this a necessary topic and one that has undergone itself great evolution during the course of the TCGA. So, what I'm going to do for the next 15 minutes is review the approaches to somatic mutation calling to consider what it is to call mutations with one caller. Talk about the early benchmarking of somatic mutation callers that we did early on in the project. Look at the early trials of three-center calling and adoption of standards for the three-center calling and look at the current status of multi-center calling and new developments in mutation calling. So, the whole trick to this game is to be able to distinguish error from real variation, the real biological variation that we seek. There are two sources of that error. Some comes from the sequencing machines themselves and are inherent in the base callers and fortunately most of this is randomly distributed and the base calls come with calibrations of Q values that enable us to distinguish the truth from error and we're able to filter most of this and yet some of this escapes because there's a fair amount of it across the 50 billion reads that we take for exome and when these errors happen to be coincident, they're found then at allele fractions that might be similar to what we're looking for in the very heterogeneous tumor environments that we have that's been described in other talks. I'm not going to go into that in this talk but then there is systematic error that comes from mapping and alignment ambiguities. There's a lot of difficulty with 100 base reads and the structure of the genome that we have that makes this a very tricky problem and this leads to high quality errors that are much more or high quality base differences that actually are reflective of true base differences but the read just happens to be in the wrong place and so they can easily be mistaken for mutations. So the way the current callers work is that they all have some what I'll call for lack of a better term a truth engine that distinguishes the real variation from sequencing error and these formulations which are largely Bayesian or log odds based formulations give or output tens of thousands to hundreds of thousands of events which then which then have to be filtered and it's at this point that heuristics come into play and each calling center applies heuristics in a slightly different way and most of the variation or a very large fraction of the variation that passes those so-called truth filters then get filtered out. The best documentation of this is in the paper by my colleague Chris Sebolskis at the Broad when they did their mutech publication and what this shows is that at each sequencing depth you're filtering a significant fraction of the variance away due to these various characteristics which are being heuristically applied. This amounts to up to 90 percent of the variation that came through the first step is now disappearing and this is going to be important as we go along because I believe this is where most of the variation between callers emerges. So with the mutations that we collect from a single caller we get actually very nice profiles of significantly mutated genes. This is one example from colorectal cancer but now we have 10 or 12 tumors in which we've collected similar profiles as this and the profiles end up making a lot of sense. You can place these mutations into pathways that describe what's going on in the tumor in actually in detail that we've never seen before. But unfortunately if you go back and compare the calls that different callers are making on one or two tumors you see a picture that is at first somewhat disturbing and this is where we were with the first benchmark proposed by David Housler back about in February 2011 where here we have on the left what were the somatic calls passing all the filters from each of three centers and you can see that the number of variants in the center is fairly small compared to the total variation discovered by the callers in the union set and in particular a lot more seem to be going on in the variation that was unique to any given caller than there was in the overlap. So then on to benchmark two things didn't look a whole lot better there 310 was the intersect from four callers here and again you can see that going around the outside the number of events that were unique to all these different callers was much larger and so this left the sinking feeling that there was a lot of true variation possibly being left on the table and a lot going on analyzed. David took this another interesting step further and from that, from the next benchmark did a, from the next benchmark analysis where for each center that was being considered an event diagram he looked at the top 100 calls from each patient that was being studied here and looking in the top 100 calls of just the UCSC caller there were 10 calls that the UCSC caller was considering high quality that no other caller was seeing and going around the horn here we go over to the wash you caller looking at their best calls there were 143 that nobody else had seen for Baylor here there was about 1,000 these are unvalidated calls so even though the center is saying they're high quality we don't really know but in here were 55 from the Broad so potentially there was a lot not being, not being turned up in the analysis of any given cohort by a single caller so similar results were obtained in a set of colorectal samples where UCSC and Broad made calls you can see a lot more going on in the unique calls and similarly the high quality variants were numerous in the unique calls so the conclusion from this was obviously the discordance between callers was high and that was pretty dismaying and the high quality calls is defined by one caller were missed were being missed by the others so that particular discordancy was distressing so the suggestion was that at least if we did multi-center calling we would ameliorate some of the possibilities of false negatives okay by the way it's also the fact that if you were to apply callers just to diploid analysis where here you have the tremendous benefit of an expectation of about 50-50 in the reference and variant alleles which is a very powerful constraint on the data if you look at the results from five different callers on diploid genomes they only agree amongst themselves by 57 percent so it's not just in somatic mutation calling that there can be difficulties but even in this business as a whole and my sense is that a lot of this as I said before is coming from the fact that we use heuristic callers to help filter the data and that those differences that we have had in the data are we're basically sampling a very large multivariate space and choosing different components of that space so nonetheless going forward we then began using three-center calling on each of the cancers and the results coming out of that where we now were able to superimpose validation data on it showed that at least in the overlap what was being called was extremely highly accurate so in the three-center overlap shown here 99 percent of the variants validated and even when there were just two centers making the call the percentage of validation was very high out in the Unix the validation rates are much lower nonetheless validation was occurring in the Unix so that the three-center calls at least can pick up false negatives at least there on the table for scientists to consider and accept or reject here's a similar analysis of lung adenocarcinoma data done by the Broad and once again very high 100 percent validation rate in the three-center overlap lower validation rates in the around the outside and more recently in the this is the kidney clear cell project where we had 500 patients this was the largest cohort at this time and looking at validation within 177 of the cases this shows the number of mutations that were validated and the percentage that were valid in each segment of the overlap and you can see a very high validation rate in the three in the at least if two centers made a call and a much lower validation rate in the unique data so this led to two developments the first was a metacolor developed by Terry speed and this metacolor he demonstrated was very highly accurate and so it leads to a recipe for being able to take the multi-center calling data and make it highly accurate in order to achieve that high accuracy though it's necessary to have validation data to calibrate each of the callers that are involved but once that's done you can make very accurate calls from the data and so one possibility unfortunately prior to the marker papers coming out we don't have the validation from all the centers and so one thing that could be done retrospectively now is to go back and with the validation data available and the calls from each caller apply his method and generate even more comprehensive mutation data sets. The other thing this did was lead to a formalization of the multi-center calling and with the recognition that the mutation callers are improving overall and different callers detect different events, the validation cycles were taking too long to lead to expeditious publication of the marker papers and so we began using multi-center calling in order to provide a significantly mutated gene list that's based on the calibrated accuracies of calls made by two or more centers. So this gave us a path forward and broke the conundrum of the problem we ran into with reviewers who never wanted to take at face value the mutation calls that were being provided without validation. So we accelerate the submission of the marker papers now and we don't abandon validation, a validation is still brought in but that can go on while the paper is under review, validation requiring a second independent sequencing event. Okay, so where are we now? The other thing that the multi-center calling does is enable other potentially interested researchers to add their callers in and experiment with the development of new methods and so now we have for the Adrenal Cortical Carcinoma project which is underway and nearing a paper I think earlier in this, earlier in yesterday's session we saw a review of the work with this tumor but now we've got five centers calling. You can see from this that there are still large numbers of unique calls but now in the center through the advancement and improved sophistication of the calling the center is the most heavily weighted so the overlap is looking much better. The fact that we still have a number of calls on the outside and preliminary data that I'm sorry I can't show you here is that among these calls in the unique section we look in the RNA seek data now and there are hundreds of events out here that are part of, that appear to be being expressed and therefore are probably valid somatic mutations. So again we continue to sample from a large space and every new logic that is applied picks up more of that sampling. Most of what's out here are events that are very low allele fractions so they're subclonal events in general although not all of them are. Some of them have just escaped one or more of the heuristic parameters that are used in filtering. So now we have a second generation of mutation callers coming on and one of them is being developed by the genome center in collaboration with MD Anderson which measures a distance per position per sample to reflect a mutation evolution and uncertainty estimates are based on Bayesian Markov model and therefore this method will come with a calibrated certainty akin to a Q value. Method is now being refined called Viper by WashU and there's a mutex version 2 on the drawing board. So the next round of mutation callers is going to have even better accuracy and better sensitivity at low allele fraction. I don't think we'll see these outer unique sections decrease at all and that's a good thing because that is going to be pulling in these very low allele fraction subclonal mutations that we want to have. These new callers are being tested in the dream challenge as we go through dream one, two and now we're in three. The callers from the sequencing centers you can see are at the very top of the list in what is probably a statistical dead heat. I think that speaks very well of the TCGA sequencing centers and so we'll be looking forward to actually the final rendition of the dream challenge which will use real data instead of synthetic data. So in conclusion the TCGA paradigm for mutation discovery is improved by multi-center calling. This enables us to decrease the false negative rates. It delivers a set of somatic SNVs of calibrated accuracy, accelerates submission of marker papers and stimulates development of new mutation callers by providing benchmarking on the fly. A formal metacolor was developed which may be useful in retrospectively refining mutation calls from TCGA tumor sets and finally one parting thought that I didn't talk about is that we're now starting to use validation in or mutation calling in a lot of different contexts now. We have RNA fusions, we have structural variation, all of those mutation modes are likely to experience a similar phenomenon as we see in the SNV and so that needs to be checked and probably multi-algorithm calling will be required there too. So I'll just end with my acknowledgments of my colleagues who all contributed to this talk. Thank you. Okay, one question, one burning question please. For those multi-colors are you using send underlying alignment like PWA or BOTI? These are, yes, they're using the same set of BAM files. Okay, we observe that there are some software files for like PWA, BOTI, if you use different software alignment they maybe have some different mutation call or mismatch. Yes, well yeah, that's going to be important for comparing mutation profiles across tumors and so it is necessary to go back and that's being done in the ICGCTCGA, whole genome analysis now, but that is a very important point. So TCGA is recommended BWA for overall aligner? I'm sorry, TCGA what? Basically I saw lots of data was using BWA, so is this like... Well this, I was talking about whole exome here, so, sorry, yeah, so yeah, we were using BWA, everybody's using BWA, but in different ways actually. Okay, I'd like to welcome our next speaker, Kyung-Fan Lehmann, who is from Memorial's Long Kettering Cancer Center. We'll talk about extensive trans- and cis-QTLs revealed by large-scale cancer genome analysis, please.


ServiceNow Knowledge 14 is sponsored by ServiceNow. Here are your hosts, Dave Vellante and Jeff Frick. Hi everybody, this is Dave Vellante with Jeff Frick. We're here live at Knowledge 14. This is ServiceNow's big customer event. About 6,600 people, up from about 4,000 last year as we've been saying. It's kind of tracking the growth of ServiceNow which has been pretty meteoric. We heard from Mike Scarpelli, the CFO, Frank Sluteman. They're really doubling down and it's exciting to see. We're here in San Francisco where all the action is. Jeffrey Moore is here, author, consultant, pundit, all around smart guy, cube alum. Great to see you again. Thanks for coming back. Thanks for being here. So you're speaking at the CIO decisions. I love the fact that they got so many CIOs here. Real CIOs, right? Yeah, yeah, yeah, yeah. A lot of times these conferences get the infrastructure guys, but so what's the vibe like over there? Well, you know, it's kind of cool because if you think about ServiceNow and you go back, say, 10 years, this was all about how to make IT more productive around the ITIL model and you'd use these automated services to do this stuff. What's happening, and Frank nailed it in the keynote, he said, look, this infrastructure can be turned inside out and you can service enable the entire enterprise, not just IT, you can service enterprise. HR, marketing, any other shared service you can turn into a bunch of services that you can sort of call in and use ServiceNow as a platform. So the CIOs, it was all about, well, that's a different vision and so how do we map from the old way of sort of thinking about this as an internal productivity facility to this new way of saying, no, this is an enterprise enablement platform. That's a big move. It's a little bit like Salesforce going to force.com. It's that same flavor. Yeah, so Frank's keynote was talking about how the CIO has to become more business savvy and of course we've heard that a lot for years and years and years, but in fact, a number of the folks that we've had on here at ServiceNow are actually of that ilk, maybe they came from the business, but most CIOs didn't necessarily come from the business. They weren't P&L managers, they weren't running sales. Do you see that changing? Yeah, I think what happened in the 20th century was IT was sufficiently complex that frankly you had to be a technical person to do it. It was just really hard and yes, you needed business consultants, but at the end of the day, you needed 10% business consultants and 90% technical people. I think we've come a long way since then and the next generation of stuff is more around systems of engagement, these things that communicate with each other as opposed to systems of record. And so the profile of the winning IT strategy is migrating from help us run information about our business in the back office to help us actually re-engineer the dynamics of our business in the world, in the present. And that's like going from data to behavior. We call it going from systems of record to systems of engagement. It's a big show. And is that transition, in your mind, is very disruptive? So what happens to all those purveyors and buyers of systems of engagement? Do they morph into systems of record? Do they morph into systems of engagement? Do they just get blown away? No, it's interesting. So first of all, you're never going to get rid of your systems of record but at the margin, we've probably extracted most of the lifetime value from that investment already. So you need to maintain them. And so the industry is consolidating around an anchor set of vendors who we trust to do that. But the growth is going to be, like if you look at systems of engagement, we might have gotten 5% of the lifetime value there. So at the margin, if you have a dollar to spend, people want to spend it in there. So the challenge of being an incumbent is I'm not going to lose my base, but man, the growth is happening over here. So the real challenge for the incumbent vendors is how can I participate in the new world and still maintain my relationships in the old world? Whereas the new guys are just coming in saying, I'll leave the old world to you guys. I just want to play over here. I can get your take on the structure of the IT business. You've observed, as have I, sort of these disruptions and these changes over time. Obviously when we went from mainframe to PC, you saw the competitive line started to get more disintegrated if I could use that term. And competition occurred on those. I see it in Intel's Ascendancy and Microsoft and Oracle, the best database company and EMC was the storage company. And everything was sort of siloed. But leadership, the leadership matrix has largely stayed intact. I mean, even IBM and HP said it's up and down, but it's largely stayed intact. Do you see the cloud changing that, fundamentally changing the economics? Yeah, yes, I think what happened is, so in the client server era we did, we built the stack, what you just described, and every layer of the stack had a leader. Now I think since 2000 Y2K, that stack is being compressed, meaning there are fewer and fewer vendors that are still in that leadership cadre. And as we go to like cloud and computing as a service, you start saying, well, yeah, I still have Cisco in there, I still have IBM in there, but maybe I'm buying them as a service rather than as a set of equipment. So you kind of can feel that world just, I think compressing is the right word. And whereas the experimentation, the opportunity to sort of find new places to go to is very much in this world, outboard of the IT data center where it is about engaging, engaging with your customer, engaging with your employee, engaging with your supply chain. And using mobile things and social and analytics and cloud and all these new technologies, the freedom to do that is actually outboard of the old style IT. So what you described is sort of phenolicopoly and you got these big whales and I'm always asking guys who follow this, are we going to see somebody to disrupt that Amazon is the obvious one, you got a $3 billion company growing at 60% a year with marginal economics of services that look like software. But at the same time, it's okay, they've got this huge lead, but it doesn't just make sense to me that it's sustainable. I mean, because hardware economics never will go to zero. So you would think that somebody was almost like the IBM early PC days. Remember IBM had the lead and they were dominant play. That's kind of what kind of way Amazon is now. Do you see that? Do you see more competition from Amazon? Why is it that they don't have direct competition? So the last book I wrote and the last thing I've been working on most recently is around why is it so hard for the established incumbents to catch the next wave? And the problem is, so you look at why Amazon, why is Amazon so unopposed in many of its initiatives? Well, their business model and their economic model is completely divorced from the incumbent model. And so you look at the incumbents and they're going, it's not that I don't see what the guys are doing. I get what they're doing. I just don't see how I can get my investors or my whole infrastructure onto that new place. And my example of that was Kodak. So Antonio Perez came from HP. He knew what he was getting into. He understood digital. Everybody at Kodak understood digital, but they couldn't get to the other place. So it would call it escape velocity. How do you free yourself from your own past? And you really do have to take a pretty dramatic approach to it. And I think, by the way, I think I'm looking at Microsoft in particular. I think Microsoft is going to give a very, very big run at doing that. But I think that they're still more the exception than the rule. You would wish that every one of those vendors would say, look, you know, because every CIO here, if any of those vendors came in and said, hey, we're going to really try to play here. Will you help? They'd say, yes. They don't want to change their relationships.  It's classic Clayton Christensen, right? In a measure's dilemma. And it also begs the question, as you said, David, it's been the same characters, kind of changing companies. Had not Jeff Bezos and Amazon come in with a completely different model to drive cloud, would the other people have to transfer? So the people you want to give credit to, you want to give credit to Bezos. You want to give credit to Benioff. By the way, Benioff has been the kind of the prowl of a ship that brings in, and Neil Busri at work there, brings in NetSuite, brings in ServiceNow. So the software as a service thing is coming in at one level. And remember, if you were an on-premise guy, how do you move? It's very, very, I mean, how many years did SAP commit an enormous amount of money to say, we're going to have a great cloud offering? And it's just, it's so hard. So, and then you're looking now at these sort of this next layer of collaborative IT and you're seeing Box and Akta and Peng, all these cool things. Analytics and Splunk and Sumo Logic and all these companies are going, really?  I mean, They got a lot of t-shirts. They got a lot of t-shirts. But the point is this free space and they're saying, there's these cool problems to solve. We're not encumbered by any of the legacy. We're going to race ahead. And so if you're a CIO, what we spent most of our time with the CIOs today was, okay, I have established a set of relationships here. I'm not going to abandon them, but at the margin I need them to help me think about the future. I've got these really sparkly new startups. Some, I'm sure not going to exist next year, but some are going to be the leaders. So how do I play that game right now? And the pressure it's putting on the IT organization is the people I know that are good at this are not the people that are good at this. And so how do I, so we had talked about talent and how do you manage and how do you create career paths and do you have an infrastructure officer versus an innovation officer? I mean, it was all around that same problem. Right, and then, oh, by the way, there's Hadoop and Mobile and Big Data and some of these other just open source innovations that are being just thrown on these guys' plate. It is, so from a technology plate, from a technology plate, if you're a technologist it's like bring it on, right? Right. But I think the interesting thing is in most of my career, IT was about the business. So you ran a business and you had IT systems which gave you information about your business. What's happened in the last 15 years is that more and more sectors of the economy, IT is becoming the business. So you saw what happened in newspapers. So IT isn't about the newspaper business. IT is displacing the newspaper business. Google is displacing the media business. Amazon is displacing retail. You know, mobile banking is displacing banking, Airbnb, Uber. I mean, so now we have the taxi guys who worry them. And so you start saying it isn't, IT isn't about the business. It's a digital world. And so all of us, and that was, I think that was probably at the core of the discussion. So which CIO, what do I have permission to be? Which, do my colleagues get this? You know, am I competent to do it if they do? I mean. You've talked about this a lot and you've given a number of examples. So was Nick Carr just dead wrong in 2003? Or was he just too narrow and as far- What he was saying, I believe, is that systems of record are dead. I think at that time, by the way, it wasn't obvious there was anything else. Because, no, seriously, I can remember in truth, you know, the whole venture community kind of abandoned IT for about six or seven years. Drive up and down one-on-one, it was your first time. Yeah, it was, it was. And even in the physical infrastructure, there's still, the IT is the basis of the competitive advantage, not the reporting system. Yeah, and I think this issue about, so I think there's still a few businesses where really IT still is about the business. And you know what, you can kind of stick with whatever you were doing, you'll be okay. But if your business is under an existential threat, meaning the new IT model eviscerates your business model, which arguably, you could say, all those incumbent stack vendors, you know, I mean, Cloud does eviscerate the on-premise hardware data center business model, which was the fundamental foundation of IT as I knew it for all my business career. And now all of a sudden it's like, holy, how do I, how do I, how do I deal with that?  Salesforce is obviously, it's kind of, but they've been around since 99, so there's kind of the exception that proves the rule. I don't, maybe a service now or a work day, you know, we'll see if this market is big enough. It looks like it might be. What often happens is these guys just get gobbled up or Larry Ellison writes a check. You remember he used to say, he's the denigrate people who write checks, not code. And then he became the biggest check writer. And he got such massive cash. He's never been afraid to reinvent himself. You have to say that about Larry. Absolutely not. You've got to know what Ellison is. He changed the game, right? He changed the dynamics of the industry. So do you think we will see another big player and where will that come from? Will it be the SaaS guys? Will it be some of the guys out of the Hadoop world? So here's what I don't think will work. I don't think you can be an established incumbent vendor under this compression power and write a check and get yourself out. I think what happens when you write a check is you just bring a hot property into cold molecules and it loses its thermal. Which is equally evident. Exactly, exactly. So I don't think that will work. I think if you want to be one of these incumbents and succeed over here, you have to actually pull part of your own DNA and capability and literally just jump. And then I think you can acquire it to build a thing there. But what Larry did was he consolidated. He basically was the first guy to figure out Nick Carr is right. I need to buy up all the properties and run a maintenance business, which by the way can... He paints a red and charges more. Yeah, well, computer associates had that playwork in the 80s. It's the same playbook. This is a different playbook. Well, I love what you're saying. And EMC is an interesting one to watch the way Tucci is setting up this federation with Pivotal and VMware and you know, we'll see what happens with the core EMC. And I think that that is, I mean VMware is one of the wonderful examples of where a company did not cause the hot molecules to become the cold molecules. The thing you wonder there though is it feels a little bit like a holding company, if you will. And so, and by the way, VMware is in a curious tweener, right? Like they kind of were the most, they made the old stack incredibly productive. So in some sense, they can feel like they're part of the old world, right? They're probably the newest kid on the old world. But then you think, well, yeah, but I want to, I mean, you look at their plan now, they want to be into software defined networks, they want to be in software defined data centers. They definitely want to play over here. And it's an escape velocity problem. So do all their partners. It's arguably they created the cloud, right? One could argue that that was the basis of what became the cloud, right? Yeah, yeah. Absolutely, absolutely. So what are you working on these days that's exciting? Well, so I think this issue of working with management teams to say, okay, look, this is a self-imposed exile that we're putting ourselves under. You know, we get it, I'll call it the Kodak problem because I don't want to talk about anybody in high tech specifically at the moment. But the point is every management team in the established vendor group puts itself on a self-imposed discipline to make certain kinds of EPS things, certain kinds of growth, you know, whatever it is, the expectations of their investors. And you look at the situation and you say guys, that is a slow glide path to extinction. We all know that. And by the way, off the record. They know it. No, I'm told it, I'm told it. This is not a failure of intellect. This is a failure of will. So then the question is, well, so how do you negotiate a different path? And part of it is you have to be able to tell a story to your investors. Part of it is you have to negotiate a different operating model inside the company. And what they've done so far is they said, well, okay, we've got our established businesses and we've got our innovative businesses and we know enough to keep them apart. So that part is not the problem. And they actually come up with cool stuff. The moment of truth is when can you scale any of these innovative businesses to compete, to actually be a material part of your historical portfolio. Meaning in my terminology, at least 10% of your total revenue going to 20%. And what happens in that journey is at a key point, you have to draw on the resources of your established business and all the people that make their living and they're compensated on getting the next quarter and the next quarter go, guys, I can't make the quarter and do this. And you've got to find a way to say, if we don't figure out a way to pull some of that resource over here and play our next hand, we'll invent everything in the world, but we'll never get it to scale.  Idea number one is it's a go-to-market problem, not an R&D problem. You do not have an innovation problem, you can't get your thing to market. And the second cool idea is you can only do one at a time. And everybody says, well, but the risk is so high, you gotta three or four or five of these things maybe want to work. It's like, no, the sacrifice is so great. If you put two or more horses in the race, people won't even run. So, tough one. That's right. Yeah, focus and don't, it's okay not to make the quarter. That's like on American. It is. Well then you went to like Michael Doe, right? I mean, that's ostensibly what he's hoping to be able to do. And I think one of the reasons you see people go private is to say, I can't play this game by normal public company protocol. I mean, I'd like to, but I can't get there from here. Now, I actually don't think every company, a lot of have to go private to do this, but I think they do have to change their playbook. All right, Jeff, we have to leave it there. Hey, great to see you. Thanks very much. We feel smarter just hanging out with you. Keep it right there, buddy. We'll be right back after this. This is theCUBE.


So we're here at the Lenaro Connect here in San Francisco, and who are you? Hi, I'm Mark Ogotsky. I'm the director of the Lenaro Digital Home Group. We've had a great week here in San Francisco with all of our members, and just wanted to share with you some of the latest things we're doing in the Lenaro Digital Home Group. There's a lot of activity and a lot of traction and a lot of the things happening at the home group, right? Yes, over the years we've been working on a lot of core technologies, and now we're really seeing them get a lot of traction in both the open source community and in some of the deployments that people are doing using some of our technology. So in the Lenaro Digital Home Group we've been focusing on secure media playback using OPTI with DRM integrations. Some of this stuff here, it says here play ready DRM with OPTI. Yes, so the commercial DRM integrations consist of both play ready and wide vine for Linux and for Android, AOSP. So we're covering all the bases, we have play ready integrations there, and again, wide vine integrations as well. So if you can just check it out, so right now there's DRM support on this one? How is it supported? Alright, so let me introduce Alexander Zhutra, the assignee from NXP. I should mention that NXP is the newest member to join the Lenaro Digital Home Group, so Alexander will describe his demo that he has set up on the high key. Right, you're from NXP? Yes, I'm an assignee for the Lenaro. So there's a whole bunch of stuff NXP is doing with the home group? For example, what do you have here? The demo is not quite set up yet, but I have play ready integrated on Android 8 with OPTI available. So it's running right now on a high key development board with all this stuff going on here. So when you talk about AOSP8, all this DRM stuff, OPTI, LHG is always the first one to get everything to work. Well, in terms of integrating with OPTI, a lot of people notice that in the industry OPTI is gaining more and more traction. So we've always been promoting OPTI as an open source trusted execution environment. And then the additional DRM integrations we do are seen as something of a very high value to our members. And there it mentions AOSP8.0 and right here later I'm going to check it out, something to do with AOSP TV. But what's the latest stuff that's going on with Android TV, maybe? How do you help with that kind of environment? Well, right now we focus on core enablement on AOSP in terms of media features through the TV input framework for Linux TVB libraries. But again, the other component that we bring to AOSP is the OPTI DRM integrations. And our members then are looking forward to doing OPTI integrations on their own Android TV deployments. And there's all this video for Linux too, there's hardware acceleration work. Right, so primarily on the Linux media framework side, in the last six, nine months we've been really focusing on essentially making media run better on ARM. And that is looking at integrations of Gstreamer with V4L2 as well as some work that we've leveraged for FFMPEG for V4L2. And we can see that when we run our media pipelines with V4L2 as well as Wale and Weston, DRM KMS for the graphics, we can see that we really offload all of the computation onto the video codec as well as the GPU. And we see that the CPU utilization is actually very low and we achieve very high optimized performance media playback on Linux with V4L2 driver framework. And for example, there was this conference at IBC recently, you were going over there, there's a lot of set-up box over there going on. And what kind of things are you noticing in the LHG is maybe a couple years old now? Well, no, I just celebrated my fourth year working anniversary here. So yeah, we're into our fourth year. Because just before this conference, I was at IBC in Amsterdam, can you pause it? Yeah, no problem. Done, done, done. Yeah, so the fourth year. Okay, so LHG is in its fourth year right now and I was at IBC and like I said, it's interesting that the technologies we're working on are getting to be more known in the industry. So for the set-up OEMs, the Opti integrations with DRM are very important. Really people are looking at being able to interoperate with multiple DRMs and they're looking for lusting the burden on integrations and Opti provides that. And the other thing, of course, that's really big is ultra high-def and high dynamic range, which incorporates many technologies. So what do you do with the ultra high-def and HDR? You had a presentation about this, right? Yes, I had a presentation on this earlier in the week and really it was to provide an overview of what the technologies are and also what the, you know, how they're currently supported in AOSP as well as Linux before L2. So right now, it's an exciting new technology. We're looking at incredible color reproduction playback on user devices with extended high dynamic range luminance as well as the wide color gametes and really produce some stunning playback on some of the newer monitors. Is HLG becoming kind of like standard? Everybody is supporting that or there's also the Dolby Vision? It looks pretty good, right? I would say there's primarily three HDR systems that have been gaining traction. One of course is HDR10, which is the open standard and it uses something called a perception quantization curve developed by Dolby for what they call an electrical optical transfer function. So HDR10 is something that is being supported by almost universally because it is open source. Dolby Vision is proprietary, but it actually extends the high dynamic range to 12 bits with a 2-bit enhancement layer over HDR10. And Dolby is able to provide a scalable solution where they can actually support SDR standard dynamic range output HDR10 as well as Dolby Vision. They had an impressive demo at the IBC, right? Yes, really what HDR is showing is that the content that is created and mastered is really what you can actually end up seeing on the screen. In the past, there would be so many transfer functions and sort of degradation of the content and then the monitor at the end would only be able to capture a very limited range. Now with the newer monitors that support the UHD Alliance requirements, you can capture essentially pretty close to the mastered content and preserve the artistic intent that was involved when they actually mastered the content. It's quite impressive. The other standard is HLG, which is Hybrid Log Gamma. This is another electro-optical transfer function and this is used primarily with live content. And really all this is doing is extending its staying at the power law that's used for the standard gamma-based transfer functions, but then it extends it at the high end in order to support higher luminance. Did you do some work with that in an hour? We're not working with it right now, but it is one of the supported types in Android AOSP at this time and we expect support for it to be added to V4L2 very soon. So we're looking at UHD HDR technologies as being the next really, really big technology. So it's really quite stunning in terms of the quality of pictures that can be produced with 4K and UHD HDR. When all this on the market is getting these amazing 4K HDR displays for $500 and less and stuff and you need to connect awesome set-up boxes, how about the UI? Is that in 4K too? Does that mean the Nero has even more work to make things smooth? Well, that's where you're looking at issues with GPUs where you want to be able to support typically the video processor, the VPU path is where a lot of the HDR work is done. But then the GPU is responsible for then also supporting some video textures as well as the graphics overlay at 4K resolution. So ultimately what we're looking at are better quality monitors supporting this HDR content and then with fantastic 4K graphics as well. So some of the UIs that we've seen that are being shown at consumer trade shows and maybe in there a few years out before they'll be really commonly available on high-end TVs. These user interfaces and guides are just incredible. It's going to be like they're remarkable. Like entering a new world kind of on your TV, you'll be like, oh, I want to get into this UI. It's so beautiful. Yeah, and the other thing that's interesting is that we're seeing more voice activation. So for example, as you know with Google, Amazon all have their home, like the home assistant in Alexa. We see that the trend will be that people will be, in the next few years, will be talking to either their smart TV or remote to change channels or do whatever. And how is Nero involved in that kind of world? Well, I think at this point what we're looking at is really some of the, we've been talking with some of our members and looking at their initial concern right now is for voice activation to have very low latency. So that when certain commands are uttered by the user that the device or smart TV, the set-top or smart TV can react very quickly to these. So you may be in kind of a low power mode and just listening for some key words. Once you hear those key words, it will wake up and start doing whatever the user wants. So that's really one of the things that some of our members are saying. We want to be able to be listening in low power mode. And as soon as we hear these key words, we can fire up the system. Could it be that it has kind of like a connection with the lights group? Also now all these set-top boxes want to have IoT implemented, right? Everything's starting to converge, of course, in what they call the smart home or connected home. That terminology has been around for a while. But now with these voice activation controls, the interface for the user is much more natural. And so not only will they probably be controlling TV channels and things like that, they can make queries just as you do with your Google Assistant, for example. And they can also, you know, it can easily be extended to like the smart home and control other elements in your house. So there would need to be an involvement in what all the stuff that light group is doing and putting that into the set-top box. Well, I think... Maybe some of the lower power chipsets. It could also be included in the SOC to have that kind of functionality. Yeah, but typically what happens is the set-top boxes in that are usually... They have very high-powered CPUs and GPUs in them. So to add IoT capabilities just for the home, it's usually not...  And can we look just around a little bit more? So there's some stuff over here that has to do with the RDK. Hey, is your demo is working? Sit up. Yeah, we are trying to set up. Still we don't have the networking. Yeah, so this is Shiva who's working as an assignee from Comcast and he'll talk to you about what he's doing with his demo here. So what are you showing here? Yeah. So this is RDK ported on Dragonboard 410C, which is one of the 96 ports. So this uses V4 L2 acceleration and optimized DMA path. So the complete 1080p video playback we are able to do less than a 20% CPU usage, which gives 80% of the CPU to the MSOs to do different loader, different applications. So you have the full hardware acceleration, accelerated video. And what is the A306 GPU? Yeah, that is Adreno 306 GPU. So that is used for graphics acceleration using Mesa, the latest Mesa driver. All right. So there's Android stuff. There's what's called RDK. What kind of platforms are being worked on? So for the RDK, we've been working on the Dragonboard to start doing some prototyping of RDK with video for Linux to driver framework. And as she was mentioning on the Dragonboard 410C using the video decoder as well as the GPU, we can achieve very high performance playback. All right. Should we check a little bit further over there? Yeah. So let's see. This is Kellyanne. He is also an assignment for Comcast. And he's actually done a nice kind of proof of concept work over the last six months. And I'll let him describe what he's done. So what are you showing here? Yeah. So the secure boot implementation we are doing on a high-key platform. This one? This is a high-key board. Yeah. And we implemented the reference solution, secure boot and DRI disaster recovery image on 96 board. So that is the reference implementation. So you have, it says here, a chain of trust for authenticating an OS loader. What is that? Yes. So in the secure boot loader, we are authenticating the Linux kernel and the root of s. So to authenticate those Linux kernel, the key, public key has to be placed in a different partition. For root of s also you have to place the key in a different partition. So the image and the corresponding key will be in a different location. That way we are meeting the chain of trust. All right. Yeah. Cool. Cool. And then there's all this stuff that has to do also with the Android TV world. We were talking about it a little bit before, but there's a lot of things happening there, right? Right. Again, some of the work that we're doing, we're looking to enable technologies that our members can then integrate into their solutions. And some of those solutions, some of our members are working with Android TV as well. So in AOSP, we're trying to provide improved enablement for some of the DBB libraries as well as these security integrations with Opti. And that's something that our members see as valuable and want to incorporate into their final offerings. So there's going to be a lot more stuff to happen in the months to come at the home group, right? Yeah. In the year to come. Again, when I go to shows, now people, instead of me always pitching to them Opti, now they're coming to me and saying, we really want to talk to you about Opti and things like that because they're looking at having that integration and having an open portable trusted execution environment that gives them commonality for all their DRM integration. So when they have to go from one DRM to another or work with another SOC vendor, it helps them in all of their integrations because they've got one element that's in common in a very important area. And yesterday was a very interesting keynote about the project treble, right? How much is that going to happen here with the home group? Well, yeah, that's an effort that's going on within Lenovo and the home group is really kind of a consumer of what is being done there. We look to at one point support things like the Android common kernel. Again, these are provided to us and our SOC vendors now for Android O are now responsible for supporting these vendor test suites and implementing what's been done in trouble. So this is in the kernel space, this is a very big area, but LHG is really using the output from other groups within Lenovo. I can imagine that once it would be awesome if the setup box that gets Android 8 just get automatically updated forever with treble. Yeah, treble was introduced to make a cleaner delineation between the SOC components and one of the goals was to enable security updates to be done much more quickly. So again, there's other kernel folks here in Lenovo who are more deeply involved with this than we are. We're using the output that they provide. And there's even more demos? For now, this is it.


Welcome everybody to San Diego and to the spring 2018 CNI member meeting. We have a great turnout and I'm delighted to see you all here. I hope you're enjoying San Diego, especially those of you from the snowbound states over the last few months. It's a nice change. I have a couple of logistical announcements before we get started. I'm Cliff Lynch by the way. For those of you I haven't met, I'm the director of CNI and I know based on the new member meeting we have quite a few folks who are at CNI for the first time here. I hope you find this participation worthwhile and enjoyable. So a couple of quick logistical things. As you can see from your program all of the meeting rooms that we'll be using for the main meeting are on this floor except for the harbor room which is on the third floor. The floor above us it can be reached either by stairs or by the elevator. We will be having meals on the fourth floor I believe as well. Wireless information is in your packet. We have posted on the message board in front of registration a list of the sessions that we're going to attempt to capture via video. There is one schedule change so far. The next generation repository session which was originally scheduled for Friday in Plaza AB has been moved to the imperial room on today at 5 p.m. So we will also post a notification of that up on the message board. Sorry about that last minute change. I'd like to take a minute to welcome a new member by the way. Middlebury College has joined us and we're delighted to have them with us. And I think with that I don't have any more announcements. I gather travel here has been relatively smooth. We're not having any major blizzards, hurricanes or other things at major air hubs. So that's a nice change of pace as well. So let me move on to the main reason we're here and introduce Dr. Joan Lippincott who will be giving the opening plenary. It's very hard for me in a way to introduce Joan because Joan has been with CNI since the earliest days. She's been our associate director since before I got there. And her contribution to this organization are just incredibly wide, deep and sustained. Over the last decade or so she has been practicing a particularly intriguing, complex, kind of multidisciplinary and generous scholarship that looks at various interplays between space, organization, scholarly practice. She's really a fascinating kind of meld of approaches. And she's gone very deep into some areas. So some of you will know her work on assessment. Some of you will know her recent work on digital scholarship centers. And some of you will be familiar with the deep studies she's made of space and organization and how those play together. Today I think she's going to do something really special though. And that's to open up these very deep verticals of her work to all of us through a synthesis, a sort of a view across all of these interplays. And one that is centered around the experience of the user. So with that introduction, you'd probably much rather hear from her than from me because I certainly can't do her topic the kind of justice she can. Joan, please take the podium.  Welcome, Joan. And I do believe that's the one and only thing I may be better than Cliff asked, the visuals. Thank you, Cliff, for that gracious introduction. I'm truly delighted to speak with you today on some issues that are particularly meaningful to me and my work at CNI. And in my talk I'm going to highlight some of the content on the CNI website and how it reflects the ways that we think about and highlight important trends and projects. But I also want you to know how much I've valued the input of truly all of you in this room and all of the people who have participated in CNI over the years.  And I thank you for that. So today my theme is keeping the focus, keeping the user at the center. The center of what? The center of the mission of our universities and colleges. Faculty and students are at the center of the core mission of universities, research, teaching, and learning. And as information professionals we can sometimes focus too narrowly, I believe, on our work and our unit, our primary unit rather than on the university as a whole, and implement things in an incremental way that really doesn't always advance what we can do in the most meaningful way. I think it's important periodically to step back and think about why we're doing what we're doing. And to really focus on the user community, how we can advance research, how we can enhance teaching and learning, and how we can provide links to our community. In each of these areas, our users, our constituents, our students, faculty, and staff, and our community, however that's conceived in your institution, should be at the center of your thinking as we develop infrastructure and programs, as we allocate resources, and as we make plans for the future. I'm going to try to illustrate this by focusing on three themes today. Discoverability, teaching and learning, and places and spaces. We've made astonishing progress in making content of all types available on the internet, and by we, I mean the research and education community, universities and colleges, museums, historical societies, and similar institutions. In addition, government entities have provided a wealth of information on the internet, and many are providing content in ways that are openly accessible to the global community. Here on the left, we see some illustrations from the Smithsonian Image Gallery, a selection of book illustrations, photos, drawings, and other images from across the Smithsonian Libraries. Most are out of copyright and are available for anyone to use. And on the right, the David Rumsey map collection, which focuses on rare 16th through 21st century maps. They began digitizing these in 1996. The physical map collection is located at Stanford. One of the features I've always liked, and they've had it for a long time, is the so-called ticker that you'll see at the bottom of the photo. It's one means of browsing the collection as the images march across the screen. I wish more libraries would have such innovative ways of looking at content. The evolution of journal publishing, which is one of the first initiatives of CNI and its member organizations, has also preceded a pace, although perhaps not in all the ways we initially envisioned. Initially libraries did not fully recognize that they would cease ownership of many journal titles, which are now rented through licensing agreements. They assumed that the move to journal content on the internet would result in much broader access to content, which has not necessarily taken place unless you count such services as SciHub or academia.edu, and I am not recommending those resources. On the other hand, the directory of open access journals documents the impressive growth of those resources legally available to all. Users of data have also become increasingly important in the digital world, and you'll see on the right data.gov, which makes available data collected by the U.S. government, the federal government, and tools by which users can analyze that data. But if we step back and think about the enormous varieties of digital content out there in so many formats, monographs, journals, images, video, audio, data sets, multimedia objects, 3D visualizations, and on and on, how do we as information professionals make sense of all of this information to our user community when they're looking for something? I know that many in the library information technology publishing and information systems community are working hard on this issue. I searched the CNI website, just our own website, not a broad search, using the term discoverability, and I got 89 hits, that's just from things from mostly from sessions at our meeting. Six topics included discoverability of items in repositories, of media, manuscripts, biomedical data sets, folklore resources, and techniques such as linked data and use of identifiers. I'd like to illustrate more concretely some of my concerns regarding discoverability and keeping the user at the center. So here's an example. A faculty member is preparing a grant proposal to NSF and needs a quick literature review. She wants to just quickly read relevant articles and has little time. She uses Google Scholar and finds 10 articles of high relevance and she needs them all. So does she use the library website to identify which she can gain access to and where she can find an openly available preprint if the library doesn't license it? Does she use academia.u or Sci Hub? What is the most efficient process? What pathways does the library provide that will most easily identify how to get access without fee to items? Does the library know if a preprint will fulfill the researcher's need? And if so, is there a clear cut way to find that preprint? These are problems that we know about and have studied. Roger Schoenfeld from Ithaca S&R published a report meeting researchers where they start streamlining access to scholarly resources, which literally left my head spinning in confirmation that users are not at the center of most of our discovery systems. At our CNI meetings, we've had briefings comparing preprints and published article texts and we'll have a panel at this meeting discussing what links the library should provide to versions of content. But at present, there is no standard procedure, no common philosophy, no consensus on best practice among information professionals. How do we expect our users to make sense of this environment? And many of our users switch institutions where the practices may vary. Here's another example. A student who, and by the way, I just made these up. They're not from anything in particular. A student who graduated last year works in a nonprofit that serves the homeless and wants to read some studies on success rates of different types of education or training programs for the homeless. Many of the college and university library run information literacy programs still focus on what I call the find three articles in a database process. We maintain that we want to instill lifelong information literacy skills in our students. But will the strategy of teaching students how to find licensed articles in licensed databases truly serve our students when they enter their next phase, their careers, where many are unlikely to have access to licensed resources? At times, the philosophy of, well, we paid for it, so that's what we need to teach students to show these resources are being widely used, seems to go against the longer-term interests in teaching students about the nature of the information environment in their field and the economic factors that govern access to information. Fortunately, some organizations such as JSTOR have made provisions for those without direct ties to an institution to get access to publications under their control. We've had briefings on this program at CNI, and at this meeting, we'll have a session on the needs of independent scholars. Cliff Lynch has written about this issue, and Allison Head touched on it in her plenary session at last spring's meeting. But the bottom line is that this leads me to ask, is our standard mode of teaching students about access to journal articles truly keeping users at the center of our thinking? Even when librarians prepare their ubiquitous lib guides, many prepare what I'd call laundry lists of every conceivable resource in a broad field rather than something that would more closely model a reference interview. By the way, I prepared those well before it was called, they had the product lib guides, so I'm criticizing myself as well. Years ago, I admired a style developed by University of Pennsylvania in their business library which led users online through a brief set of questions about their topic, and this resulted in a short, often three-title list of where to begin and links to those sources. This may be easier to do in a professional field where there are more defined information topics, but I think we need to have more user-oriented ways of presenting information and guiding people to resources. We also know that searching library catalogs and selecting databases can be confusing and frustrating for many library users, particularly undergraduates. But how many libraries have come up with really creative solutions to address this problem? University of Texas San Antonio described their online chat program in a spring 2016 session at CNI, and I quote, at a time when many libraries continue to experience low-reference activity, the University of Texas at San Antonio libraries have seen a 489% increase, yes, 489% increase in two years from 4,600 questions, this is just by their chat reference service, to 27,000. After implementing a proactive, context-sensitive chat system, that was created for online businesses. The complexity of these online transactions, they said, is also notably higher than the complexity of questions received at the physical information desk. Here's an example of a student's question. I'm researching corporate power in America with a focus on the political point of view. Which database would be best to use? They also found that faculty were asking questions, which happens very infrequently at physical reference tests these days. Here's an example of a faculty question. I'm looking for respondent-level data sets for my students to analyze using SPSS. What do you recommend? These are the kinds of interactions that reference and liaison librarians wish they would have more regularly with their constituencies. Are we truly thinking of our user communities and the variations within those communities as we develop the information ecosystem, implement discovery tools, and teach our constituents about them? For me, a former reference instruction librarian, there is a modicum of incoherence of access mechanisms to all types of content and a paucity of real reflection about how we should conceptualize and teach users about how to find scholarly information resources broadly conceived. Connecting our user community with information is a key way that we support the research teaching and learning mission of the university. Discoverability applies to more than finding content. Our user community is full of content creators who need resources as well. Discoverability of things like programs, expertise, and technologies on campus also demonstrates a lack of clear guidance for users. A lack of coherence was an ongoing theme in the work that we did in preparing the paper with ECAR called Building Capacity for Digital Humanities. In this case, availability of and access to networks, storage, specialized software, equipment, digital content, and professional expertise is often administered by various units, many units, which may include central IT, departmental or college IT, libraries, and faculty institutes, such as humanity centers not affiliated with libraries. Often there is no institutional governance structure that assists in determining who has access to what resources, what will be provided for fee or for free, how priorities are developed for support of particular projects, teams, or individuals, who's responsible for the curation of the content created by the projects, and who will fund facilities, equipment, and staff for these various purposes. Just like library content systems, digital scholarship infrastructure, and support have often developed piece meal on campus. In their case, it's often when outside funding or targeted internal funding is available, or when particular individuals, colleges, or departments, or research groups put resources into developing programs, which may or may not be sustainable. This is understandable in the relatively short history of digital scholarship. However, I believe that we're at a point where stepping back and taking a more critical and programmatic look at where we're going and what we're doing in this area is needed in most institutions. Some have implemented robust needs assessments. Yale University and University of Pittsburgh were highlighted for their work during the Brown University Digital Scholarship Planning workshop, which we co-sponsored there last September. And at this meeting, University of Houston will be presenting a session, and they did an extensive needs assessment for the development of their digital scholarship program. Many people have a visceral dislike of assessment, but doing some kinds of studies are, in my view, the only way to determine whether your programs are achieving their goals and whether they're genuinely useful to your communities. One model that's gaining some fans and some visibility is the Europeana Impact Playbook, developed to assist all types of cultural heritage institutions in determining the value of their programs. They state that their intent is to collaboratively define the relationship between the things that you do and the impact you expect to achieve. What activities are you deploying? Who do these benefit? How does this make a difference? And how do you measure that? This sounds so simple and so obvious and reflects principles stated by many others. I'd note the work that ACRL is doing in this regard. However, in my experience, few institutions are making serious attempts to carry out meaningful assessments of their programs. The second area I'd like to address is teaching and learning. As information professionals, librarians, information technologists, instructional technologists, media professionals, we need to make conscious choices on where our resources go to support teaching and learning. I believe, first and foremost, we need to focus on the educational mission of our institution and what we want students to gain from their education and not on systems. For me, the sweet spot for supporting students and their ability to move from college into fulfilling professional work is for librarians and information technologists to focus on integrating 21st century skills and understandings with a particular emphasis on creativity into students' course assignments, capstone projects, and theses in coordination with departmental faculty. Focusing on how information professionals can integrate important concepts about information into the curriculum was a large part of the thinking of the working group that developed the ACRL framework for information literacy, of which I was a part. In libraries at present, I believe, a very large percentage of professional library in time is spent on teaching basic information literacy skills to freshman English or equivalent classes. Information technology units in a number of institutions are spending significant sums of money, along with a great deal of staff time, on developing learning analytics systems generally, but not always for introductory large enrollment STEM classes, particularly those that have had a poor pass rate in the past, which in turn affects students' persistence at the university. While I do understand the importance of those kinds of programs, I would note that they are not particularly focused on developing 21st century skills or are encouraging creativity among students. Administrators, faculty, and information professionals need to make judgments as to where the emphasis should be in the services that we offer supporting teaching and learning. I believe we have a lot to bring to the conversation in terms of highlighting the importance of developing 21st century skills and emphasizing assignments that develop our students' creativity. There are a number of resources that provide guidance on learning goals, one that I find particularly useful is called the Degree Qualifications Profile, or DQP, developed by the National Institution for Learning Outcomes Assessment, or NILOA, along with Illumina Foundation and others. Having the language to adapt and adopt in local institutions can be a useful way to begin conversations on what outcomes we want students to achieve in their learning. Here are two examples of what the DQP refers to as proficiencies. One, construct sustained coherent arguments in writing, okay, nothing new there, but then they add, and at least one other medium. That's the significant piece. The second, negotiates a strategy for group research or performance and communicates the results, and we know how important team or group work is for 21st century workers.  The student used the Esri StoryMap GIS software and data from the China Internet Network Information Center to create a visualization of the relationship between purchasing power per capita of each province in China to internet penetration. This is a student who was developing an academic argument in a different medium that included texts but included many other things and helped develop his 21st century skills. I've long admired the program developed at the Jones Media Center at Dartmouth, where information professionals work with faculty on developing video projects for class assignments. I highly recommend their website, which gives a wealth of information on the wide range of classes that they work with and also includes testimonials from faculty and students. Here's what Professor Deborah Brooks of the government department had to say, I'm quoting. Using video projects within a competitive campaign simulation was an invaluable component of my political communication seminar. Beyond the study of campaign advertisements in scholarly articles and books, my students also created their own political ads. Through the process, they learned just how much information can be included in 30-second ads. They learned that information in this context goes well beyond words and includes potentially powerful music, sound effects, voiceovers, text, colors, photos, and video selections. As a result of this exercise, we were able to debate the roles of emotion, information, partisanship, strategy, the increasing professionalization of campaigns, ethics, and voter knowledge within a democracy in a far more dynamic manner than would have been possible otherwise. She continued, few of the students in the class had ever worked with video equipment or editing prior to this class. Remember, this is at Dartmouth, and all emerged from the course with near professional level work by the end. The quality of their assignments was truly impressive and was only possible with the training and support provided by Jones Media and RWIT, which is a combined program of the library, the writing center, and the academic computing at Dartmouth. Recently, EDUCAUSE Review had an article on a framework called My Ways on creative competencies for student success in a world of change. One of their four domains is creative know-how, and you can see how the points in that area are so relevant to what we as information professionals do and can contribute to our students' education. At the Virginia Tech Library, they've been incrementally renovating spaces to develop a network of studios, which includes a 3D design studio, fusion studio, media design, 3D visualization, virtual environment, and data transformation. Patrick Tomlin, the director of learning environments for the library said, the studios exist because we believe that creativity is as critical a skill as literacy. We also believe that well-designed learning spaces can be tools to shape meaningful interactions, make innovation tangible, and spark community. Teaching and learning often involves students working in physical spaces, whether classroom, library, or other. As I transition to my final topic, Spaces and Places, I'd like to discuss what I see as a trend needing both examination and policy development. Cameras have been used to study utilization of space for a long time. In library settings, this usually requires IRB approval and posted signs alerting students or others to the presence of the cameras. Often the data collection is of a limited time and limited space of libraries so students don't wanna be recorded, they'll go somewhere to a different area. However, as more and more people believe that use of analytics in educational settings will provide the answers for achieving student success, we may see the development of more intrusive systems. Unless you think I'm being an alarmist, I'm now going to move to a quote from a Stanford professor who discussed his research in an interview in Inside Higher Ed recently. I'll bet a number of you saw this. He said, and I quote, "'I have very little doubt that tracking data from VR systems, how students move their heads, hands, and bodies during learning can be used to predict retention of course materials and likely learning transfer when that material from class gets applied to the world. Indeed, my lab has provided preliminary data showing this capability. However, the privacy and policy implications of this are clearly vast and there need to be tough conversations before a system that can categorize learners based on their body movements is implemented." He continued, "'Imagine after 10 hours of body tracking, machine learning algorithms could predict with high accuracy what your final grade of the class would be. Based on thousands of students who came before you who'd contributed training data based on both body language during the instruction as input and final grade as output. Would you, as a student, want to know that result after the first month of class? Would you want your professor to know?' And he also said, "'As far as I know," and I talk to policy makers about VR fairly often, there aren't many conversations ongoing. I was recently in a conversation with some architects and a university professor who's interested in monitoring students in learning spaces. By monitoring, she meant having students using wearable technology that would track their physical reactions that in turn had implications for things like psychological stress, things like measuring sweating of students. This, and this was purely conjectural, she's not actually conducting the research at this point, would be relayed the data, would be relayed to the faculty member who could monitor students enrolled during the class and monitor their emotional health. I asked her if she had given any thought as to what a faculty member might be expected to do. If he or she perceived that the student was experiencing stress, was the faculty member going to be trained in counseling or make sure that the student made an appointment at the campus counseling center. I also asked whether she thought that students would be comfortable having a faculty member know in real time what I would consider intimate details about their psychological state of being. She did not have answers and I couldn't tell whether she'd even contemplated these issues and that really surprised me. Fortunately, there's an excellent resources for campuses to use when developing policy related to student data and privacy issues. IMS Global has developed the data and analytics principles. We had a session on this at our last spring meeting and there's a video of that session and you can also access the full document on the IMS Global website. I urge you to consider going back to your home institutions and scheduling a conversation about this important work either with your colleagues or as a university-wide conversation. So my final topic, are spaces still important? Why? The answer is yes. Why are physical space is still important in a technology-driven world? Students still need spaces where they can work together. They need access to specialized technologies and sometimes large screens and they need access to expertise. At least for the near term, spaces will continue to play a key role in teaching and learning and to some degree in research. Here we see a portion of the Digital Scholarship Center, the curve at Georgia State University and the Digital Toolshed, an incubator for innovative digital research, teaching and learning at the Claremont Colleges Library. As information professionals plan new spaces or think through a space renovation, I believe that it's very important for planners to focus on what difference the spaces will make in learning and in research rather than on the comfort, style of furniture, et cetera. After establishing the goals for the physical space, I'd encourage you to think about those principles from Europeana that I mentioned earlier. What activities are you deploying? Who do these benefit? How does this make a difference and how do you measure that? The Learning Space Toolkit developed by North Carolina State University and others offers guidance on planning and assessment and I served on their advisory committee. If we focus on what people will be able to do in our new and renovated spaces, it will make much more sense to our community, especially our faculty, than showing them photos of the nice furniture we plan to purchase. Smith College is gutting its library, renovating it and planning an addition and like most places, they've had some pushback from faculty, from some faculty. In preparation for the new facility and as a means to develop some sense of what kind of programming they might want to provide in the renovated facility, the library held a competition for many grants in a space that they called and to host those experiments by students and faculty in a space they called the Knowledge Lab. One successful grant was Visualizing Russian Feminism. In an article describing the program, Brendan O'Connell, a Smith Librarian and disclosure, my son-in-law. Wrote, quote, the exhibit which ran for a week in the Knowledge Lab featured propaganda posters and other printed images, video clips, writing and a slideshow of Russian feminist photography, showcasing artistic expressions of female identity in Russia from the early revolutionary days of the Bolsheviks to the 21st century. The students also served authentic Russian fruit from a local restaurant to guests at a lively opening event. He continued, the event served as a powerful reminder of the value of making ideas about the new library concrete and real for our campus community. A faculty member who previously had not been enthusiastic about the library renovation project approached me during the exhibit opening and told me that seeing the students display their scholarship in the library was the first time she had been convinced that the library renovation was a good idea. I would also note how this project brought together 21st century skills and creativity. Places for digital scholarship are also important. At Brown they initially thought that their first space, which is not shown in this photo, would be used by faculty members for the research, but they soon found that faculty were bringing their students either as part of their team or their classes to use the products of their scholarship. So they developed a second space so they had more room for workshops and also so they had a space where students could continue their project work outside of class time. At the University of Calgary, an in-depth user study of faculty involved in interdisciplinary work, and I consulted on that work and Tom Hickerson has spoken about it at prior CNI meetings and at the designing libraries conferences, led to the renovation of space to form lab next, which extends the availability of high-end technologies and collaborative spaces in the Taylor Family Digital Library. You can visit that space if you come to the Designing Libraries Conference in this September. There's information in your packet about that. It's important to understand that beyond technology and direct relationship to specific courses, libraries can play important functions for our students in providing a sense that they belong to a community, hosting events such as this forum on fake news at the University of Rochester Library in a pop-up space in the renovated Lamb Square is an example. The library is a place for exchange of ideas and digging into information. Another part of developing community is to understand the variety of needs of different subgroups of the student population and understanding that some social or affective factors could have important ramifications for student success. Recently, University of Toronto's Robarts Library opened Canada's first academic library, Family Study Space. I know some U.S. university libraries have had such spaces for some time. Providing break or lunch areas in digital scholarship center labs or other specialized spaces like graduate areas and providing library cafes are mechanisms for developing communities where users can share information while engaging in social interaction. And last but not least, library spaces can continue to provide inspiration, whether a modernized, traditional James Mullen's reading room, a surprisingly popular spot in the otherwise very modern and technology-intensive Will Smith Learning Center at Purdue University or representations of manuscripts as murals in libraries at Connecticut College and University of Texas or the art, digital and analog, that's an integral part of the Hunt Library at North Carolina State University. All of these mechanisms help to keep the user at the center of the place and space. So moving forward, I encourage all of you to continue to share what you're doing to create innovative programs and spaces that keep the user at the center, at your own institution. Let me know what you're doing. Put in proposals for sessions at the CNI meeting. Invite me to your campus to see what innovations you're preparing that will help make our information systems more coherent and understandable. Tell me about how you're working with faculty on integrating 21st century skills and creativity into the curriculum and show me some of the wonderful spaces that you're creating that encourage innovation, imagination and community. Thank you. So the question is, how do we resolve maybe a tension between wanting users to do creative things and try new technologies and et cetera, versus them maybe just wanting to be comfortable? Does that encapsulate it? Okay. So in most, fortunately, in most university or college libraries, there is room for both. And I never want to minimize the role of solo quiet study space in libraries because it's really the only building on campus where most students have access to that. It's just not the center of what I'm focused on, but I still believe that it's very important. That isn't quite what you asked. So let me then focus on the creativity and the innovation and use of technology. I'm really a firm believer that information professionals in general can't broadly speaking make that happen for students. I am a big believer in this happening through assignments in their coursework. And that's why I think it's so important to work closely with faculty. Students just, now maker spaces may be the exception. We have many student clubs and other activities while the session at this meeting with at least one maker space has a really active, actually two, I think both of the, that I visited at Berkeley and at San Diego State, they have really strong student cohorts that use the space in an extracurricular manner as well as some use related to the curriculum. So that would be an exception, but I would say that working with faculty to help them understand the importance of 21st century skills and then working with them, and that's why I think this Jones Media Center, it's not the only one that does things like this, but they document it on their website. They'll give you a whole rubric for how they, and a whole development plan of how they work with faculty to create the assignments and then how they work with faculty to assess the student products. So that's what I would suggest. Hope that answers your question. Yes. In your talk when you have the three areas of discoverability, teaching and learning and the places and spaces, it represents a transition away from collections as the center of what libraries do, and you talked about collections kind of indirectly in terms of open access and things like that, but I think it does represent a shift in thinking away from a collection-centric model of university libraries, and the teaching and learning and the discoverability don't assume that these are library collections necessarily, so I wonder if you can talk just more openly and directly about that shift in your brain. Let me see if I can represent that well. I believe libraries need to focus like a level higher on the research, teaching and learning and community mission of the university, and not as a we build the collection because we're the library. So to me, the collection supports research, teaching and learning and in some cases community, and so the shift in focus, it is definitely there, and what I'm saying is that actually even in pre-technology days, in my view, it should have been there at that time. Now, there are a few libraries in every country like Library of Congress, maybe Harvard and Yale and some others that I do believe the collection, this is a national cultural resource, but I personally believe that's a limited number who build the collection just for the collection's sake. I think it needs to be built for the research, teaching and learning mission of the university. Now, in addition, just how that Dartmouth professor says, the students need images, sounds, videos, all of these things, and I find it very concerning how librarians focus so much on what they own and license. I don't think that's a positive thing. I think it is limiting the thinking of students or they may just go around and think, well, you don't really know, you don't really understand what I need. And also working with classes where the primary function is to find the three journal articles takes time away from the classes, which I think is such a routine thing that it could be done in different ways, takes time away from these more creative assignments that may take more analysis and more discovery of all kinds of information resources. So you may not agree with me and I'm happy to have you push back, but I think that may be my answer to your question. I actually totally agree and I'm really glad to hear it articulated that clearly, so thank you. Thanks. Yes, please. Hi, Jennifer Venapal from The Ohio State University. So this is a question about trust in reaction to the stories that you told about the faculty who were thinking about collecting personal, very personal data from their students. And as a parent of a 17-year-old who is extremely skeptical about institutionalized learning to such an extent that she's considering not going to college right now, I'm just wondering if you talk to students or have a sense from the faculty that you've been speaking to or the librarians, we consider ourselves trusted resources. And to what extent it's your sense that students are horrified by or pushing back against this kind of tracking and what that can do to our reputations as trusted places for sharing knowledge? Well, I think that's a great question and my answer might surprise you. From my reading of the use of learning analytics in some institutions, particularly for the large enrollment STEM classes, my impression is that not only do students not particularly push back on it, but some of them are quite grateful. And they get things that some systems call nudges.  And honestly, I find that kind of flabbergasted at the beginning. But part of my thinking has changed in that since we have such a broader population of students, we have many students, because a lot of these are for first year courses, a lot of students who haven't had the highly educated parents who kind of put them through their paces in high school. And I gather some as helicopter parents of students in college who are in fact keeping students on track. Now, myself and as a parent, my daughters are much older than yours at this point. I felt after high school, that was it, that my role in nudging was over. And I really, my personal philosophy as part of going to college is independence, learning to think for yourself, learning to budget your time and making some mistakes is part of learning those life skills. But in some cases, in some student populations, and I wanna be clear, I'm not saying those students aren't as good or as, you know, it's more that they haven't had the same life experience and they may need more upfront. And so my guess is that that's why this is okay. What I think is important is what's gonna happen, I'm not as clear whether students always have a chance to opt out. And I have people talk about learning analytics that well, if we don't get all the data, it's worthless. I've heard comments like that, which surprises me given methodology classes I've taken. I mean, who gets 100% survey result, for example, I haven't delved into the machine learning. I know nothing about the actual mechanics of creating algorithms and things like that. That's cliff's area, not mine. And so I don't know, but I think that is very important. But then comes the question, can the student really opt out? For example, if they need that course and it's only offered that way and there is no provision or they don't feel they can do it, do they really have that opportunity? And I think we need to be watchful of that. Does that help? Is that enough of an answer? Thank you. Okay, I think that we're probably ready to go to our next sessions. I think Cliff is going to come up and give us some final. He's going to come up and say thank you very much. What a wonderful survey and synthesis. Thanks. And thank you also for just highlighting how much of a resource that website that we've got now represents for insights into this kind of thing. And so much of that, I've got to say is really you're doing in helping to track the right sessions over the years and highlight them. So thanks again for a tremendous talk and I hope you all have a good conference. I'll see you back tomorrow for the close. Thanks.