Welcome everyone, let me welcome you all to the spring 2016 CNI membership meeting and what looks like a really nice day in San Antonio, which we arranged special for you all. I am delighted you're all here. I hope that your travels have been relatively painless. I'd like to make a special welcome to a couple of folks. We have with us the new cadre of clear fellows at this meeting and could you all just kind of wave your hands in the air? Yes, we have quite a group of them over there in the back anyway. You'll have an opportunity I'm sure to meet them at sessions and at the reception tonight and we're delighted that they're here with us. I'd also like to extend a particular welcome to our foreign visitors. Getting here from abroad can be quite an adventure and we are very very pleased that you're here with us. I'd like to welcome four new members who have joined us mid-program year and I think at least three of them are actually physically with us at this meeting. Those four new members are Skidmore College, the Crest Foundation, index data, and Swarthmore College. Welcome new members. Changes in the program happen and I want to direct your attention to the message board out by the registration desk. Thus far while we have had a couple of changes of speaker which I won't go through with you, the only actual session change that I'm aware of is that in the 515 slot today we've had to cancel the breakout on the open library of the humanities. Martin Eve unfortunately is not well and can't be with us but we're hoping we can reschedule that soon for a future meeting. I just you know sort of urge you to keep an eye on that board for changes as we go along. Now with those bits of news out of the way I'll just note that there are instructions for how to get on the Wi-Fi in your registration packet. If you have trouble with that just let one of us know and we'll try and figure out what's going on. There also are some announcements of upcoming meetings and I may I'll say a word or two more about those when we gather tomorrow afternoon for the closing session. So with that stuff out of the way I get to do something really good. I get to introduce Victoria Stodden. I am so pleased to have her here doing the opening plenary today. We go back a while now. I think we probably first met at AAAS a few years back but I wouldn't absolutely swear to that. I think we've been at a number of the same events dealing with various aspects of cyber infrastructure, the future of scholarly communication and things like that. Victoria has been a pioneer in really trying to think through issues about the replicability of scholarship in the digital age which is a tremendously important topic. It really not only is an important topic to the quality of scholarship itself but also I think to the public support of and the public honoring of scholarly work. The public sinks a lot of money into underwriting research of various kinds and I think replicability of research results and understanding what that means is an integral part to having confidence in the record and the outcomes of research and to earning and maintaining public support for the research enterprise. So the stakes here are very high. Victoria I think brings a really powerful set of insights into how the great shift towards computational technologies and data intensive technologies and network technologies are really changing the entire nature of the scholarly enterprise and she's going to share some of that thinking with us and also help us to think about the implications of this for the communication and the documentation of scholarly work and for the scholarly record itself going forward into the digital age. So please join me in welcoming Victoria Stottin. So it's such a pleasure to be here and I was so delighted to get the invitation to come and speak and address you all and I feel like this is a perfect audience for the types of things that I love to talk about. So one of the things I've done a couple of things to help facilitate our discussion so my slides are on my website if you prefer to follow along on your iPad or click on some of the links in the slides and going to try very hard to leave time at the end for many questions and I'll be around actually for the entire meeting so also if you don't get a chance to ask your question please feel free to just grab me or talk to me. Defining the scholarly record for computational research as Clifford very generously mentioned I have spent the last few years of my career trying to think about what does it mean to do research when you have a computer involved in the research that is something that I'm loosely calling sort of high integrity research. So one of the things I'm going to try to do in my talk this afternoon is really build up what that means or how we might conceptualize something like that when we're using computers to drive our results building all sorts of software around these scientific findings working with massive data sets that are complicated and coming at us at high velocity and what does that mean for things like transparency in the research process, second-order verification of the results if you are a reader or a user of these these outputs of research and how do we start thinking about this because it's all really kind of very new about how we share or disseminate research. Okay so I was going to touch on three aspects in my talk today the first two I'm going to go through fairly quickly but they set the stage for the rest of the talk so the first one is really trying to pull apart and understand what are these technological changes that are impacting say for example our conception of what the scholarly record could be if we can tease those apart we can start to understand how they're interacting with the research and the reuse and how we can actually start to think about what each of these changes might actually demand from us. Then I wanted to touch on well if we're thinking about what we should be doing in some sense I've introduced a normative dimension to our discussion. So in research we have long-standing norms so I wanted to talk a little bit about those norms and whether we can lean on them for strength thinking about where we should be going with all these changes. Then the bulk of my talk will actually be the impact on the scholarly record and what I'll do is take these different threads in this framework that we've been building up and to see if we can start to understand different ways that we might be reconceptualizing or thinking through what the scholarly record might look like in this age of digital, deeply computational, high impact, high velocity data research. Okay conceptualizing the technological change so I think there are several ways to break down how technology is really causing these really sort of turning how we do research on its head. So the first one I think everybody knows we've got enormous amounts of data. It's not just more data of the same where we need to scale the same that we've always been doing. It's totally different types of data so we're doing all sorts of different types of research. We have a very high dimensional data which is new and streaming data and these very different types of data so it's not really just a question of size it's all a question of character of these different types of data sets. The second one probably people in the room have thought about this one deeply as well. We also have much faster computers so what this has done separate to the increase in the size and availability of data the very fast computers have allowed us to do different types of scientific research so for example we're asking questions of say a physical system where what we want to do is take that computer simulate the physical system change the parameters rerun it change the parameters again rerun it and ask questions about our world using that computational technology. It may or may not have data involved and then the third aspect that I think is causing very deep changes in the research and in the dissemination that I hear almost nobody talking about is the role of software so my background is as a statistician and if you think about the methods that are applied to these data when you're doing the research all sorts of munging of the data sets preparation of the data sets you need to understand where there are outliers what's good data bad data to even get it ready for the analysis then when you apply the analysis in the models and do at the actual research in the inference all of this is being done in software and it's not in the publication so then the question becomes well we're actually making these contributions to research to discovery and how we actually have how we do inference in this digital research world that are appearing only in the software so how do we actually have that as part of transparency or reproducibility and how do we communicate that aspect to others in the research community it used to be that we could actually include this in the method section now the scale of the work that's actually happening on the computational side and the production of some of software around particular scientific results it's it's not possible to include a complete description in the method section so I have a little photo here this is a screen grab of Leor Pachter who is a professor of mathematics and biology at Berkeley and he's giving a keynote in 2013 and in the course of the keynote he happens to mention that the software that's being generated say in biological research he goes the software contains ideas that enable biology so we all have an understanding that the scholarly record of course contains ideas that enable biology or whatever your field of interest is however now it's actually appearing in the software as well so what do we do about that okay another couple of comments about technology so of course everybody's aware of the changes in communication due to the internet or due to network technology and it's not just a question of digitization but also of access and I won't go into the myriad examples that are well known the last thing I'll mention about changes in technology is the role of intellectual property law and this is something where traditionally in research we haven't thought that much about it even when I first became a grad student you could still find some professors who would you know write a letter to other professors asking for a preprint and the preprints would get mailed around I'm sure people remember those days and and we didn't think much about what that meant in terms of intellectual property now making things available on the web intellectual property is everywhere so one of the things that travels with this discussion is a subtext all the way through on all these components is well what intellectual property rights adhere how can I actually make that work accessible reusable something people can use say to reproduce the work and extend it and those are big barriers so I'll get to that at the end of the talk a little bit more okay now grounding these changes in scientific norms so hopefully you feel very motivated there are lots of deep changes happening in the way that research is carried out and then the the opportunities that we have for dissemination of that research but if we think about what researchers are trying to do what mindset they have what the end goal actually is then we end up in this discussion of norms so one of the things that I like to do in framing a discussion around reproducibility is been the concept into three rough groups there's overlap and so on and all of them but I feel like parsing out these different aspects of reproducibility is very helpful for framing the discussion so one of the things that when people talk about reproducibility in research they mean empirical reproducibility so the way I think about this is the type of reproducibility we've had since the 1660s when we started to communicate results and findings with this notion that you should be able to read the paper and independently you don't have to contact the original researchers independently reproduce and verify those results so empirical reproducibility would be things like at the bench in a biology lab actually doing things in physical space and so this that's our traditional notion and there are many interesting questions around that notion and and and it has actually garnered a lot of attention in the last couple of years however nothing's actually changed within that discussion those all those technological affordances that I just mentioned they're not impacting empirical reproducibility they're things like can I get the reagents that you used in your experiment can I access your stem lines and so on okay so contrast this with the notion of statistical reproducibility so doing inference have I done things like design the experiment in such a way that I would expect the experiment to give the same results in it when it's replicated in a different setting for example and so I won't go much into statistical reproducibility here mostly in the interests of time but there are many mistakes and errors that can be made just in the statistical aspects of the research that would cause the research not to replicate so if you have your ducks in a row statistically then I think you could satisfy statistical reproducibility what I will be spending time on today is this idea of computational reproducibility so there the idea would be can I regenerate your results say for example if you have digital raw data you have as I mentioned you may have done some pre-processing you may have some analysis steps and software that you've applied to the data and then you've published some findings so the question would be well can I actually replicate the computational aspects of your work can I get the figures that are in your paper or the tables or the output so probably most of you are thinking right now well okay so maybe if I can regenerate results using the same software the same data I haven't really you know said anything about the correctness of those results are they scientifically valid maybe I can get the same wrong tables and the same wrong figures in your paper and that's all true however if the way that I think about it is that's in a sense a baseline level of what we should be shooting for and computational reproducibility at least I can get through and check that the machine ran the stuff the right way on your data and got the results in the publication after that I can do things like independently try to re-implement your methods and then see if I can get some similar results if I can't rerun your your code on your data and actually regenerate your findings how am I going to understand why our results differ if I do an independent replication and they will definitely differ so we in a sense we need both types of research but I do need that deep level of transparency in being able to inspect all those computational steps that you went through to get to your final published results and of course right now there are there are researchers that are taking this on and and making it a priority in their work but generally speaking you just can't do that from that traditional publication because the pretty traditional publication was of course defined for empirical reproducibility so the question in my talk well what do we need for computational reproducibility what does that publication look like okay so empirical reproducibility I won't go too much into this I feel like I explained it pretty well but a couple of examples here about how labs biology labs tried to reconcile results around empirical reproducibility and the the point here is it's not easy either so it's not a question of any of it being easy or difficult the computational aspects are really the the new aspects that I've been focusing on okay so statistical and computational okay so I wanted to peel back the covers on computational reproducibility a little bit so we're leaning on a development of norms around computational reproducibility to try and understand what we what would be appropriate in this context and traditionally we've thought of scientific research is having two branches of the scientific method so we thought about the deductive branch of scientific reasoning encompassing formal logic or mathematics all this type of deductive reasoning and then we also have the inductive branch of the scientific method so for example statistical analysis of controlled experiments you're certainly not applying deductive logic in that context there's an enormous amount of chatter around how how technology as I've described it at the beginning of the talk is creating new branches of the scientific method so you've probably all heard this the third branch of the scientific method around simulation and around intensive computation fourth branch of the scientific method around data-driven discovery and big data I put a provocative question mark there on branch three and branch four and so my argument that I'm submitting to you is that computation presents only a potential third or fourth branch of the scientific method so why do we even have a scientific method can't we just you know we kind of have an idea of what science is or what research is and what we need to do why do we really need a formalized method around this and the presupposition that led us to the idea of a scientific method is that everything we're doing in research and in discovery we're in new worlds we're trying to generate new information that explains more about our world it's fraught with error I mean we're just people trying to make sense of things we're going to make mistakes all the time we do make mistakes all the time we never get to a notion of certainty in in research we just kind of feel like we have a better grasp on things or things are more likely to be right or we have not a particularly good grasp on things and we're always trying to move to that to that feeling of greater certainty around our understanding of the world so in the deductive branch that branch around mathematics and formal logic the way that this hunt to root out error has manifest itself is in the notion of the proof right you wouldn't publish a mathematical finding without telling the community how you got there through that formal notion of the proof similarly in the empirical branch they have an analogous notion where we have an entire machinery of hypothesis testing a way that you would set up this problem you apply appropriate statistical methods and then you have a very structured way that you communicate those results through that method section so I challenge anyone here to publish results and just leave the methods section blank won't be published and it's very very important for that transparency in that communication of how you actually got to that results those results so what I think we need is the development of comparable standards standards similar to what we have for the first and second branch of the scientific method but for the computational research the third branch fourth branch of the scientific method so what's our idea of the proof what do we need to communicate so that other people in the community feel satisfied that we rooted out the error the best that we could so there are notions around this so many people doing researchers are have been bugged by this and have thought about it I'm certainly not the only one who's interested in this issue and there's a notion of really reproducible research that was developed in 1992 or promulgated in 1992 by now emeritus Stanford professor John Clare about and David Donahoe who did the quote in the middle it was my thesis advisor and he was paraphrasing Clare about as follows so he says the idea of really reproducible research is that an article in about computational science in a scientific publication is not the scholarship itself so that PDF is not characterized as the scholarship it's merely advertising of the scholarship the actual scholarship is the complete set of instructions and data which generated the figures and tables and the results in the paper and as I've mentioned I think I've touched on this already this difference in sort of running through the same code on the same data and regenerating the results versus an independent implementation and so what what Donahoe and Clare about are saying here is we need to be able to get that the software steps the data that were used that underlie that that scientific result that's being given to the community through the scholarly record and we need to be able to deliver those along with that that result okay so there they're all of these changes happening in the research context and as we know research is governed by these norms and so thinking about what norms could actually guide our responses or how we actually instantiate that delivery of these digital objects as part of the scholarly record so there's many different ways you could lean on different norms I just I looked at Merton scientific norms from 1942 and I'm presenting them somewhat uncritically here and I should mention that there is quite a bit of criticism of these norms and development and so on but the germ of these norms I think is correct and also useful for these discussions that we're having today so Merton postulated five norms Communalism which I think he actually called Communism but its sense evolved into Communalism scientific results are the common property of the community I might even push on that and say they're the common property of the public universe universalism all scientists can contribute to science regardless of race nationality culture or gender disinterestedness as researchers were acting for the benefit of the scientific enterprise rather than for our own personal gain originality so making a contribution to the community research contribution must be original or adding something new to the discussion and skepticism scientific claims must be exposed to critical scrutiny before being accepted so the two that I focus on the most in my research or I found most useful is the idea of Communalism being able to share the results in such a way that they're broadly available and skepticism exposing these claims to critical scrutiny okay so skepticism I can you can trace this lineage back to the 1660s this is a picture of Robert Boyle as many of you I'm sure recognized so skepticism requiring that the claim can be independently verified so not content you don't have to email the author for example to try and understand what they actually did in their publication so what this means is we need to have a notion of transparency in the communication of the research process so what went into generating those results and like I said that's from the 1660s okay so what does all this mean about the scholarly record so I have a I have a couple of ideas if we're thinking about what the scholarly record might mean in this computational context so we need to use the scholarly record to be able to access or regenerate scientific research findings so I need to be able to get a hold of items that were relied on in the generation of those published results I need to be able to have whatever information is material for an independent replication or for reproducibility so one of the things that the reason I have two there two options is because every publication around a scientific discovery is in some sense a stylized narrative you never have a sort of a diary of everything those tried and everything that went wrong and all the dead ends the researcher went to if you read these these articles it's always this smooth progression you know right to the results and in a sense that's that's right I mean we don't need to be bothered by the mistakes and the typos and things like that on the other hand we do miss this sort of intellectual growth that the researchers went through and more importantly to statistical reproducibility these avenues that didn't work out can actually be extremely material for understanding the the significance of the ones that did work out so in a sense we've got some thinking to do about well what parts of the story are relevant for inclusion when I'm actually telling others about the results that I've found okay so what what do I mean by these items that we'd want to share we can think about articles or manuscripts text code software data workflow information so how did I actually implement those scripts and the software on the data what were my parameter settings what order were the scripts applied in so how do I actually use all those pieces to knit them together to get those results that are in the publication research environment details what did my computational setting look like when I did those that that inference on on the data and then other items again you'll recognize this is to be part of the discussion around empirical reproducibility that I'm not emphasizing so much in this talk our material objects like the reagents lab equipment instruments text historical artifacts and so on so just as important for reproducibility just not the type of reproducibility I'm emphasizing today so we're sharing items we're sharing items that can change data sets can be corrected updated renormalized software is extremely dynamic and fluid when people are actively using it changing very frequently so it brings in this notion of versioning as absolutely crucial to everything we're doing and we're sharing these types of information okay so as I mentioned I'm not the only one who's noticed these problems a number of researchers have noticed as well so I put together this slide of some of the tools that people have built to try and help with this problem so one thing that you may think is well you know this is a computation if you're talking about computational reproducibility this is research that's happening in silica for example so why don't we build some computational tools that would do things like capture that version of the data set or what software you actually used and bundle it all up and maybe make it more shareable in a more automated way and I think that's a very natural response and I think that's an incredibly important part of the solution and so people have been doing this type of thing and I'll just note that for most of the research tools that I'm pointing out on this slide and it's a hopelessly incomplete list so I don't it's almost a haphazard list so I apologize in advance to anyone that I forgot but the but what's going on that's notable with these tools is most of them these are just faculty members who have developed tools and instruments to help with the research and help with reproducibility on the side of their day job because they think it's just an incredibly important thing in a gap that's missing and one thing I've noticed as this discussion progresses and I'm starting to see this more and more is now commercial interests are starting to come in more and more and provide the support tools so that's just part of the discussion in a dimension that is worth considering so I roughly group these tools into three categories dissemination platforms so I work on research compendia the first one there and I'll talk about that in a moment I Paul image processing online machine learning open source software so this is these are these are sort of post publication I want to deliver you some digital object that I think should be running with my publication and so these are these are some solutions that are that are trying to fill that gap and finding ways to get say data sets to you software to you workflow information to you as the reader and consumer of the of the publication pre publication there are a number of tools also that are being created so as I mentioned helping capture what might be important for sharing before you actually get to that point of publication so these are the workflow tracking in research environments and many tools with a lot of deep work behind them to help capture those environments what variables were actually use what your inputs were to the software and deliver that as part of the publication as well if the author chooses the final category and these are all kind of a little bit overlapping the final category embedded publishing so we still our currency and scholarly record is still the static PDF can we do something a little more congruent to the types of research that we're actually doing in the computational sphere so for example could I deliver you a PDF where you can click into images and grab the code or click in and find the data or maybe even have it regenerate within the PDF for example so I've been calling those embedded publishing and there's a number of efforts around there as well so I think the fact that we're seeing these tools sort of popping up and organically arising many from a grassroots level really speaks to the importance of the problem where the researchers are stepping in and trying to build this these infrastructure responses on their own to try and just do better research what I started the talk sort of loosely calling you know research with integrity okay so I'll mention also research compendia a little bit the one that I've been working on so the notion the sort of the fundamental idea of research compendia is the scholarly record is proceeding more and more publications being added to it kind of every second and many of those publications have data that underlie the results have coded underlie the results I'd actually even make an argument that the majority at this point do and so the question is then is there a way to persistently house data software workflow information those computational artifacts that should be traveling in my opinion with the publication in such a way that they're persistently available to the reader of that original publication so the idea of research compendia was really as a pilot project so we could start to understand well what do researchers need how difficult is this type of sharing what problems and barriers do we start to run into so we're trying to link the data and code to these published claims enable reuse and develop a better understanding or guide for people who are doing this research and being able to promulgate these these additional aspects we also realize that once you have the code and the data we could actually just run it ourselves and certify results even though we're not actually maybe the intended recipient the lots of the work that we've supported on research compendia publishing fields where I don't have expertise for example but we can still get the code running we can still use their data and so we're able to sort of say well we were able to get those same figures same tables as in your paper and so we could do that baseline kind of verification of at least the computational soundness of the results not saying that they're correct but at least saying I could run this the the digital objects that you gave me and I could get the results okay it also opens up other of their opportunities so in a particular area around a research question if people are sharing their data and their software with associated with their publications we could start putting the data together and running the software that we have on a much larger data set and validating those results in a much more powerful way and do things like stability checks or sensitivity checks in the methods and in the data itself so things that aren't the traditional way that we would think about the reason or the purpose of sharing scholarly information okay so here's an example of those pages I was talking about so I just grabbed a page on research compendia the blue title at the top will link back to the published article so this isn't unless it's an open open access journal we're not taking local copies of the publication and making them available for obvious reasons with copyright but at least we can link back to the article in this one this happens to be an open access article so you can see in those buttons about two thirds of the way down you can click and grab the article directly from the compendia page and you can grab code and you can grab data and so the idea here is the abstract describes code and data the digital objects that are traveling with the publication the authors might be programmers or data curators for example people who have contributed deeply to the success of the paper yet may not be listed in the traditional authorship role on the publication there's a lot of kind of norms and standards about how those authors are decided on and then we have additional information around how the intellectual property is licensed here which I'll get to in a moment it's all open source so we have research compendia sitting there on github we do by the way assign do is to code data to the research compendia page and then we have a DOI for research compendia as well and the idea is assigning these DOIs in such a way that they're linked hopefully with the publication that involves more of an engagement and discussion with publishers to do this before publication but at the minimum will have them hierarchically linked within themselves so that they're discoverable as a bundle and then we applied the MIT license very open permissive license to our platform the software that's on github and available so all of you can just grab a copy stand up research compendia your version yourself and work with it or extend it okay so I'd like to finish with a few comments on intellectual property so one of the things that happens in the interaction between software and intellectual property is that software falls under copyright by default so copyright as you probably know extends to any original expression of an underlying idea so the underlying idea is never barred with intellectual property however as soon as you have this in fixed form then this is then the fixed form itself is then copyright to the original author so there's a nice phrase that copyright follows the author's pen across the page you don't have to register copyright this is a default and so we're very used to thinking about copyright in the context of text but it also is equally as applicable in the context of software where we have an idea for what we might like the machine to do and then we go ahead and we code it in some language and that again is putting in fixed form some original expression of an underlying idea so copyright applies so Stalman was one of the first people to notice that this created a big problem for sharing of software so for example if you're the copyright holder I have to ask you if I want to make a copy or reproduce your work so that's very burdensome so I can't just go out go and download software from the web that's sitting out there unless I ask you if it's okay if I go ahead and use it or maybe change it a bit or apply it in a new setting and that lasts of course as we know 70 years plus the life of the author and probably something to do with the longevity of Mickey Mouse as well so the 70 years keeps kind of extending and extending so I consider copyright essentially to be in perpetuity and certainly for software and for these computational objects 70 years is essentially infinite anyway so what Stalman did his big innovation was well instead of having people email me or contact me to ask my permission to use my software that I've put on the web why don't I just attach a little bit of text that says you've got my permission already and here's the permission that I'm giving you you can use it under these circumstances so that was the advent of open source licensing and Stalman's background was to try and ferment and create an open source software community and often his insight around licensing is credited with the successful creation of the open source software entire movement and community however the norms of the open source software community they don't map to the norms of the research community perfectly so there are gaps and one of the things that I thought about was well what would be appropriate leaning on already the enormous amount of work that's happened in open licensing for making these digital objects that travel with research available so we go back to things like Communalism let's make things as broadly available as we can research operates by citation that's how we get our credit so there are many licenses that will do exactly that that will make software for example or copyrighted objects available to do whatever you'd like with as long as you attribute the original author so one of the museum it license that we saw on research compendia for example so my recommendation is take these software objects that are associated with research and make them available with an attribution only license or make them available in the public domain these are the light these are the ways of structuring intellectual property that most closely match the norms that we already have in the research community your media components so the article figures tables making this available under attribution only license for text not a software license like for example the creative commons attribution license that would match our long-standing norms now I just sort of swept aside the whole issue with publishers and copyright but we're just talking in ideals right now the final object data data is trickier to license because there is no copyright on raw facts in the US however and of course the licenses adhere to copyright however the notion of what a raw fact actually is in research I bet every single person here would have a different answer for me and the courts and there nobody's really answered that question so I think arguably if you have done some original selection and arrangement of the raw data in the court in the Supreme Court's words there may be copyright that adheres to at least that original selection and arrangement of the data although not the raw facts so perhaps an open license could travel with data for example or if you want to just forget that whole conversation putting data out in the in the public domain is another way to make it broadly available so the idea is of the reproducible research standard is take away that barrier that's introduced by copyright to the promulgation of Clare about notion of really reproducible research where I can take computational findings and I can get the code I can get the original data I could re-run this presumably and verify those results extend those results take the code put it on my new data or take the data mix it with something that I'm doing and extend the research in what would I think to everyone in this room seem like a very natural thing to do so taking that IP framework and making it aligned with these long-standing norms that I discussed earlier okay so I wanted to just finish with a couple of slides with open questions and queries to sort of sort of really make us think as imaginatively as we can about what this scholarly record could look like so queries that I would like to make on scholarly computational publications I'd like to be able to ask the scholarly record to show me a table of effect sizes p values so statistical output on all phase three clinical trials from melanoma published after 1994 that's a very hard query today I would like to query the scholarly record and ask for all the image denoising algorithms that have been applied to remove white noise from the famous barber image the barber image is used all over signal processing and image processing so tell me what all those publications were and actually give me the citations to of where those public where those algorithms were used and introduced give me all the classification algorithms that were used on the famous acute lymphoblastic leukemia data set this is one also that's very common in a certain strain of literature along with type 1 type 2 error rates of some statistical output as well the output would all be in those papers so we're not really computing anything I just want to deliver what's been published give me a unified data set containing all the published whole genome sequences identified with the mutation in the gene Bracka one that's also a very hard query to do I would like to randomly reassign treatment and control labels to cases around a particular clinical trial published cases and then I could calculate effect size then what I could do is say repeat this multiple times and do it for every clinical trial published in the year 2003 listing trial name and this particular histogram giving these reevaluated results side by side so we could get a sense of the effectiveness of these clinical trials in the in the published scholarly record these are hard today but I would like to move to a reality where we can routinely make these kinds of queries okay so cyber infrastructure thinking about tools I think successful tools need to minimize the amount of time the researcher is going to take in learning the tool and using the tool I don't think a successful tool is something that's going to take a huge amount of research or time and effort and energy we need to think about automating as much of the discovery and dissemination process as possible not automating the intellectual aspects but automating things like what version of that data set was used to produce figures for for example that's something the machine knows and we can just capture from the machine with the appropriate tools what particular frozen form of the software was actually used can I grab that so it doesn't stop us from doing things like fixing bugs and software all software's buggy doesn't stop us from doing things like fixing mistakes and updating data sets for example but we still need those versions that produced those results in the scholarly record mistakes and all words and all as I mentioned facilitating queries across the scholarly record that go to those computational aspects data software algorithms for example and then capturing all that information needed in the research process to allow people in the field to verify and assess those findings it's all really about delivering evidence for those results okay there have been a couple of community responses that I'll just touch on in Yale Yale Law School we did a round table in 2009 with different stakeholders and we called it we had a published work workshop report called addressing the need for data and code sharing and computational science at Brown University we had a similar round table for the week long ISERM series in 2012 so there we titled our workshop report setting the default to reproducible in computational science and research so getting the code out there getting the data out there getting these artifacts out there and accompanying their results in 2014 in the summer with the exceed project which is a project that is meant to facilitate access to high performance computing technologies and machines and systems for users that may not typically work there so acting as middleware software they sort of thought well maybe that's somewhere where we could really be enabling some of the cyber infrared cyber infrastructure aspects of reproducibility since we've already got software wrapped around their processes is middleware maybe we can go ahead and capture some of those things like code data versions and so on machine state for example I won't spend time but of course we're being pushed as everybody knows from the White House with their mandates around access to publications and access to data now notice that access to code access to workflows wasn't mentioned I think that's up to us as the community to wrap those into our discussion and have a more holistic way of presenting the results or really it's about evidence and it's about reproducibility the federal agencies are moving beyond what OSTP is requiring them to do so last year National Science Foundation in September had a workshop they called reliable science the past the robust research results so that's going beyond just dissemination of data for example and NSF actually has a rigor and reproducibility page I'm sorry NIH has a rigor and reproducibility page talking about their efforts to make data available software available and so on and then journal requirements journals another stakeholder also kicking in so I did a study a few years ago on well what are journals saying about code requirements and publication data requirements and publication and they are not to the point where this is solving the problem however they are taking nibbles of the problem and making data available and making code available okay so two different ways to think about this so on the production side we have things like crowdsourcing public engagement and science we're putting data online software online with our published results this is really different from a societal perspective than one researcher writing a letter to another researcher asking to mail the preprints right this is something now people are looking at this from all walks of life and all backgrounds it's not an internal dialogue anymore so this pipeline so opening up this accent to these coherent digital scholarly objects coherent is in quotes because I'm trying to communicate how it should be attached to a published narrative how there should be usability to enable reuse having licensing that enables reuse for example mechanisms for evaluating new findings so what happens when people start playing with our work in the scholarly community and then they come up with really interesting new recent new new insights and research do we have a way of folding that into the larger discussion or we just expecting them to submit their publication to the journal like all of us do and then like I mentioned legal issues of our reuse and privacy on the other side around crowd sort of an opposition to crowdsourcing in this public engagement is the use of the output of our research more broadly than ever so these findings and publications coming from what I've been calling third branch fourth branch computational research are feeding into all sorts of policymaking everybody has heard evidence-based whatever right evidence-based health care evidence-based policy evidence based medicine so this is this is something that's now becoming part of the currency of our policy dialogue itself so how do we know that those findings are really right unless we can go in and look at the code look at the data rerun things and do those verification checks that have always been part of science that need to require more thought and more effort to extend these to the computational aspects okay so for cyber infrastructure dreams and wishes data access software access persistent linking to publications that I've mentioned I'm including in there also the workflow parts of the conversation link do I assignment on articles data code workflows making this entire compendia that we're publishing rather than this static PDF I've talked about data access I haven't talked about any of the barriers to data and code access so for example can we be more innovative when we have say legal barriers to the sharing of data or ethical barriers so for example you have human subjects in your data and we're not just going to be able to make those data available if you do education research we're not able to make student data publicly available but can we be more innovative about how we might provide access rather than thinking of it as a binary that's sort of closed or open I think yes are we using robust methods producing stable results and findings from these data do we have our emphasis correctly on reliability and reproducibility from the findings that were actually garnering from the from the data and I think the cyber infrastructure should be open source and should be inspectable reusable and so on okay and I'll just make a quick note of the Google flu trends argument or sort of experience where if you remember Google decided to start making its trend data available or not well sort of started making announcements and predictions about how it was using its trend data that were actually better at predicting flu trends in the US than the CDC for example and they got a lot of you know press and attention on this because it looked like people's search terms in Google were actually providing a better picture of what's happening in our society then sort of doctor reporting and hospital reporting to CDC that we long relied on so people are very excited about this if you had you know if if our conversation today was of interest to you then you might have had questions like yeah but how do we know Google's getting that right and what are their prediction methods what are their models what's the underlying data all of that was opaque and then in 2013 2014 Google flu trends suddenly stopped working the predictions weren't as good to this day I don't think anybody except the people the sort of small group within Google knows why so the question then becomes when can we rely on this output in this information and when can we not and shouldn't we have better mechanisms rather than just trust me so here are principles for cyber infrastructure supporting the scientific norms I've put a few out there today as I said I didn't put them out in a definitive way but there are these long-standing norms that we do need to support in terms of research that characterizes the research itself supporting best practices in science so this is I think a more open discussion so what should the best practices look like in the computational arena taking a holistic approach to cyber infrastructure so we think about many of the descriptions of examples that I've used in the talk today would easily fall in the paradigm of data science so what does it mean to do data science well does it mean understanding what your data generation mechanism was where you got the data what are metadata associated with the data what have you done to prepare that data for analysis what analysis have you done as I mentioned those are all encoded in software almost never shared and then how why should I believe this what are you doing to convince me that the same machine learning algorithms or whatever it is that analysis you've applied to the data is really right can I even take a look at this did you tweak those parameters for and just get really good results and then publish those or is this something that's more broadly stable and applicable right now it's difficult to understand that for many of the published results so having this and understanding this end-to-end research pipeline right through to reuse sharing and how it's handed off into the next project okay I just mentioned is a teaser there the social and political environment because of course all of these discussions who pays for cyber infrastructure for example they start to involve government entities and funders and we're always always these discussions are embedded in a larger political context in a social context that we need to be aware of and so I'll finish with some open questions who funds and supports say for example tool development cyber infrastructure who owns data code research outputs so I have been loosely talking about it throughout the talk as if the researcher generated and is making it available oftentimes the researchers using data that were generated somewhere else by other groups do they want to put this in a trusted repository that their community uses so we have a generator we have a user who's created results we have a repository who's adding value through metadata persistence discoverability what about the funder who funded the project wouldn't the university say well I also supported that those are my data so we have all these sort of tugs around who owns data we haven't even gotten to the conversation about who owns code and it's something similar so these research outputs I think that's a conversation that's coming who's controlling access who's controlling the gateways having a sort of discussion before it's a calamity I think is important what are the community standards around documentation citation standards best practices who's going to enforce this should it be different for different research groups do we have sort of larger principles that we can communicate to the scientific community then act on and embed in the cyber infrastructure we're developing how do you cite the cyber infrastructure itself how do you reward this this goes back to number one who's going to fund and support it how do you know what's working and what's not working what are the incentives to do this development of cyber infrastructure and how should we be thinking about a healthy system that actually encourages this type of behavior I'll stop here and I think we have maybe not enough time for questions but a minute or two so I'll just open the floor to questions and of course like as I said I'll be around through the whole workshop if you have other questions so thank you hi I was interested in your work on research compendium and whether that sounded like something that could be automated as a confirmation in the future and secondly if there are any success stories hidden in failures where something didn't pass those are those are great questions so one of the things that we've been working on that I didn't show you here is sort of having small virtual machines around the computational deliverable so around the code and around the data so you we're not doing this but you could imagine having this sort of minimal virtual machine that a researcher might even work in and then just deliver that as part of their publication and they you would know that things sort of worked and you'd have that confirmation in advance from the researcher and we're not at that stage yet but I can certainly see that and that level of automation and I and it's a great question I'd really like to see research compendium even more automated than it actually is we just sort of lay we're layering right now on top of the existing system you know so it's clunky in a way and we can certainly be smoother in the future so that's a great question yeah Dave Rosenthal from Stanford two quick points on that firstly availability of the source does not guarantee reproducible of course execution we have the example of the emulation of the chase software that was done by the team at CMU where this the code was deposited with the paper but it requires a specific version of Ubuntu and a specific set of libraries in order to run it yeah and the other is well people are starting to deliver virtual machines we noticed that the feed that we get from ACM for the clocks archive last fall suddenly became bigger than the feed we get from Elsa there which was quite a surprise the reason was too far one of them was it started being filled up with videos of people giving presentations which take a lot of space the other was it started being filled up with the ends yeah and most of these are VMware virtual machines which are huge right which are huge which there is no guarantee that we can run those in the future and which contain all sorts of potentially licensed software that we have no idea what it is yeah and so we were forced to consult our agreement with ACM and point out that they were indemnifying us against copyright violations by the stuff that they were delivering us and we were going to depend on it yeah yeah no those are excellent points i could have spent the whole talk just talking about software sharing and all this sort of pitfalls and issues so one of the one of the one of the points that you're raising that's extremely important is just having the source code doesn't necessarily mean anything right maybe you can inspect it but it's hundreds of thousands of lines long how do you even open source something that big it'll take you years to even read it so so the way that i've been thinking about solutions around that is again actually leaning on the traditions of the open source software community where they would deliver software with tests and we do not do that at all in our research in this community in the research community and we have no notion of sort of software testing so if i wanted to sort of understand how a piece of software worked and maybe whether or not i should trust it i would immediately start leaning on tests and that's a that's a conversation that's just starting so certainly software is neither necessary nor sufficient for reproducibility however i think in the majority of cases it's extraordinarily helpful and of course the ideal would be running the software and i just sort of skimmed over that like how we would just run it all and that's non-trivial as well in terms of what needs to be delivered to actually run the software and as one person said at one of our workshops you can't virtualize blue waters so sometimes you're on these very specialized pieces of hardware and what does reproducibility mean in that context so there are fascinating edge cases i think around these issues however i think what i'm talking about probably applies to upwards of 80 or 90 percent of the research that's being done where you have relatively short scripts on static databases and we could do a better job sharing that as a default openly and then start to think about well maybe i can use sort of very lightweight virtual machines or start to innovate around this i don't think these problems are insoluble i think there will always be very challenging cases though yes oftentimes in the web development things that are just good enough tend to to take it take off and take advantage over more worked out scholarly and formally devised science systems jupiter hub stands strikes me as one example of that because it seems to embody a lot of elements i'm just curious about your reaction to that as a submission mechanism for journals and things of that sort yeah no this is this is taking off and i think this may become the next standard so i do oh my slides aren't on there but it's fine so that that slide that i had with the infrastructure responses one of the ones that i point to is jupiter on there and this is being used everywhere people um if you remember we had the breakthrough on the detection of gravitational waves well the computational aspects are reproducible and they're shared in a jupiter notebook for example and so that's something where we sort of start the conversation in the right way and then we can sort of fail fast right and it's i think we certainly are going to do things like research companion is an example where i'm building on an existing system it's certainly not beautiful or sort of highly developed software or what not but we're trying different things out and we're prepared to make mistakes and sort of have an agile response and my understanding from jupiter too is that that's similar so we can actually be extensible i mean um Fernando Perez evolved it from um python out to say our Julia and other um pieces of software that are being used to all over science as well so it's certainly the case that we're not going to hit perfect the first time and i'm fine with not having the beautifully engineered solution as you mentioned and just taking um what we can get and start building on it we just need to be sensitive to the brittleness of software which is once standards are evolved and in place particularly with software it is difficult to change course so just sort of thinking about what what would it mean to maximize that ability to be agile and that's part of the reason that underlies my push for open source in the platforms not just in the research as well okay it looks like we are out of questions and out of time so thank you very much thank you very much victoria that um that was a really illuminating talk um and there's a lot to think about i suspect you're going to be having a lot of whole conversations in the next uh 24 hours or so thank you again


What's going on, everyone? Sam here with Layton Sportsguards doing 2019 and 21 Panini Contenders football hobby, four box mixer number two random team style. Here's a look again at your random results. I'm starting at Donald Day right here. As always, you can find these on the website under the results tab, as well as the link that's currently in chat. I would say probably do that instead of look here because we're a little bit down, so there we go. So good luck guys. Let's do some contenders, man. It's always fun to do some older contenders, as well as last year's. What's yours 2019 and 21? 19 and 21. Yeah, no one else is around. True. What's the name? Perfect Corner? You were David Oh, right? David Oh, let me double check. I might have typed in the wrong initial. That's a Sam specialty. That is a complete Sam specialty. It seems like every night that happens at least once. Yep. Yeah, you're in this, David. You got the Colts. You've got the Colts and I did type it in right. So yep, David, you got the Colts in this one. Good job. Thank you. Thank you. I think you're graduating the five. Oh, wow. Look at me. Jacobi Myers. Nice. That is going to the Patriots. Congrats on that New England, Troy W. One of those lesser known guys that was in the set that ended up being pretty good. So congrats on that, Patriots. Legacy contenders for El Davis. I'll card ship on this. Shooters if that base doesn't, that might be the case. Yeah. If that's how I'll, uh, beautiful. No organizing that I have. Oh, we're going to have to do it double love, actually. Yes. Soon to be franchise tagged to Colby Myers. That's true. Von Miller. You're good, David. No worries, dude. Biggest blunder of sports. Oh, nice. Meekle Hardman Jr. on the on-card rookie ticket. Congrats on that, Chiefs. Heck yeah. We'll take that. Mac, you see. There you go, buddy. That's Super Bowl participant Meekle Hardman. That's Super Bowl champ Meekle Hardman Jr. right there, I think. Yes. Yeah. He won in, uh... He won with, in what, 19? Is rookie or maybe? Yeah, 2019, uh, the one that they didn't lose the bus. Yes. Um, round numbers, one, two, three, top, four, five, six, bottom. Uh, this will go for the rest of the break, because the round numbers are top and bottom for 21 as well. That is going to be a six going to the bottom. This will be going to the bottom. So this will go with, uh, 20 Pollard. Thank you, Lucis, 19. That's what I thought, it's wasn't sure. Colby Myers is QB for the Raiders. I like, uh, I like rooting for chaos on that one, illegal. I'll root for chaos with you. JJ Arcega White side. That is Donnie's boy. That is Donnie and Bryston's boy right there. I'm going to have to say no, man. Bryston. Bryston. JJ Arcega White side is your boy, right? No. He said yes, by the way. So why? There's Tyrell Williams to 175 for the Raiders. For the Spider-ish. You're something. You are something, all right. Harry McClearn. Scary, Terry. Scary, Terry, uh, cracked ice would be cool out of this, maybe. A lot of good receivers this year, dude. Points. I'm going to flip it over. Uh, it's going to be random to all at the end. 23 to go in the Super Bowl mixer. Oh boy. Oh boy. Make sure to get your spots now. That is going to be fun. That is going to be absolutely insanity. Chaos over. We're actually going to rip the 2000, uh, the 2000 Black Diamond first so Rich doesn't get to. Ben Benagu. That is going to the Colts. Nice one there. There you go, David. Congrats, buddy. First break mojo. Some Ben Benagu auto. David Montgomery rookie to your contenders and base. That is Quadrae Olson for Pittsburgh. No, for the Falcons. Oh, from Pittsburgh. That's why. That's for the Falcons. I thought it was a, uh, Pittsburgh card, but nice. That is going to the Falcons, obviously. Can only get three spots. Donnie said he's going to buy a spot for me for my birthday. That's my birthday present. That's my birthday present from now. Thanks, Donnie. Interesting. Interesting, dude. I appreciate your man. Interesting, this guy. Did I generally think if I won the lottery, I'd buy like 10 spots in the mix? I literally would. I, yeah, I'm freaking insane. I just buy out the rest of it. Yeah, honestly. MVP contenders, the Sean Watts in the 49 base. There's, uh, Trevon Wesco for the Jets on the rookie ticket auto. Trevon Wesco. There's some like really low key good guys in here too. Like I think Dawson Knox has his ticket in here. Which is pretty solid. Yeah, there's some, there's some good like of those sticker auto guys. Cause then you'll, every now and then you'll get like a good kicker and a good lineman and stuff like that too, where that this will be like their only autos. Right. We'll be in contenders and it's like theoretically their best card, I would say. It would be like their like a contenders ice or something. Yeah. Of like guys who don't have a lot. There's Greedy Williams. Nice for the Brown Scrats on that Browns. Heck yeah, we'll take that Brandon C on the combo spot. Greedy. That is a, uh, right there. Me and my buddies would always do like Madden franchises. We'd all pick a team. Greedy Williams was like always drafted by somebody in the late rounds as like their slot corner just because just because he was always available and he was like a 70, like six overall. So you'd always get like an on-card airsid bucker. Geez. That's aggressive. I'm going to watch out for a Lucis. Deandre Baker. Oh boy. For the giants. Nice one there. GMN Chris H bottom with DK. It's like an AJ Brown or something. That'd be cool. We got Miko Hardman. So that means we should get an AJ Brown. I mean, right? We are a chin. Oh, Rashawn Gary. Nice one there. It's a 99 for the Packers. Congrats, Green Bay. Very solid. Donald A. There you go, buddy. Nice one there. Nice Rashawn Gary. We'll take that. Preston Williams. Devonta. Yeah, I agree. Devonta Smith being the Superbowl X Factor. Yeah, that's right. Hawk. Deandre Hopkins. Redemption. It's going to be Hunter Renfro, rookie championship ticket RPS. We'll be going to the Raiders. Kenny S, again, not. It is expired. I would still reach out, see if they've got it for you. Nice one there, though. Renfro on the on-card base. Oh yeah, VJax. VJax is awesome, dude. RIP, man. Debo rookie. Base. And base. All right, on to our 2021 now. Let's see a big QB. Let us see a big quarter back. Down to five to go, guys. Five to go on Zenith and two to go on Spectra. So only seven group break spots left tonight, guys. Thank you again for getting everything filled up. You appreciate it as always. Best customer base in the world. Not even close. Not even close. That's Pete Verner. Nice one there for the Saints on their rookie ticket audit. That is winning ticket, A.J. Brown. I know that was sad, dude. That was really sad. Chain movers, Aaron Rodgers. Again, guys, if you head over to Drip, we do have a stream set up for the Superbowl Mixer on Saturday. If you guys go in and just bookmark it, all you have to do is bookmark it. That's all you have to do. And if we get enough people to bookmark it, we will get free shipping on Drip for like a month for all of our customers. So you don't have to watch the stream if you don't want. All you have to do is bookmark it. But it's also helpful because if you do bookmark the stream, then it'll tell you when we go live so you don't miss anything. It'll notify you when we do go live. Tony Gonzalez, Legendary Contenders, the 99 for the Chiefs. That is going through the Rams. Rookie ticket audit. Odafe, OA for the Ravens. Nice. The 25, 18 of 25. There you go, Baltimore. Nice one there. Still looking for our on-card here as well. See if you can get somebody big. MVP, Contenders, T-Law. So that's going to be a fun day. Saturday is going to be a lot of fun. Yeah. Yeah. Got Travon Morig for the Raiders. Nice one there. Starting safety in Vegas right now from TCU. Should have been a Raven. A lot of should have been Ravens in this. We got a redemption of Elijah Moore. Rookie ticket variation. RPS going to the Jets. Nice one there, New York. That is Kevin M. Congrats, Kevin. Excited for this year's Contenders to come out. It's going to be fun, man. Going to the Jets. To the Jets. Just like Rogers. Justin Fields, Supernatural Rookie. Rookie of the Year, Contenders, Michael Card. Rookie of the Year, Contenders, Davis Mills. Next one. Good luck, guys. See what we've got here. That is Zach Wilson, Supernatural. Check up Harris. Going to be our on-card auto for the Rams. That is Rookie. Nice one there. Congrats, LA. That is Troy C on the Combo spot. TCU Atwell. CMC. Jaylen Hertz, MVP Contenders. I remember when I was ripping Contenders way before the season started, I saw that and was like, ha, ha, good one. I got proved wrong. Yeah, how are you feeling now?  I'll eat my words on that one. There's Joe Montana. Playoff ticket to 99 for the Niners. Miscalculated. Yes. Yes, Ryan. I look like Matt Stafford a lot actually. Since I've started breaking, I've got at least like five or six people saying that I look like Matt Stafford, which I will take. I will take that any day. He literally talked about that like yesterday. Yeah. Ernest Jones for the Rams. Nice one there. Rams has gotten a lot of hits out of this. Very nice. I look like Matt Stafford with a patchy beard. That's what I look like. Matt Stafford, had a really bad beard. Yeah, sir. Matt Stafford had a good beard that wouldn't grow right there very well. Winning ticket there in Waller. Got Davion Nixon for the Panthers. Rookie ticket auto. On the rookie ticket auto for the Packers. Nice one there. It was under Tralance. Hey, you ended up being right though. Got to give you that. Bobby Wagner power. We'll see if Hertz could really or not write a title. Literally. I wouldn't mind. I really do like Jalen Hertz. I would watch him more like his interview stuff. He seems like he's really in it. Yeah. Oh yeah. I always really liked him ever since he got benched at Bama in the national championship and transferred to Oklahoma. Do you remember that?  Tim Brown, 19 out of 20 contenders to Canton. Nice one, Raiders. Kenny S. Congrats, buddy. Love these on-card Super Bowl autos and base. Let's do a quick recap here. No, I'm like, I would generally see it. I feel like he plays like a chip on our shoulder for that. You know what I mean? Oh, we got to do the points random. I'm going to do the points random really quick. No, just that way you said it.  No, I do like Jalen Hertz though. I do. All right, guys, going to do the random for 250 points. I'm going to roll the dice, whatever teams on top after 12 times. We'll get the points. So good luck, guys. One, two, three, four, five, six, seven, eight, nine, 10, 11, 12. Going to the Bears. Bears getting the points. Do a quick recap, everybody. We had really cool Tim Brown for the Raiders, Kylin Hill Packers, Davion Nixon-Panthers, Ernest Jones-Rams, Jacob Harris-Rams, Elijah Moore Jets, Trayvon Morrig Raiders, Odafe O.A. Ravens, Bobby Brown-Rams, Pete Verner for the Saints, Hunter Renpo Raiders, Rashon Gary Packers, Deandre Baker-Giants, Grie Williams-Browns, Tramon Wesco for the Jets, Quadrae Oleson-Falkens, Ben Menagu for the Colts, Miko Hardman for the Chiefs, and Jacobi Meyers-Patriots. We'll do it for the break, guys. Thanks again for the fill. Next up, we have more football.


that, yes, has arrived from ASIX, but I'm not going to give you my thoughts on it very much because, what that on? Alright, I remember yesterday we tightened the screws a little bit, tightened the screws down, feeling it today in the legs, but it feels good to go to work in the Nike Alpha Fly Next Percent. Oh man, if you haven't seen it, it's a lot of people watching it up a right hand corner. It was a fun one just to get the first run under the belt. Alright, good morning and I'm walking out of the studio. Here we go. Yesterday's 60 degrees, beautiful day for a 23-miler.  Let me just set the camera up here. Okay, we are set up and ready to go. I'm realizing you probably can't see the ASIX Gel Nimbus light, which we're going to talk about today down there, so I'm going to place it, replace it with a Hoka Elevon right there and yes, I took the Gel Nimbus light out for another run today. Now it's 75 miles in these shoes from ASIX. Today's run, 10 miles, 915 to 920 a mile. Nice and easy. Classic bopping along. Yes, a lot of questions on Strava about why am I running so slow. Just got to go look at yesterday's run and then watch a vlog about, I explain why I run slow in order to race fast. So go check out that vlog if you want to understand a little bit more of my approach to easy day running and yeah, there it is on your screening kilometers, I believe it's 545 per kilometer. So here's the deal. It's snowing out. I was trying to come up with a game plan for today's vlog and I was like, gosh, I wanted to really film today but I'm glad I didn't because my hands were freezing by the end of the run. I definitely wore the wrong gloves today. That was my fault because it hasn't snowed a little while in Denver. So I'm glad I didn't film but because of that, I was pivoting. I was thinking, you know what, let's, I could talk about Hoka. I could talk about Nike. I could talk about Soccany, but why not talk about ASIX and the ASIX running shoe 2020 lineup, which if you would, I mean, I would have taken that bet any day. If you would have said Seth, I bet in nine months from now, from last summer to this point, you will be making a new, and yes, here's a shout out to a new playlist upper right hand corner that Seth, you will make a new playlist all about ASIX running shoes at some point in 2020. I would have laughed. I would have laughed. I would have taken the, what is it, the over on that or the under the, oh, I don't know. I'm not a betting guy, but anyway, I'm shocked that. Yes. Today we're talking about the ASIX, we'll start here, the ASIX glide ride, the ASIX EVO ride, the ASIX Nova blast, and the ASIX gel Nimbus light and a new one just for, just for fun, the ASIX GT 1009. I'll open this up in a minute. So bottom line, it's amazing. I'm, I'm excited. I'm excited. Why competition in the running shoe marketplace? We hear so much about Nike. We hear a lot about Adidas, especially over in Europe. We're hearing a lot more about Hoka in the, let's say the last five years, which is all good. But ASIX is a classic company, an older company that I think, as I said in 2019, and I may be eating my words now, maybe lost their way a little bit in the innovation department for running shoes. At least that was my experience over the past five years, by going to running shoe stores, picking up the shoes, holding them in my hands and just being like, ah, this just doesn't feel good. It feels a little heavy, feels maybe a little stiff. Well, I think ASIX is finally coming around updating their running shoe lineups. Now I will say that I do regret in 2019, not purchasing the MetaRide. So I don't own the MetaRide. Just so everyone knows the MetaRide is the high-end version of the guide sole technology lineup from ASIX. So you've got the glide ride, you've got the EVO ride, and then you've got the MetaRide. I do not own the MetaRide. I might pick a pair up. Now listen, they're not cheap, $250, but I might wait until summertime to see if they come out with a version two for the MetaRide. So that's the only shoe that I cannot put into this discussion about ASIX running shoes. And now rather than go over all of the drops and all of the weights for all of these shoes, again, I made a brand new playlist upper right hand corner with the first impressions and full reviews of all of these ASIX shoes. Okay. So go check it out. If you really want to go deeper into the stats behind the shoe, rather I want to mix these shoes into the running shoe rotation discussion. I'm a little amazed at how this vlog did from about, gosh, maybe two months ago now, I think it's almost at 50,000 views where I basically laid out my strategy for building up a running shoe rotation. Now, what is that? What do I mean by running shoe rotation?  And over time it takes time, takes patience, but over time, and yes, there is a shoe that I think can accomplish many of these tasks that I'll get to in a minute. Uh, but over time you can purchase running shoes that pair or match the type of run that you're going on that given day. Uh, so that is one of the goals of this blog today. Okay. And one last shoe that yes has arrived from ASICs, but I'm, I'm not going to give you my thoughts on it very much because, um, I haven't run in it yet. All right. But it is. Okay. So shout out to all of, uh, I do feel a little bad. Uh, these are all neutral running shoes. Okay. So people that don't overpronate or supinate, uh, too much, uh, this shoe, and I, it's just like that's hot. That's the type of shoe that I prefer, but this shoe has some, um, uh, basic, uh, a little bit of stability built into the outsole, uh, mostly. Okay. So the hard outsole, uh, ASICs high abrasion rubber is just really built up on the outsole of this, uh, ASICs GT 1009. Okay. So overall, um, and whoa, if for all the stability runners out there $99 on running warehouse right now, $99. I'm excited. And yes, it does have the flight foam midsole, the, uh, uh, midsole foam that you also see in all these shoes over here, but bottom line, I cannot wait to try the shoe out shoe out for all of you. It's just a stability shoe and it's not on the top of my list, but for $99. Hello, brand new. That is pretty amazing. All right. Now let's dive into the four shoes. Once again, the glide ride, the EVO ride, the ASICs gel nimbus light and the Nova blast. All right. And I'm going to start with my favorite. You all know how much I love the glide ride shoe. So this has the flight foam midsole technology, which provides this, uh, really amazing bounce and responsiveness through your foot strike. I use this shoe for my long runs. It's, it's a little heavy. You all, I've said that a lot. It's a little heavy, but the toe off. So the shape of the midsole that you're going to find in the glide ride, in the EVO ride, and from what I've read also in the meta ride as well. It's, it's, it's reminiscent of what Hoka is attempting to do, but the midsole in all in the EVO ride and the glide ride is just a little more, I'll just say stiff, but not in the sense where your foot is landing and it feels like you're hitting the ground hard, like it's a hard landing. It's more, um, almost like a, uh, a spring that's like a really tight spring. And that just gives you an incredible from what I have found in all the testing I've done. And yes, I've taken this shoe, I think to 175 miles, a prop right around there, which tells you, you all know, like, that means I love this shoe because I usually don't take shoes to that high a mile is just because there's so many different shoes to test out for all of you. So overall love the glide ride for the long runs, but jumping down to the EVO ride, um, this shoe is much lighter than the glide ride. In fact, there it is on your screen, some stats for you as far as weight versus the glide ride, EVO ride, all right there for you. So definitely a tempo day shoe. Absolutely. Um, I don't think it's an easy day shoe. It's, it's just not, it doesn't have quite enough forgiveness through the midsole, but it's lighter. Okay. It's lighter than the glide ride. And that is why I do believe the EVO ride would be perfect. If you want a shoe that's dialed into tempo day paces, I think this could be the shoe for you. All right. So loving the EVO ride, um, even though, and yes, it does, I will just mention it does have that ASICs high abrasion rubber built into the outsole to help increase the durability. All right. Moving on to the ASICs gel nimbus light. Now when this shoe arrived at my house, I was a little, um, I wasn't excited. I'll just put it that I was not excited. I was like, ah, this looks like a, it kind of looks like an old classic ASICs that, uh, from maybe let's say five years ago that just didn't have that, um, forgiveness through the midsole as far as landing. But sure enough, I've already saw today, I think pushed me to 75 miles in this shoe, which tells you, like even though I've passed 50 miles, I'm still running in the ASICs gel nimbus light. It has that flight foam midsole, uh, foam. And I love the flight foam just so everyone knows, but it's a little more forgiving than the, definitely more forgiving than the EVO ride and the glide ride. Okay. So if you just do the kind of the compression test with your thumbs, just pressing in there, you can see, I don't know if you can see that, but it's, it's got some pretty good forgiveness, which, uh, tells me, so this is what I said in the full review of this shoe. It's the classic daily trainer, classic daily trainer. Um, and can you use it for an easy day? Absolutely. I think it's a solid easy day shoe, but like today, 10 miles, yes, I was going easy day pace, uh, but I think you could also use this shoe for a little more, uh, kind of those easy days, stezy days and even potentially steady days. All right. So I think it's got a little more potential than just an easy day shoe. So if you're looking for an all around daily trainer, I think this would be a good choice. All right. Moving on. Once I, I really like this shoe, everyone. And again, plenty of outsole rubber, uh, that ASICs, high abrasion rubber, but the rubber, this scares me. I don't like rubber this much rubber on an outsole. I think for me, when I see this much, it means the landing is going to be a little too hard. No, no, no, no. I don't know what ASICs did, but the rubber is durable, but still forgiving. All right. So again, I don't know. I like, I am at 75 miles and I barely see anywhere and tear on this outsole. Just so maybe just a little bit through the forefoot, just a little bit. Okay. Moving on to, oh baby, the ASICs Nova Blast. Okay. So the ad, I should say this is a different midsole foam, the flight foam blast FF blast, which I need to do a little more research as to how they changed it from, let's say the gel nimbus light. I think what it basically means is the midsole foam is not as dense and it has a little more bounce. Okay. And yes, I'm going to say the keyword for this shoe is bounce. It is bouncy. It is forgiving. And remember, I said a minute ago, like what could be a shoe that you could use for a lot of different tasks in your day, in your weekly training, um, as far as a shoe that, um, as far as a shoe that can accomplish a lot of different tasks, but it's you only need to buy one shoe. This is it. The Nova Blast. One big caveat. You all know it gave me blisters on my Achilles tendon. I'm actually going to take this shoe out for more running either this weekend or next week. And I'm going to put band-aids on my Achilles tendon just to see if that helps a lot. I don't know what's going on, but other than that issue, this is, oh, this is, this has the potential to be my favorite running shoe so far in 2020 and definitely from ASIC so far in 2020, even despite it giving me those, uh, those blisters. So that tells you how much I love this shoe. It's bouncy. It's forgiving. You could use it for easy days. It's here's the weight of the shoe. So you can also use it for tempo days as well. And I think it's got enough midsole stack height that yes, you could use this for a 15 to 20 mile long runner, middle distance run, a hundred percent. And I need to give ASICs a call and see if other runners are reporting that they are getting blisters through their Achilles tendon. Maybe it's just me. I don't know, but I'm excited to try this shoe out in the summertime because the breathability through the toe box for those hot, hot summer days, which crazy enough, even though it's snowing outside, they are on the horizon, those hot running days, at least in the Northern hemisphere. Um, so bottom line really excited about the Nova blast.  The MetaRide is supposed to be, um, you know, potential for a long run shoe, but even a racing day shoe, it better be at that price point. Uh, but it's really fascinating that it is a zero drop running shoe. It's, it's kind of gets complex because of how they construct the midsole. But bottom line, I plan to test out the MetaRide for all of you sooner rather than later. And one more time. Here we go. The glide ride for your long runs, the EVO ride for your tempo runs, or I think you could also take this on a track and do some 1k repeats or some 800 meter repeats. I don't think I would use it for, let's say 400 meter repeats, but absolutely I would take this out on the track and then your gel nimbus light, just your classic daily trainer. I don't think I would use it for a long run, but a classic daily trainer. Lastly, to accomplish all the tasks in the toolbox, uh, you got your A6 Nova Blast, A6 Nova Blast. And of course this GT 1000, which for the stability runners out there, um, I will test this out for you sooner rather than later. I promise. I promise. All right, everyone, question of the day. What is a, uh, what is, which running shoe company has surprised you the most? Let's say in the last six months, 12 months or the last two years. Okay. What running shoe company has surprised you the most with their running shoe innovation with shoes that you've tested out and, and liked. Uh, for me hands down, A6 hands down. It's not even a close contest. So that's the question of the day. Thanks for chiming in. Thanks for listening and a little update for all of you on some A6 running shoe. We're just one day at a time here as we work through this, uh, global situation together. We've got this everyone. We've got this. All right, everyone. Thanks for being here. Thanks for watching. We're going to toss it back to, uh, we'll toss it back to the A6 Nova Blast full review. Uh, that was mixed in with some other shoes as well. A6 Nova Blast right there. Uh-huh. Check it out. All right, everyone. Thanks for being here. Thanks for watching. See beauty or card and love each other. See you tomorrow.


Ariana Grande is easily one of the biggest stars in the industry right now and her album Thank You Next has hit the ground running. The title track off of the album broke more than one record upon release, including most plays in one day by a female artist on Spotify. This album's honestly amazing and I love every song. Not to mention Ariana just took home her first Grammy this year, winning Best Pop Vocal Album for her previous album Sweetener. We all know she's an incredible vocalist but in this video we're going to be talking less about her talent and more about her transformation over the years. From her red hair and victorious to her now infamous ponytail which has become part of her persona. What's going on guys and girls, my name is Cara and this series we've titled Before and After. We're going to talk about Ariana's style, her hair, skin, and any rumored cosmetic procedures. I've done other celebrity transformation videos on the likes of Gigi Gorgeous and Farrah Abraham. We'll have links to those at the end of this video. If there's anyone else who want me to document, let me know in the comments down below. Now let's get into this video. With Ariana having spent so many years in the spotlight, there are a couple of looks she's been known for. Who can forget her bright red hair on Victorious? If you look back at old reruns, I feel as though she looks like a totally different person. There's definitely speculation about what work the star has gotten done over the years. I think the changes have been gradual enough to not really notice unless you look at photos side by side of Ariana then and now. I think she's always looked amazing and she's one of my favorite females in Hollywood. Here we'll look at Ariana's beauty transformation over the years and see what she's done to get the look that she has now. Starting with Ariana's face, I think the most noticeable things about her that have changed are her eyebrows and eye area and her nose. Although it hasn't been confirmed, most likely Ariana has gotten some form of eyebrow lift. A brow lift or forehead lift is a cosmetic procedure that raises low or drooping eyebrows. Before 2013 or 2014, her eyebrows were closer to her eyes and lower on her forehead than they are today. One website spoke about her eyebrow changes stating that in their new position Ariana's face became decidedly more softer and feminine. A side effect of this plastic surgery procedure is that the pulling up of her skin has made her eyelids look larger. The skin and fatty tissue no longer weighs down her eyes. Next we have Ariana's nose. Although she's always had a cute and feminine face, her nose appears to have gotten smaller. She hasn't spoken about getting a rhinoplasty either but if you compare her nose from some years back to her nose now, the changes noticeable in her nasal bridge. Today Ariana's nose is more refined, sleek and points upwards. Rumors have also circulated that Ariana may have gotten lip injections. Although her lips have been looking bigger, it's not that extreme. Moving on to Ariana's skin, Ariana always seems to have a glow. I don't remember the last time I saw her without a tan so you think it's natural considering her Italian-American background but that's not the case. If we look back to her early days, Ariana was actually pretty pale. She probably got used to sporting a glow or fake tan and liked the way it looked, enough to keep it going permanently. I can't blame her though because I always feel more confident when I have a tan and it definitely suits her. Looking at Ariana's body, she doesn't seem to have any work done as far as I can see. However, her tiny feminine figure can often make her look like a little girl, especially since she stands at only 5'1". Ariana also appears to have gotten thinner since she was a star in Nickelodeon. These days she seems to generally stay the same size. I'm not sure if she's any diet secrets but Ariana told Mail Online, I've been a vegan for the last few years and it has been a great experience for me. Maybe that helps, who knows? Ariana's also said, I try and live a healthy lifestyle, eating right, exercise and sleep are all so important. I try and stay really active all the time. I like to hike when I can and being on stage is a great workout. Moving on to Ariana's hair, Ariana is naturally a brunette. In 2010, we were introduced to Ariana as Cat Valentine on Victorious, where her signature look was bright red hair. To keep up with this unnatural look, Ariana had to constantly color her hair and wear extensions, which resulted in a lot of damage. While starring on Sam and Cat, Ariana stated, I had to bleach my hair and dye it red every other week for the first four years of playing Cat, as one would assume that completely destroyed my hair. I now wear a wig on Sam and Cat. My real hair is back to brown and I wear extensions, but I wear it in a ponytail because my actual hair is so broken that it looks absolutely ratchet and absurd when I let it down. Around 2013, we started to see the beginnings of Ariana's high ponytail and she went for a lighter, dirty blonde color on her hair. From then on, although the high ponytail hairstyle went through variations, it became a staple of hers. She also doesn't hide the fact that she wears extensions. I mean, listen to the lyrics of Seven Rings. Although she switched it up like her platinum blonde hair for her sweetener album and No Tears Left to Cry video, she's back to her brunette roots at the moment. Looking at Ariana's style, her fashion choices have also evolved over the years. Out of the countless outfits and looks she's been photographed in, there are a few styles that appear more often than others. Ariana used to wear cat ears a lot and she discovered them on a trip to Japan where they're popular. I think the look gave Ariana a playful image and she's worn them many times over the last few years. Sort of like throwback. I love an animal ear. Clearly. She also seems to love dressing in oversized clothes, often wearing big hoodies or shirts as dresses and pairing them with thigh high boots. This trend is also known as lamp shading. One could assume that thigh high boots are her favorite type of footwear, considering she wears them all the time. And they probably make her taller. She also always looks great on the red carpet. One of my favorite looks was definitely this fluffy black dress she wore. Ariana has embraced many fun changes over the years, but despite her constant evolution, she's always a trendsetter. Plenty of her looks have been copied by popular clothing stores and I totally get why. Regardless of what the star may or may not have gotten done, she looks gorgeous and she's definitely the envy of many girls out there. Ariana has gone from an innocent looking teen star to an icon over the past few years and she keeps getting better. All right guys and girls, that wraps up this video here on Before and After. If you wanna talk more about her transformation, you can hit me up on Instagram.  Let me know who else you would like me to make a Before and After video on and I'll see you guys in the next video. Bye.


In this presentation, we will discuss materiality, determination, or in other words, how to determine materiality. As we go through this, note that we're still really in the planning of the audit stage. And when we break out the audit, we broke out the audit into stages. We're thinking about this in the stages. We had the client acceptance or continuance. We had the preliminary engagement activities. We got the planning of the audit. Then we consider an audit internal controls. Then we go through those audit business process and related accounts, those substantive tests. Then we complete the audit and finally evaluate results and issue the audit report. Important to keep that in mind. We're typically here, we're focusing here mainly when we're thinking about the evaluation of materiality at the beginning of the audit. So as we go through the audit, we have an idea of what those materiality levels will be, allowing us to better plan the audit and then execute that plan through the audit process. So then first, what is materiality once again?