But he's going to give us a green light, or something? I don't know. I'm guessing. Oh, yeah. Mic is on. Perfect. Thank you. You ready? Yep. OK, cool. Well, hi, everyone. Thanks for being here. Today, we're going to be talking about a project by the OpenSSF called Scorecard. And we're going to walk you through what it does and how you can use it to assess the risks of open source projects. So just a little bit about ourselves. I'm Laurent. I work at Google as a security engineer.  I work on Scorecard, which is the topic of this talk, and on Salsa, which I think you know all about by now. And this is Navin. Hi, I'm Navin Srinivas, and I contribute to a few supply chain projects. I work for Endor, and I'm also maintaining along with Laurent on Scorecard. All right, so all of you know about the OpenSSF, so I'm going to skip that slide. If you're interested in joining the public weekly meetings or the Slack channels, just visit the openssf.org website. All right. Thank you. Thanks. So the agenda for today, it's a packed agenda. So we're going to start by talking about the attack vectors in supply chain in general. And then we'll introduce Scorecard, which we built to kind of have this holistic view about all the different kind of attacks and try to detect them. We'll show you a demo on how you can use Scorecard on your own repository by installing the GitHub Action. And then to make the talk a little bit more interactive, we have prepared a little quiz to show you the kind of stuff that Scorecard can detect. And for those of you who participate and answer, we have several OpenSSF t-shirts that we'll distribute at the end of the talk. So stay tuned. And then in the second part of the talk, we'll show you how you can use Scorecard on your dependencies to assess their risks. We'll show you the demo of the CLI, and we'll talk to you about the weekly scan that we have that we scan every week about one million project. And we have a publicly available data set that you can use to learn more about your dependencies. All right. So I don't think I need to motivate very much the problem with the supply chain. So real quick, supply chain attacks are on the rise. And people are realizing that it's a lot easier to compromise a dependency or attack the supply chain than it is to find a bug and exploit it in the software. And over the past few years, we've seen significant rise in supply chain attacks. Real quick, this is a short timeline about some of the prominent attacks that happened over the past couple of years. One of them is KotKov, which affected around 30,000 customers. And for about two months, the attackers had access to the CI CD system of customers. So if you use KotKov, say, in the same environment where you were maybe storing credentials or private keys, those might have been compromised. In this slide, we've tried to visualize the various components of the supply chain. And as you can see, there's a lot of squares and a lot of places where an attacker could try to target your project or your infrastructure, starting by sending malicious pull requests to your project. If you have an overprivileged CI CD system, it's possible that a malicious pull request could abuse the overprivileged CI and do something malicious with it. I will show a demo of this live later in this talk. Of course, someone could also try to compromise the repository itself where you store the code. This happened in the PHP attack a couple of years ago. The build pipeline also is a place where attackers might strike. I don't have to motivate this. It happened to SolarWind. Another place is taking over package manager's account. This was, again, demonstrated, I think, a couple of weeks ago. Some researchers showed that by scanning for expired domains, they could re-register them and then initiate a password reset and then take over the account and push malicious packages. And last but not least, the dependencies, which are the backbone of the supply chain, attackers tend to try to use this technique called typosquatting or package squatting. They simply register names that look alike genuine packages in the hope of tricking people into installing the malicious version of the package. And because of this large attack surface, this is kind of the reason why the OpenSSF scorecard project was created. To kind of give you this holistic view about the different signals and the different security risks of a project. So in a nutshell, a scorecard is an automated tool that assesses a number of important security risks or signals with projects and their software development cycle, starting with projects that are natively developed on GitHub. We started on GitHub, but we're trying to expand to other providers, Bitbucket and GitLab, but today we currently only support GitHub. So scorecard tries to detect good practices that are followed by a project. Things like, are you using fuzzing, static analysis and continuous testing? Those are important signals, because it tells you about the quality of the code and whether bugs are likely to be caught early on in the development cycle. Scorecard will look at other things such as, do you have a tool to manage and update your dependencies? And will warn you if you don't. Scorecard was way beyond that though and it also checks for some of your settings on GitHub to make sure you've enabled safe settings. For example, we will check for the so-called branch protection setting, which prevents an attacker from pushing directly to your repository if you have a malicious contributor, for example. We also check for, say, good configuration of webhooks. We verify that you have enabled authentication, for example. And we have about 18 checks today that you can look at the documentation online. Besides good practices, Scorecard also tries to flag anti-patterns. So let's say if you have a secret, some secrets available in a pull request, Scorecard will warn you. Scorecard will also warn you if you have over-privileged CI runs, which is really important because, as I said earlier, an attacker could try to abuse those permissions bypass code reviews and push to your repository. And we will see a demo of this later in the talk. There are two main use cases that we think Scorecard is useful for. First is for projects that you own. You can install it as a GitHub action and it will alert and flag all the sort of things that I just talked about. And then if you have dependencies, the second use case is to run Scorecard against your dependencies and learn kind of the risks that you're taking by using them. So for example, if you have a dependency that hasn't seen any activity for a couple of months, Scorecard may warn you that maybe your dependency is not maintained and is unlikely to receive security updates in the future. So with that, let's take a demo on the first use case, how you can install Scorecard on your own GitHub repository. So let me quickly switch here. You can see, yeah. So this is just an example repository. To install Scorecard is pretty simple. You can click on the GitHub action, sorry, the action tab at the top and then select, sorry, new workflow on the right. Scorecard is integrated with GitHub so you can search for it right in the UI and then simply click configure. That will automatically create a new file, scorecard.yaml, and its content is auto-populated. As you can see, here, Scorecard runs on a schedule every day and also on every push to the main branch. And as soon as Scorecard runs, it will start populating alerts in the code scanning dashboard. We'll take a look at this in a second. Something I wanna mention is that Scorecard takes an optional PAT token. You don't have to use it, but be aware that if you don't use a PAT token, some of the checks like the branch protection check will be disabled. And this is due to a current limitation of the GitHub APIs. We hope to resolve this soon by working with GitHub. So all you do at this point is just commit and push this to your repository. I'm going to cancel the changes because I have already installed Scorecard. And we can take a look at some of the results that have been populated. So let's look at this one, fuzzing. Not surprisingly, this one is just, Scorecard is complaining that the project is not being fuzzed as part of CI, which is correct. Something else you can see is branch protection. Here, Scorecard is simply complaining that we haven't enabled branch protection on the repository. And then we have all sorts of other checks that you can try to look into on your own. All right, so let's go back to the main slide. All right, so that's the part of the talk where you are starting to get involved. So we've set up a few, so that is quiz number one. It's a GitHub workflow. As you can see at the top, it's running on every push to the main branch. And it's checking out the repo, so basically cloning the repo, and then it uses this, an action called do slash some work. So I'll pause for 30 seconds and see if someone in the audience wants to get a shot. So does anyone have any ideas about what's wrong with this workflow? Yeah, over there. He's an insider. Go ahead, it's fine. Yeah, so what he said is that the permissions are not set. By the way, this is Varun. He's working with us on, he doesn't work on Scorecard, but he helps us, he has a tool to remediate those kinds of problems that we recommend as part of Scorecard. So like he said, the workflows by default on GitHub have right permissions, except pull requests. So all you need to know, all you need to do is to follow the least privilege principle is to add these permissions read all at the top. And then if any of those actions that you're calling in the workflow, one day it becomes malicious, then at least you'll be protected and they will not be able to do anything really terrible. All right, quiz number two. So this one is a little trickier. So let me walk you through what it's actually doing. It's working, it's running on pull request. Every time the pull request comes in, this code will run. And I'm skipping the thing in between, like a magician basically, trying to not show you what's happening. But basically what's happening, we're just echoing to the log the title of the pull request. So it looks pretty innocuous. So the question is why is this vulnerable? Yeah, yeah, so I repeat what he said. He basically said that there's a script injection vulnerability. So what's happening is when GitHub runs the script, it substitutes this entire variable, the one that's in between the dollar sign and the double bracket. It replaces that string with the value provided by the person, by the title of the pull request. And that basically allows someone who submits a pull request to change the code and achieve remote code execution. So the fix is to declare an environment variable and use it inside of your script. This way there's no substitution happening. Title is just resolved by the shell. So that's where some of you will come in again. So if you click on the link, it's a repository with the exact same workflow. It's vulnerable and if you exploit it by the end of this talk, we'll hand over another T-shirt. So yeah, good luck. Try to still pay attention to the talk. All right, so demo time. How many of you are aware of Devs at Dev? Cool. Devs at Dev is a cool tool. It gives you open source insights of what and all the open source packages that you have. It shows what it has dependencies and what are the other things that depends on that. Why am I talking about this? I will showcase a demo of utilizing this. Another critical thing that Devs at Dev provides also is scorecard data on the right side. You can see it gives all those checks and gives whether any of the open source packages whether or how do they score against using scorecard score. Like what Lauren mentioned, scorecard runs this for a million repositories every week and you don't have to, one of the easiest way if anybody can look at score for any of the open source packages is using Devs at Dev. So I'm gonna switch to a terminal and run something. In this demo, I'm gonna showcase how to use scorecard CLI. I'm gonna give shout out to Sasha where I'll build this package called demo. I'm not gonna type and talk because I'm gonna fumble. I'm gonna, so that's why I'm gonna feel like a superhero but I'm actually gonna be running the commands but not typing for that. How to install scorecard, scorecard CLI. If the easiest three ways are brew install, if you're a next shell fan like I am, you can do that. And the last one is go to scorecard, get a release page and you should be able to download and install. Now that I've installed scorecard, what's the best way to start? The best way to start is scorecard dash dash help. This will essentially give you all the things scorecard can do. It can be overwhelming initially to see all those things. The most important thing is the checks. Like what Lauren mentioned, there are 18 different checks that scorecard does. And all of this is using open data that's available there. No, we're not using any private APIs or any of that. All of this is samples as just openly publicly available. Scorecard action in turn uses the scorecard CLI engine for doing all of that. Everything is only based on this. So like I mentioned, actually those comments I need to type. A fumble and that's why I made that automated. So the easiest thing is now, let's use scorecard to check scorecard, especially on one check called branch prediction. We're gonna keep talking about branch prediction pretty often because that is one thing if you can walk out of this talk and say please go enable branch prediction, it'll probably solve a lot of problems. We'll talk about it, why it does. So right now it's running the branch prediction check and that comes back with the result. This is how it starts with the results says what's the aggregate score? The score of eight was a 10. The name, the reason why it gave that. And if you need any documentation, the mediation, all that is available in the scorecard repo. Okay, this is great, but why we didn't get 10? Can scorecard help us identify why we didn't get 10 or why we got eight? Yes. There's another flag that you append to the scorecard client dash dash show details. And it should essentially give you that result specifically. And this is that result which I tag along show details and comes back and clearly says, hey, all the results we perfect with either info or warning. So here right now it's coming back and giving the result saying, hey, everything is great, but you have only one required reviewer to merge domain. So that's a reason it's not be given 10 out of 10. Okay, this is cool. Can I get some JSON love? Because everybody wants JSON, right? So you tag along format output JSON. There you go. This should give you scorecard details in JSON format.  So you can essentially get a, you can store this data into wherever you want. And this data is where this data comes into the BigQuery which I'll demonstrate later, which will clearly show on every what commit shots that scorecard ran against those million repositories and you'll be able to utilize that information for what you need to. Next question, somebody asked me yesterday in another talk, does scorecard support non-github repos? Yes, but not with all the checks. Scorecard has an option called dash dash local and you can point to a local repository. It could be a GitLab, private repositories, any of the repositories and you should be able to run against those repositories. But it'll not be able to provide all the checks. It'll not be able to give you all the checks, can only run few checks on that. That's about scorecard fly to start up. The next thing that I wanna demonstrate right now is the BigQuery weekly table scans. How many of you are aware of BigQuery weekly table scans? Okay. The BigQuery, there's a scorecard once about a million repositories, checks for a million repositories and stores them in BigQuery. And so anybody should be, and it's public data, all this data is public data, anybody can go utilize that and run that. So it runs for a million repositories and stores that. Like I was demonstrating about depth to dev. Depth to dev also, depth to dev parses the dependencies and stores that in another BigQuery table. So in this demo, I'm gonna take the depth to dev data and just take the scorecard data, mishmash it and get some details out of that. I'm gonna cheat right now because running this query takes a little while. So this is the big query for getting dependencies of scorecard. So what I'm gonna do right now is I'm gonna get all of scorecard dependencies and use that dependencies to go run one check against the BigQuery for scorecard itself. So you'll see more over here right now. So I'm saying, hey, get me all scorecard dependencies from depth to dev and take that data and go run against BigQuery just for branch prediction, especially branch prediction that have force push enabled. How many of you know what is force push enabled? So I can force push to master without any checks. So scorecards has about 160 dependencies of that 124 of them have force push enabled. So that means any of scorecard dependencies can be compromised. So this is something that is not good. And all of this data is available in BigQuery. This is great. Wouldn't it be nice to know how were the dependencies doing in these last three months? The more I've been telling that the BigQuery has been running for a while, I'm not wrong, almost eight and about 10 to 12 months. So I can go back in time and say, hey, please give me how was the state three months before? Again, we're just doing for branch prediction and for force push enabled. It's not that things went good to bad. We didn't scan those additional repositories. That's why it was 54, went to 124. That's the demo. We wanna talk next. You don't have to be independent. Oh, thank you. Another, I completely forgot. Thanks, thanks, thanks, Lauren. One of the things that I also wanna talk about is, pin dependencies. How many of you are aware of what pin dependencies are? Great. So this is a GitHub action, it's a sample demo action. When I took the sample code from GitHub and ran, built this project. So in this specifically, you can see I am running my own action that I'm utilizing and I've tagged it to V1. What are the problem with V1 tagging? You can move git tags. So, sorry? Yes, it is mutable. So when I run this action, anytime I push, push, this is what once. It says, hello, Mona, the Octocat. That's what it, it's a sample, simple demo. So now I'm a malicious, a malicious player, which says all your data belongs to me. What can happen? This is a malicious user, force pushing a different branch with the same git tag. And let's see what happens. I'm gonna force push another branch called navin slash debased. I'm gonna force push that with a different, with the same tag and see what happens. So with that one pushed, now if I go back to my action, no, my action is running again. And as for us, everybody else, it was the same V1, nothing changed. Let it run, if people can see this, now I was able to push something to debased.com. Nothing else changed.  Scorecard specifically wants about those things. Good, okay, cool. Thanks, Lauren, for reminding me. You wanna switch over? Yeah. All right, thanks for the demo. So to summarize what we've learned so far, declare your permissions, follow the principle of least privilege in your CEI, pin your dependencies, and install Scorecard to catch things like script injection. So let's go back to the quiz number two and see if someone has managed to exploit the repository. I suppose someone might have, I haven't checked. Okay, no one has. So let me give you a demo. So as you see, the latest commit on this repository dates back to around six hours ago. So let's create a pull request. Oh, someone has a pull request. Yeah, did it run? Yeah, I think it ran, yeah. No, it ran, it ran, it ran. Yeah, so actually it has permissions to write to the repository. So let me, yeah. Let me, let me just show you the demo. Let's create another pull request. It takes a little bit of work to make it, you know, in 15 minutes it's not easy, so. So let's create a new pull request and then let's give it this weird looking title. So the title looks very similar to what Jonathan did. We are escaping the quote from the echo command and then we're just downloading a payload and piping it into bash, all right? So let's go and see the action that is running. So you see here without intervention from the maintainers, the CI run, like the pre-submit ran. And here we can see in the logs that it said, you know, open source summit at North America was here. Okay, so we managed to run, to have code execution and let's see what we achieved. Now there's a new commit to this repository from 36 seconds ago, which basically exploited the repository just changing a file, all right? If you had a branch protection enabled, it would have been protected. So that's why it's so important that you enable it and you also reduce the permission that you use in your workflow. All right, so yeah, you can visit and try it for yourself. All right, so before we conclude, I wanted to point out to a few success stories that we've had with a scorecard. Some very popular projects have installed it, such as Envoy Proxy, Flutter, TensorFlow, and Codecov. I'd like to point everyone to this OpenSSF project, which I think was blocked about yesterday. It's called the sos.dev reward program. And if you improve or fix some of the scorecard alerts in critical repositories, you can be financially rewarded with $505 or even more if you do more work. We have plenty of stuff coming up. We have scorecard badges that we're working on. We're hoping to release in the next few months. We have a website coming up, and then we are currently working on a project to improve our GitHub action. With everything that Navin talked about, very soon you'll be able to see the results of scorecard for your dependencies. Every time there's a new pull request that changes or add a new dependency to your project, you'll be able to see the results and hopefully make some informed decision about whether you want to proceed or not proceed.  It's a companion app to scorecard. It allows you to enforce policies on some of the scorecard results at scale across an entire organization at GitHub. And then again, the OpenSSF community works in the open. You're highly encouraged to join the conversation if you're interested. And that's the end of our talk. Thanks for your attention. Thank you. Any question people may have? Yeah, yep. Yeah, we're always happy to take more heuristics. That's what we have right now, but if you have some good ones, we're happy to consider them and add them. A lot of people have actually suggested and contributed new checks. So yeah, we're super happy if you want to add some support for new checks. Adding to what Lauren mentioned, Webhook was something, another external contributor came in and said this would be an amazing great addition. Did that whole pull request and got that done. So yes, we are. Thank you. Yeah, someone over there. Sorry, can you repeat the first question a little louder? So you're talking about, are you saying you want some... There's a microphone behind you, just so that. Oh, yes. One question would be an option to remove, to know well, proactively remove tests which are not irrelevant for scorecard. You show an example in the demo that for example, fuzzing give you a negative score, but you're not using the fuzzing in the CI. This is a question, one. Second question is, is there any plans to look at the results? Not just what tests you do, but what are the scores? So for example, you can get a 10 out of 10 for scorecard, but all the results of the tests are terrible. Like everything is a critical one, critical stuff like that. So question number one was, or can we ignore some of the, so right now you cannot do that. We're working on a project where we're gonna give the results without the score, and we're gonna make that data available publicly. So you'll be able to look at those without the score, and so I guess maybe what you're asking is your own project, your score is going down because it's spotting, is it a consumer on the consumer side, or is it you as a maintainer? You'd like to remove some, you'd like to configure a scorecard so that it doesn't bombard you with alerts. Consumer side. Right, so for maintainers, yeah, we'd like to improve and make it more configurable. We haven't done it so. It's just, we try to sleep at night, so we haven't had the time to do it. But yes, it's something that we want to do, and on the consumer side, I think the results without the score would allow you to, we're planning on having a more structured output that you can pass yourself and take a better decision without looking at the score and decide whether something matters for your use case or not. So we're working on this, but it's, I think this year we should have a prototype for it. Yep, yep, and also another one thing that like what Lauren mentioned, if you're a consumer you can, you don't have to pick all the checks. You can pick what checks are critical for you and decide which ones to run. Again, because not every repository requires a fuzzing check. So you can, on that specific, you can decide which ones you want to run against and get those results. And like what Lauren mentioned, then later with that, be able to determine, especially with the raw data, it's talking about the raw data without scores and able to determine what's the risk factor on consuming those dependencies. Yeah, we'd like also to support annotations. So if a developer says, I don't care, I don't have, you know, this check is failing, but this is the reason why we'd like the consumer to be able to see that. So that's something we're thinking about, but we don't have it implemented yet. So yeah, so I think for the second question, sorry, I forgot, what was it? Good results of fuzzing, per se, or? No, so right now the check of fuzzing just checks if you use fuzzing in your CI, let's say you have OSS fuzz or you have Golang built-in fuzzing, then we'll give you a score of 10. We say this is good enough. And we're adding support for more languages. We started with the generic OSS fuzz and CI fuzz, and now we have added support for Go, built-in fuzzing. I mean, I think they came out with a, Golang has support for fuzzing right now, so we can use it and we detect that. And we're planning on adding support for other languages. Yeah, which one? Yeah, so I think the question was, does scorecard differentiate between adversarial inputs today and not really? We don't. Yeah, so this is something that has been in the back of our mind, but we haven't really considered it that much. I think as a scorecard grows and gain popularity, those questions are gonna come up more and more. So I think we have to think more about it, but yeah, to answer your question, we don't have, I guess what you're asking is, you'd like us to be able to at least label some of the check and say, this can be the repository can change it or not change it. And scorecard also clearly says which are critical, which are high, which are low right now. So you can, because we think based on these data that these are really critical, like having dangerous workflow is a good compromise. So giving you an example on that. I think we gotta stop right now. Yeah, sorry about that. Sorry, we'll take that question offline. Yeah, thanks everyone. Oh yeah, we can have three shots. If you answered some of the questions, come and get your shot.


Hi Charles, my name is Ashley and I work in the internet department here at Augsmore, Toyota. It has come to my attention that you are interested in a new 2016 Sienna. I went ahead and sent you over our inventory of Sienna for you to look over at your convenience. When you get a moment, please give me a call. My number is<PHONE_NUMBER> or you can also email me back. Thanks. Bye.


What's up everybody, Jason here with jaspyscasebreak.com. We finally filled it guys, Jaspys 16 box MVP mixer, Radham teams We're giving away a Yellich game use brewers hat giveaway and as you can see guys The Milwaukee Brewers Christian Yellich hat is a turn of the back turn back the clock game worn hat MLB authentication and photo matched. We'll show you guys a little bit more in-depth What is right here? The COA so actually a war a guy's which is pretty pretty crazy Game worn hat, so we'll give that away at the end of the break everybody has a shot at that We'll randomize customer names and teams and then we have the box list from here from two boxes of prism 2020 all the way down A contenders draft picks 2019 16 boxes guys. It's a great mixer. I'll hopefully pull so much fire and again no veteran base slash Papership from like Bowman Breaks there, so good luck. Let's do the break first Let's roll the dice and we got a three and a four for lucky number seven times Customers bottom straight up and if not you have a little flash three four five six and seven attached to your name So good luck, and then you're the teams so seven times one two three four five six and seven I'm covering down to Laura seven times one two three four five six and seven Giants down to the pirates So Adam with the Giants right Brian crouch with the Dodgers John with the Orioles Cohen with the Tigers Jimmy of the Rangers Hip with the Mariners Andrew with the Nationals Pasquale of the Astros Chris with the Angels Adam with the Blue Jays Kyle Jack with the A's Matthew with the White Sox Chris You have the Indians last ball Maljo Adam with the Marlins Nick with the Royals Emery with the Diamondbacks Tony with the Padres Debra with the Cubs Ben with the Red Sox Christopher Gerard with the Reds Jacob with the Mets Sethia the Phillies Adam with the twins Jimmy of the Rays Patrick With the Yankees Emery with the Brewers Ben with the Cardinals Tyler the Braves Steve of the Rockies and Laura with the Pirates There you go Anybody want to trade this is the time? I'm just gonna start ripping some packs. I'm gonna turn on Nick camera really quick That way we can get that rolling. I'll keep the trade window open for a couple minutes Just because it is a big mixer and let me tell Oliver gets the A's and A's in the museum So there's all the boxes right here as you can see Have them all laid out Like I said, I'll give you guys a couple minutes. Let's just get this gold rush right here out of the way They'll look they're gonna be some nice stuff in here, but Start opening up those packs and this gold rush is a repack product So it'll go to whatever team is represented on the card and if it's a non-license MLB product then like a domestic product or something and it'll go to The team represented the part still but if not Teams are there currently on our teams are they are feeling it's like a prospect or something and you get three Cards per pack. I said sometimes pretty solid stuff. You're gonna be in a whole box like this. There's usually one huge like huge hit usually Something like this. Let's see what hit we get out of here For you guys There's a lot of greater cards in these two, so that's actually pretty cool All right guys by the time I finish this box if there's no trace guys I'm just gonna close it and then we also go by checklist to first if there is and Scott gets the brewers brewers in museum collection Hey guys, you guys want to do museum collection to before I head off. I don't mind still doing those fresh half cases right there and Museum collection is moving rather quickly guys museum collections already down to 16 and 16 so You guys want to give it a shot so again, this is the repack product guys from Gold Rush Yeah team that is affiliated with on the card If it's a non-licensed product They're currently on He's a put it for the longest if there isn't like a team affiliation on the back So first it is Zach Thompson That is a bowman's first autograph to 499 for the Cardinals We have a showtime graded 9 from Beckett 2019 Donner's nicknames pink fireworks show hey old honey And we got it looks like from transcended to 100 sketch Harold Baines for the Chicago White Sox And we got a Owen Miller to 99 shimmer for the Padres Eddie Murray for the Orioles We got a Jack Harshman authentic autograph from 1958 tops Baltimore Orals We have a Billy Hamilton relic for the Cincinnati Reds Jose Altuve nine Beckett VGS Donner's optic mythical. That's for the Houston Astros We got a transcendent frame Miguel Cabrera number to 87 for the Tigers We got a Frank de Pino That is for the St. Louis Cardinals and then we got a rookie and yell Los Santos rookie out of 10 for the Philadelphia Phillies We got a mini 8-5 of Johan Moncada 2017 Bowman Chrome mini Chicago White Sox Edition Then we got a autograph here of Rob Robert Ruff Schneider 299 that's collegiate so we'll go by checklist, but I believe that would still be Yankees at the time But we can double check that the end of the break We got a Aaron Judge mini 8-5 Bowman Chrome for the Yankees We got a quad relic here that has Carson Fulmer, Rhett Wiseman, Dan Zbiswanson and Walker Bueller So that'll be a 14 randomizer at the end of the break. So we'll put that somewhere over here Tops now Aquino Cincinnati Reds We got a Bryce Harper BGS 9 rainbow foil from Tops Update 2018 for the Phillies We got a David Thompson nice little patch here four out of ten collegiate Right there. We'll make sure to see who that goes to as well. We got a Matt Young for the Seattle Mariners We got a pre-production proof of Juan Pablo Martinez from Leif Flash Baseball A little one-on-one again. We'll check that out see who that goes to And we got a 5 star to 50 Josh James Houston Astros Rhonda Cooner Jr. Rookie for the Braves tops now rookie Peter Lanzo We got a Evan Longoria for the Tampa Bay Rays 9 5 10 gold from 2014 Panini Prism 9 out of 10 That's pretty cool. Ray is going to Jim Warren We got a Dylan Cease to 99 for the Sox Pre-production proof of Jordan Adams Clear orange. We got a quad rally here all for the Padres Eric Cosmere, Will Myers, Ian Kinzler, and Louisa Rice David Fletcher, Rookie Autograph for the A's. I mean sorry for the Angels and a pre-production proof of Steel Walker 101 We got a booklet here of Andrew McCutcheon for Pittsburgh. I don't know what this looks like. Let's find out Julia Schill for the Yankees Ozzie Smith for the Padres We got a 9-10 auto of James Shield to the Kansas City Royals We have a rookie autograph of Justice Sheffield to 200 for the Yankees Oh, is that a Clint Kershaw? It's a fact similarly though, right? Clint Kershaw Rookie from Bowman For the Dodgers. It's pretty cool. And the last one is another Lee Valiant This is Ryan Rollison 101 pre-production proof So again, this whole stack here, we'll check and see who these teams, who these players play for are currently, especially since they will check us by for these leaf ones And then we'll see who these players are under the checklist if there is one. If not, it'll be a team that they're currently on And there you go. There's the first round right there of the mixer So that's a little buyback product from From Goldrush and then this will be a randomizer too, so we'll leave that up here. Ranger, what's going on? Let's go with some more of the heavy-based products now 19 Donner's Megabox Losiega We got a Dakota Hudson rater rookie We got a Paul Goldschmidt Goldie Riz Chance Adams No more Rosara Alex Bregman dominators Danny Janssen and a Jacob Knicks signature series autograph That'll be going to the San Diego Padres Mookie Betts Bryson Stocks Ronald Acuna Jr Bryce Harper Dennis Antana Christian Stewart rater rookie Another Mookie Betts there for the Red Sox Mimichato Mtoogie Tuzans Freeland Full Holes Trevor Storty Here's the next box 2020 Bowman hobby. Can we get a Jason Dominguez rookie in here? Good luck to Patrick and the Yankees But of course there's other teams like the Royals Nick, that are pretty solid in here and Poston with the A's, Kyle Jack So that's Aux, Ben Cubs have some good ones here Debra See if we can get out of here Are there some games earlier? There's only four games on today First thing's clear, the Portland Trailblazers are that one All right, good luck guys Gavin Lux Some paper And we got a speckle, Cooper Johnson Autograph for the Detroit Tigers Cohen Yes Bowman is usually on card Postal Josh Jung The one per box, unnumbered atomic Bobby Witt Jr You guys see the 101 On auction of Bobby Witt 9510 The Superfactor is already at like $50,000 All right, let's go to top series one 2019 There's an open foil 50-stamps solar Anderson I don't know, major big rookies in this Is it update that has all the big rookies? I think it's all right Trout Luis Uraes There's the relic Miguel Cabrera The foil stamps to $150 Golden Golds All right, 2017 Bowman draft is going to be crazy That's a nice Bowman draft here right there Keiston first Cannon Smith Delbeck Delvy Perez Number 399 Mincer Proverbs in there in the description as well Red Sox Purple It's a 250 It's going to Ben I don't know those Gavin sheets For the White Sox David Thompson to $150 All right guys a couple more packs There's one more auto hiding in here I believe Those first For the D-backs Robinson Purple All right next one select All right guys good luck Select 5, look at that Joe Palombo For Texas It's going to Jim 3 out of 5 For Tampa Bay Back-to-back hits for Jim We got a red Mike Mostakis To $199 Old Altuve Silver Cavendizio to $99 Green Oveshev Rookie Tony Gonsolin And we got Yu Chang A little jumbo relic there Cleveland Chris Aurelix Tubbs Hendricks to $50 Ronald Bolognails Elon Heminis Silver All right let's go with Contenders Draftpicks Adley Retchman the big chase here too Manoa Draftpicks variation Autograph we'll take a look at that see who that goes to And we got To number to $99 draft ticket Cameron Minzner And we got JT Sakema To $99 We'll take a look at the checklist as well At the end of the break We got Martinez She actually says Toronto to it So some of these will actually have the Teen that they're on Gunner Henderson To $99 Last one here We actually got Adley Rushman 8 out of 23 cracked ice School colors 8 out of 23 That's awesome Aurel's going to John Stallone That's pretty nice So again we'll look for these autos And see who they go to if they don't have the teen Affiliated on there So I'll put all of these to the side And see who these go to at the end of the break Let's stack over there Let's do a couple of prism boxes One, two, three, four Five, six, seven, eight more boxes guys So we're like halfway through But once we get done with this prism And Top's Chrome The finest box should be pretty quick Absolute really quick inception The baseball and museum collection I'd say I know that 20, 25 minutes tops I'll be done maybe around 12, 15 or so And again guys I'm here One ish till Joe gets here We can squeeze in something I can pop when he gets here I could probably break it for you guys And then he'll hop on And I believe he'll be here till 9 And then Sean comes in later today And he'll be breaking super late for you guys as well Good luck guys Kikuchi To 35 You guys to know My most obvious Gregory Springer Or George Springer sorry We got Brian Anderson to 60 Propendorship Philadelphia Mike Trout fireworks Colton Wong Ooh a little snakeskin 31 out of 50 Yohan Moncada Eliminators AJ Puck for Oakland Kyle Jack Luis Robert Rookie Fernando Taziz Eliminator Elimination sorry Justin Dunn Showtime 175 Shohei Otani Jose El Tuve Gavin Lux Dodgers Javier Baez and Michael Baez To Javier Baez for 149 For the Cubs And San Diego Michael Baez to 99 Tran Grisham Yosmani Grindal 199 Keiston The museum collection picker team is moving pretty smoothly Jordan Yamamoto 100 Out of 5 Shane Beaver Cleveland Chris Maxwell Looking out of 5 right there Gavin Lux Garrett Cole Lucas Jolito to 50 Edwin Rios Rookie Autograph for LA Brian Crouch Jordan Yamamoto Silver Rookie Little Bichette Blue Jacob DeGrom Little Checkerboard to 75 Brandon McKay for Tampa Bay Plain Cursall to 175 Bryce Harper Eliminator for Elimination sorry Jeff McNeil Eddie Rosario Nelson Cruz Go on with them Alrighty guys Here we go let's go with some Finest And then 2017 Top's Chrome is Cody Ballinger Rookie Year Aaron Judge, Andrew Ben Attendee Pretty strong year so we'll be after these And then after that guys Absolute, Little Autograph Baseball Museum Collection, Clearly Authentic Box Inception Baseball I said that we'll be done with the mixer This is 2020 Finest Very popular product Really quickly on Instagram When we release these we do a couple cases Take your teams for a little crazy as well Finest Baseball always seems to be a Really really good product over here We'll do that Top's Emblem and Chrome type of feel But with less cards All Chrome And you get like two Autolabel even too many Boxes right so for a total four in a box Pretty solid Alright good luck guys To 100 Oh sorry Lorenzo Kane why does he say Robinson could not That's to 150 actually Actually maybe it's one per mini box actually And it looks like it is Jorge Soler It's the Autograph Come to the Royals with that one That is a Nick Mike Trout Go to the Shed Rookie Gavin Lux Bryce Harper Purple to 250 Do we have an Orange Auto coming out Or Yamamoto For the Marlins And it is Orange out of 25 Adam Harper with the Marlins 19 bonus guys good luck Some good stuff always hiding in here We got a Fernando Tautiz And Togelea Parval de Rockies Trout Refractor Tautiz number to 250 Austin Riley We got a Gold Ronald Aquini Jr. to 50 Josh Jones Atomic Elon's a Rookie And Dustin May Little Prospect Auto Dodgers Last Auto here And Andrew Vaughn very nice to green Tonight we have another good prospect for the socks Matthew Hedges The White Sox Top's Chrome Five Autographs guys Five Autos It's got a Belly Or a Judge Rookie Autograph Billy Hamilton, Jackie Bradley Jr. And Renato Nunez Renato Nunez Yuleski, Jury L Dansby Swanson It's a Rookie Refractor as well Autograph for Houston Cody Ballinger Rookie Excellent Who do we have? Two Autos now Also guys I believe Top's Chrome is coming out this next week Sean Newcombe So you guys like the way the Top's Chrome looks Top's Chrome is coming out this next Friday or Wednesday Which I'll be live for those early release days too Nice little Aaron Judge Little Retro Billy Hamilton And a negative Robinson Pinot I think we have one Or maybe two Autos left Francisco Lindor Cal Seager It's a blue Number to 75 Come on Belly Fire Rookie And Ty Blash Yeah it's not really a red it's just a Freshman Flash or something like that There's no numberings to those If there is it'll be a different color Kenta Maeda Strom And Alex Bregman I forgot who's a rookie and it's too purple Our last two packs Rich Springer and Jason Hirsch I think we've pulled all the autographs already David Dill I can just topple these guys And we have Five Boxes left guys Absolute Baseball Box still there We've got Inception Box Clearly Authentic I believe Baseball and Museum Collection Is what we'll end off with I'm not sure Tyler I'll do a little Hit Recap at the end of the video We did get a couple Swanson Rookies in that last one I think one of them might be a Parallel too Maybe we still have this stack right here That's unclaimed and see who that goes to Too as well Unsung Hero Johann Moncada And Kershaw Jordan Alvarez Relic Rookie Material Brandon Brayu A little Jersey Auto For Houston, Back to Back Hits And Dylan Cease For the Chicago White Sox I think he's over 130 Otani to 99 Tools of the Game Abraham Toro Quadralic That's going to go to Where's Abraham at? Houston Well there's an Atlanta Bray for you Dale Murphy Absolute Heroes and that's an 8 out of 10 Little Jersey Auto And Josh Rojas 7 out of 10 For the Arizona Carnals Or sorry, Arizona Diamond Maximum Alright let's go with Inception 2018 Inception Ronald Acuna Rookie here with Shohail Tani I believe Devers I think is the Rookie in this Donaldson Francisco Mejia We're at 150 Noah Cindergarde and behind that 3 Albies At a 230 Another one for the Braves Clearly authentic We got ourselves A Miguel Cabrera Autograph And that's 4 out of 5 They put 04 out of 05 That's an out of 5 Miguel Cabrera on card Decades Best 2010s for Detroit And that is going to come Very nice Let's go to the Autograph Baseball now And we'll leave Museum for last Because that's a pretty fun product to rip And it's also 2018 And just because I want to see if we can squeeze in Something else potentially Just skip a little Who am I See if someone could just guess the auto So there's the Beckett COA And whose auto is this guys Anybody guess the auto This is the Beckett COA I'll tell you He was a pitcher for the Mets But not Mets actually Well yes Let me see really quick Now I'm getting confused myself No sorry I meant to say Braves And did play Oh he did play for the Mets actually And yes It is Tom Glavin And again out of the team he's currently on Or team he played for the longest And as you can see he played with the Braves From 1987 to 2002 From 2003 to 2007 And got back to the Braves So by baseball reference Played with the Braves for 15 seasons Oh no 17 years Sorry and then the Mets 5 years So there you go Atlanta Braves Damn Tyler heating up dog Heating up Hey is there any Braves in there And all of a sudden all these Braves come out Alright here we go Again we have 2020 museum collection On the website this is 2018 Paul Goldsmith to 150 That's right Drafted by the Kings I believe Same year as Luke Robocite Although Luke Robocite was drafted in the 9th round Isn't that funny Both had Hall of Fame careers right Alright Cessbittas Ty Cobb Two hits in this box 20 out of 75 George Springer Houston Astros Pasquale with that one We got a patch here of Kyle Seeger To 25 130s We have Trey Mancini Nice look at that Jumbo patch To 15 I think that might have been the case it right there There you go Robinson Cannell For the Mariners To 99 Relic And we got a dual patch From Yachty Molina To 35 Alright let me get to 130 Alright guys so before I do the randomizer For the Yeli hat Right here Let's see Who these ones go to So man we're going to go by Checklist first If there's a checklist on group red checklist For the collegiate stuff We will go by that if not go to the teams that they're currently on And then of course We'll go by Checklist as well so that's one product This is one product And these right here will go to the checklist if there is one And then these ones I don't think they created a checklist for Leaf Valiant But I can double check and flash But these will probably go to the teams that they're currently on as well So Let's quickly switch scenes And let's see what we can find here So we got 2019 Contenders draft picks Okay so there is a checklist it looks like What we got here Alright cool So Alec Minoa Will be for the blue jays So this is a little hit more For the blue jays The redemption Next one will be Gunnar Henderson Will be for the Orioles Next one will be Cameron Minstner Will be Marlins Raised for John Oh TJ Sakema I think should be I think he does play for the Tigers but he went to Mizzou No Yankees actually What am I thinking? Alright cool now let's go to 2015 Immaculate Collegiate Multisport I believe that one is there Or NT might be only These are all 2015 national Multisport So there is one here So we'll see who these go to right now I need the Team checklist Okay so this is just football I don't know they do have here So Yankees For that rubber rough sniper From Arizona And then the patch auto of David Thompson Will be for Oh wait that's not it Okay maybe they don't have it There's two David Thompson's I don't see his name there Let's see if they might have it here Yeah it just has Miami Hurricanes Alright so let's quickly just look at this guy really quick David Thompson baseball Went to Miami And it looks like With the New York Mets Minor League Affiliate right now so we'll give that to the Mets There you go Jacob little patch auto right there And then this Quad Relic I think it's Immaculate Yeah this Quad Relic is Immaculate Which we can't go by NT Because they're sometimes a little different So let's see I always try to look at Immaculate But I think group break checklist only made one for NT If they don't have it then we're just going to have to go to the teams that they're currently on Yeah see they only have College multi sport for NT So teams that they're currently on So Carson Fulmer right now That'll be for The Detroit Tigers So Tigers Will be part of that randomizer Dan B. Swanson Where's he at right now He is currently on the Atlanta Braves Walker Bueller That's Dodgers And the fourth player would be Brett Wiseman Which is for the Washington Nationals It looks like So as we've called randomizer we'll do that right now at the end With the Yelly And I don't think they're going to have any Checklists here for Leigh Valiant Baseball 2018 but I can double check Just to rule that out But if not these will have to go to the teams that they're currently on Okay guys, oh actually I do have one Okay so let's see what Ryan Roll is someplace for That's just a Celsius Maybe this Roll is sent as Rockies So Rocky's got a little extra hit right there Let's go to Leigh Flash now I think these are all 2018 flash So again if they have flashed Then we'll use it if not And they do, cool Alright first player is Steel Walker, White Sox White Sox, there you go Next one is Jordan Adams Will be Angels And the last one here guys Is going to be Juan Pablo Martinez Or Julio Pablo And that'll be Rangers Awesome Alright so just that quad randomizer And then the Yelly hat Let me just quickly show you guys the Yelly hat before I randomize it So Here you go Here's the hat itself So there's the hat It does come with a hologram sticker It's used He wears a size 7 and 1 4th And There's like a little CoA kind of thing Printed out game use cap There's a hologram And it was used Pittsburgh Pirates at Milwaukee Brewers June 7th, 2009 So And I'm sure you can find photos of that game I don't know if there's no any DNA I'm sure they clean out this hat I don't know if they want people to have Yellich's DNA There you go guys There was the mixer a little longer than I expected But a lot of randomizing to do and stuff So there you go And again good luck everybody can have a shot at this Whether you bop straight up or you want to spot In the mixer Through pack bricks Alright so the first randomizer We'll use the same dice roll it'll be for that quad Relics which would be Tigers Braves Dodgers And Nationals And then Everybody would have a shot at this So let's actually grab the original List of customer names There's the original list Good luck guys use the same dice roller Roll it 2 out of 2 4 times good luck For the quad first 1 2 4 Tigers at the top So the Detroit Tigers are going to win that relic Nice And good luck 4 times now On customer names 1 2 3 4 4 times 1 gets it It is going to be Tyler Wow what a break for you dude Jesus After 4 times Can you believe that you're at number 1 Boom yeli hats going your way And there you go guys that was the 16 box MVP mixer where we're giving away the Yelich game use hat Appreciate it guys Jaspyscasebricks.com


So we had two slanted approaches to the whole thing. One thing was going one by one in our heads through the pipeline and figured what are the things that just keep us from just having a very streamlined dissemination process of that. So the chronological order and there we just figured that there's things like structure repair as we call it, that is all kinds of things like adding hydrogen bonds, adding pines into places. It's not very sanitized. It's not very easy to describe. It's not very easy to follow through. So that is something we would like to see taken action on. We figured it's relatively difficult just because there's so many different ways of fixing up things and not very good dissemination. So do you envision just one tool that everybody would use or a common way of many tools being able to be used on the same thing? No, I think this goes back to everything we had today that it's mostly about ontology. It's not much about how you implement things. Do we have a way to say that I added hydrogen over here? Do we have a way of saying, OK, the structure was incomplete but I did these few fix-ups and then we shouldn't care about what tool does the actual job. So is it ontology for operations you performed on it? Yes, exactly. You would presumably also need a common format for representing the thing that has whatever items you already put in it, right? Yeah. I mean, we didn't discuss that so much. I would still prefer to say structure repair. Kind of the... I'm just taking notes. The way of describing what you've done is the first thing to do. This is my group. Can we use the whole spectrum? Is there anything that's always left on a part of the video? No, no, no. But there's the beer. Yeah. We're okay to share. OK, I just move it a bit. Yeah, as I see it. Oh, no, also there. Yeah, how about beer, man? I would just keep it in whichever... Yeah, that's easy to get. See, you had it. But pretty important. Then there's another thing. OK, that's it. Then, following up on this, even before the liquid and molecule parameterization standards are lacking a bit, especially if you don't use a standard force field board, you slightly modify your force field also there. We miss even the whole language to describe these things and a way to make sure that also when we simulate the force field is actually what we think is happening. One case, for example, are things like changing ion committees would be true. Part of the community just did pre-default on Ember 99 because so in chloride was... Crystallizing. Crystallizing is that way. It's just nowhere documented. You just had either the patch or non-patch force field that is horrible for any kind of reproducibility or a smooth isolation workflow. So also that kind of thing that you say, OK, I parameterized my molecule. I used a certain force field. These things are described in a better way, a more sanitized way, and that we also start using databases with all legal parameters more concisely. So when you say the standards, do you mean a standard way of describing how you apply the force field parameters or a standard way of describing how you apply the parameters or what's the... The first and easiest thing I think would be to give you a source form so that we keep the source of the data and the model simulations so that we can switch over to what kind of tools we have for the setup already. And while we see that things we already do have to make things happen is something like a charm GUI for setting up simulations. We're setting up free energy calculations. There's, for example, something called PMX for the BIO setup framework. There's other things. For the workflows themselves, we have a BIO sim space called the workflow language being developed. We also have openMM, for example. These are all tools that are already in place. We have workflows. Also, our MD software packages themselves we have, and that is, I think, existing tools but at the same time also concerns and difficulties of the same posted because we see, and we just briefly check, we keep here there's a 60 or more than a 60. And these software packages we just listed a few where we thought, okay, these are kind of important and all the data we posted is filled so that makes it hard also to take action. If we want to describe workflows and streamline the simulation setup we would need some kind of comprehensive way of describing what these packages can do, what they require, what they need to do. So we would need some kind of way to listing commonalities between soft barriers to listing overlaps and the blah, it's an abaction, I think. Then just like this and also with the difficulty and concert comes all these different formats. We assume we have 60 different file formats that makes an end screen for intercom really, which we all define. So we have something that is like an end screen problem for a translation between file formats and that is hardly maintainable. Things will go off-scope, things will not be implemented properly, there will be a certain transformation, translation that will break so that is something we see as an example. Yeah, if you go always to one, exactly, but this is what we have, but we need some kind of a common object model to avoid an end-screen complexity. So that is like, as it actually we put it in here, say that you need that type of common object model so that you are sure you can do that all to one translation and you know that you have a one-to-one correspondence when you do the translations. So that goes on a different side-line but what we figure was extremely important is that we start to modulate our tools on the one hand so that no tool is magic to a whole tool on the one hand but then on the other hand we are able to layer our tools and mix them up accordingly that means we have a very clear distinction between tools that do transformations of input data that change input data and tools that can bundle functionality that is we put this under two key roots that is modularization on the one hand on the long side of that stratification so we have a clear layering of responsibilities I reckon ladies this is for to have a gift then along side with this modularization need and like this historical depth we see coming with these good old tools is something that is also again related to standardized structure repair these tools that try to be auto-magic and we have this this int graph that is thrown up here we see on the x-axis it's the magic and on the y-axis it's a catastrophic failure that is for a long time you can just do lots of magic do all the wizardry you'd like to do not much happens but then there's a critical transition and suddenly very likely you just see catastrophic failure if you cross this written magic boundary we have this happening in our efforts at the moment and yeah then just in our collection of tools there's two mentions to mention MB analysis and something we should use as an existing tool and anyone so like this ENI one that was brought up but maybe you can say there's a whole state of people that are fitting in high level quantum mechanics with machine learning models and so a lot of the new types of interesting potentials for small molecules or proteins are these machine learning potentials that can run in things like TensorFlow or PyTorch so the question of how do we make things interoperable with those kinds of neural network or deep learning based methods cool, alright thanks who's next let me go over here we'll start with you think it's the easy one that was pointed out to me I even get this wrong all the time step one it will help if our tools come with correct documentation which was tested live I've got the minus P8888 per my 1888 but actually documentation for our tools is generally rubbish because we all don't like writing documentation it happens after you've written the software and everything is cool related to that is we need to basically develop more training material to develop and share best practice so again one of the things that comes up are run-lead simulations but actually what is the right way to run it and the tool is not the place that defines the right way to run it it's the community that should define the right way to run it and when they do that we're going to enter a writing training material and sharing best practice which I think is a very important and maybe a little bit more difficult then do you envision that it's something a human would read and then somehow implement or it's something that would be a pipeline that someone could run like your example notebook I think I think humans will come up with the protocol but I don't know what the best way to represent them is because the protocol is more just the data it's actually executable as well it seems like a video workflow I mean you have a protocol as an object in your example is that what you're thinking about is that darkening these sorts of things or by running the protocols or do you mean like running down all of the steps specifically in the checklist so someone could come in and understand it and implement that which I would say the training materials which you can do is as training events if you do not have the training events you can run the MDM thing using those protocols I think they will eventually end up being things like objects because it's not just data executable as well which are objects that represent protocols and then having a mechanism where we can share objects which represent protocols and then if we then got into we could do this in UML we could do this in other things and that's really what we had next is make it easier to define and share protocols which is kind of I think important but heading towards more difficult what's really difficult but is really important is that actually we're really bad with our tools of telling people why they failed so typically when you run a parameterization and the parameterization doesn't work it crashes which is a really useless way of telling you it's failed well it might work but it's done something badly wrong like parameterize your NH2 group which is a bit dodgy just because you didn't convert or you didn't do your own calculation but hey as a conversion we'll still give you charges so we're really bad in workflows we're actually reporting errors and reporting errors in a way that we can catch them so we had some way of standardizing or at least documenting how the thing reports errors which is really nice quite difficult to find there's NH2 they're very bad and you only look at them and normally the PhD students will only look at them at the end when they make the figures of placement I've done that I think it's really important that we improve its probability and tools that really fits in with the modularization stratification ultimately we should move away from the culture of a PI and the group have their monolithic tool that does everything and we will all live in that world very lax, dandy, charm and blah blah blah and move to a world where actually you don't even know what software is being run because it's all hidden in a Python script you know all the algorithms should just be pick up or put down a ball and we should have some mechanism of tracking providence of code so that the person who wrote pandas can see that's having massive impact pandas written by one individual is being used by more scientists and researchers having more impact than any empty package and yet that person who wrote pandas is almost jobless now they've only just been picked up by non-focus and so we need ways of actually breaking this I am the person who wrote the software and making our tools into profitable and to get that you kind of need to have a culture change so we need to culture change into community I'd actually say we should get used to the interoperable tools we shouldn't be making a brand new tool to do something the owner of the software should not be the one who develops it we should actually be happy that if you start a piece of software you no longer work on it in the future but one of the things I say to people who want to write new software is before you start create your exit plan because you don't want to be developing and maintaining that software in five years time you don't want to be developing it when you're 60 so you start software what's your exit plan to be happy for their software to be developed by other people does this require the character or the stick? both and so I think there's one of the things if you're carrying a software strategy then we're very much going we want to make it profitable software we're not going to be going down the model we'll let it fruit anymore and here it is, we'll give you money if you're managing to make interoperable software and that's measured in practice during the and that's what we're trying to do but ultimately just writing calls saying the call is not write a new piece of software to do it the call is how do you make interoperable or how can you add the functionality to existing codes and the fact that we got by SimSpace funded which is ultimately just a load of file format converters that's one of the first things that came out of this we didn't produce new software it's just file format converters but about the stick the stick I think is there's a community which you actually shunt towards you know ultimately you should not we shouldn't have this balkanisation you know, as a PhD student if you're being told by your supervisor use this particular package and you will let everything in this one package so classic one there's no one of Ariov-Arshall's group here before I offend anyone I really like Ariov we get on relatively well the problem with Ariov-Arshall's group is obviously they have their own stack by stack for everything and if you've been in the balkal group you will be in that stack and you will never go out of that stack and you have this whole parallel molecular simulation world that's existing in this stack and it's fantastic because the methodology develops at 20 years beyond where we are obviously we don't know how they work yet but there's a whole career to be made deciphering the papers and translating but then just have to get all the understanding we implement all the software and actually fight back and say this tool is something which I should be able to do this in other tools I should be able to choose the tools I'm using and I should be pushing towards tools which are more open by design and actually making those decisions early on in the project so are you suggesting that people are starting to change and how power they're in it's quite important because ultimately the other thing that came before is like culture change you're being told by your supervisor to use a package because they used it when they were postdoc 20 years old already that's why in this field we are still doing the same things we were doing when I was a PhD student in the 90s we are still using text based interfaces and logging on with SSH even though we have graphics now why are we using text consoles when we have graphics there's firepots it's more expressive data sharing and cleanup we need to basically share our data because we need to actually engage in a process where we're cleaning up the tools and cleaning up the data we're producing because ultimately there is a reproducibility crisis because we're just running a simulation and publishing the result immediately if something interesting happened in the simulation and we won't be running it again in case it doesn't happen again in the simulation as I say in my software engineering part I try and get people to do testing and I get them to do testing by saying at the moment the way we do testing in science is you run your script and if it all looks right it worked and if it doesn't look right you fix the script until it worked and that's the way some scientists do testing but what we have is there are major major attractions now in nature and science and other things where scientists have had their careers destroyed because the software was wrong because it wasn't tested we need to get people to actually do unit testing regression testing, performance testing and actually make it so we can take all of this data into being shared and do something with it i.e. we run it with lots of different tools so we can actually see which tools work and which tools don't but then we can regression test as we develop the versions of tools I'm getting the thing we need to accept just the discussion I had just now and I apologize for this we have a fear of writing software that depends on other people's code but other people's code is a whole cesspit that I don't want anywhere near my code and we're terrified depending on other people's code so we end up putting everything into one monolithic thing and the discussion on having a single file format kind of revolved around we want a single file format that we can write so we don't have to depend on a third party to do it but we'll give the single file format out as a third party tool for other people to use and it's fundamental contradiction you have to accept that we're going to have dependencies to do any real work you're going to build on top of other people's code so we need dependencies we have to accept that I think it's quite difficult for people to accept it but it is an important thing to do but then when we accept we have to depend on other people's code and depend on other people's software dealing with dependency management is extremely difficult I hate dependency management I hate Conda now and Conda is was like you know from Star Wars episode 3 you were the chosen one and there was a meme with Conda for that because Conda was meant to solve all of our dependency management problems but when Conda installs something it breaks everything else and changes the version of Python for crying out loud but we need to fix this if we can in the field fix dependency management and actually all got together and said let's make proper Conda packages that actually respect dependencies and report the things they depend on and we actually curate a standard set so if BIO Conda actually worked it would make it so much easier because then I could do Conda install growmax Conda install lambda Conda install all of these tools and it would work, that would be my promised land that I would like to go to and that's why installing biosinspace is such a pain in the neck because what we had to do is take Conda install everything into Conda lock the versions down and then we create a binary package and load it and then you pack it on your machine and you never run Conda update or Conda install in that package ever again I'm going to volunteer mostly to help out on this because I know that my former post-doctor Levi Eden is happy to step to weed in and resolve some of these issues and I think some of the lessons could be shared a bit more broadly there and then your programs crash too much sorry but they crash they're not robust they're not sufficiently module I think we said before should stratification but these are all problems they are problems they're not robust they're not interoperable it's really annoying the D3R challenge we would have finished it probably last Tuesday but we're running the free energy calculations they just randomly crash this is the only field where we accept randomly crashing software it's like would you want your airbus like autopilot and it's a significantly more complicated piece of code than the package even though they have to reboot most of them reboot 24 days or something like that otherwise it will fall out of the sky I think it was the 77 that was the 77 yep I'm disappointed there was no software engineering it was collapsed on much of the aviation industry and I'll now be flying airbus so it's not interchangable the other thing is about methods is if we could find a way where we could actually publish protocols and share protocols we can actually just say we use standard protocol X so I used the like you know Jameson group protocol X for setting up ligands and almost give a gig of a liver to it and if we can do things like that and actually you can always think about automating writing methods in shows because that would be really useful difficult but very very useful you're actually going to be able to find and then we're excited to find out what's going on Eric was sitting in Gromac's doesvis which is protocols.io it's an existing tool that does that I think and then the other existing tool is htmd and Biosynics Waste is not an existing tool it's an existing development I'll keep that to say it's not complete we are still halfway through the project and what we're quite keen to have is actually equally developed tools at the moment we're writing the wrappers around those tools ourselves what would be really cool is if people who develop a tool like tool to protonate a protein or a tool to parameterize a ligand or whatever we're very happy to talk to you and help you write the wrappers around your tool so that we can deal with all your file for one supporting you can then only deal with one file for that and yet you can still put it into everything else it's hardly important and data publication I can't remember just it would be really good if we published our data actually I remember as you did in the data publication so the reason I came so I've spent too long in this career working in this field and I left it for a bit and it was always in the case of being a PhD student where you always were given something from a previous PhD student you had to reproduce it or you then had a result in a paper and you had to reproduce that and just getting to the idea that somebody would actually share their starting structures and their input files and I was back in the early 2000s people were not doing that people were not sharing their starting structures and their input files and they still hold on to them thinking that somebody is about to steal them from them at any moment and rerun the simulation before them as I said all of our input files and starting structures for our D3R challenge are on github now as software developers when you're coding it's live going on to the internet now it's not waiting for publication and three months later they'll publish it you're doing it live can I just also say I think that's also really important because if you share a bunch of files you call them file 1, file 2, file 3 not really this much so there's no point in doing that you can share it on you can share it on I was looking at the product and I was like it's not a product it's not a product it's not a product it's not a product it's not going to hurt it's not going to hurt you can do more exercise this way we can go on we can go on for a run there you guys so I think we'll agree that if you want to reduce friction that's amazing how is it not important I really objected to ever target a D3R that is what it's meant to be I think I'm balancing basically there was nothing there one thing that has been mentioned already that really reduces friction is when things are documented and documentation is somewhere on the board the documentation is good when you already know what you're doing so training is important as well and training is there as well it's a big overlap between the yellows then this is one of the things actually all of this documentation training would reduce the inertia there is within a group to use that specific tool because a specific tool is used because the PI said so but also because there's some expertise people know how to use that tool people do not know how to use the other tool then you need to lose three weeks to learn how to use this other tool and then you hope for the best that you are using it properly so we are not looking enough to the other tools the one that exists already and this is partial because of documentation generally black there have been efforts to address that for example, morcy has a giant database of a bunch of random stuff in carcam but it sounds useful yeah but then does it that's not trained we know it exists we have many lines of code there because that's one of the fields but it does not tell me how to use it properly and I don't know if I used it properly until I have trained so there are many tools and we should have more tools but that requires tools to be more generally enough so they only do their thing and they do it well and that would be highly facilitated if we had a common data model and we go back to the first topic of this morning the we do not have this common data model that is agreed upon yet but we do have tools to convert between file format between project we have that can produce input for many simulation engine we have AC part that can convert topology from some format from some format to some others the last thing there would just be easier if we have this common if we could agree on what the bond is so we have all these tools all these tools are dealing nicely with what they know but then we don't do all the same thing because then we don't need to be that many if it's always to run the same simulation so we will run simulation with stuff that are slightly weird that are not covered by the tools like this weird post-transitional modification suddenly your protein is not a linear polymer anymore it's a branched one PDB to GMX breaks and so we need to be able to be more flexible to look at this non-generic aspect that breaks the assumption of the existing tools which goes back to the documentation because often it breaks because the assumption is not documented how many of the tools assume that your atom has an element and doing coarse grain I do not have atom that has an element in the very big table I have to lie to open the map but then we have all these non-generic cases that exist but nobody knows about, well the tool do not know about them and so if you are not careful you end up with something that appears to be right but does not account for that weird molecule that was there in your file and that got removed magically so we need sanity checkpoints to identify I don't know what I'm doing here and actually this has been mentioned already this is the error reporting error reporting, yes which I lost meaningful error message or something it's a little maybe no well I guess we can find it at some point and finally one of the problems we have is that people who write the tool and the people who use the tool are less and less the same people and so when I have this strange case I need to report it in a meaningful way to the person who is writing the tool and this is a and the developer we go back to the documentation need to say hey I made this assumption be careful with your stuff and this is a feedback loop between developer and user which we need and he's not completely defined yet and I think I'm going to bust it I don't know I don't agree I was telling the paper you don't have to say it or something there's tape, I brought tape so many solutions to this problem are you causing your workflow? I'm just stippling there's no way of doing flipchart man and man you guys are out of the box thinkers I'm usually among the groups here we identify things that weren't important so we haven't required the frame shift that all of the discussions have so critical among the things that we think are easy ones are that we can have validation suites for force fields particularly as a developer I frequently get requests for can we have charm 36 yes I'd love to please show me the github where there are all the inputs and outputs and the values I have to reproduce what we have in Gromax at the moment are all the things that contributed ages ago I can't say yes this absolutely includes something faithfully unless there is a test case for the force field you don't just need tests for software you need tests for our physical models as well that comes in a lot of forms that we need finding what is meaningful to validate the force field is not that trivial there are some things that are easy ones we should have free energy cycles that have to change to zero we should be able to reproduce within the limitations of the force fields that we parameterize to things like density these sort of things we won't exactly reproduce this but at least we should be able to between code versions and between codes if we come up with some limits to go and investigate a problem but initially what we need is just to reproduce the single-point energy if it is conscious that's the best one otherwise you're involving things that are not having anything to do with the force field we're choice embarrassed at their choice embarrassed at which is important but that's an capability that goes rather than a force field we can turn it back to usability because they don't know what to choose they're not told in the hands to evaluate what to choose but in terms of you being told that you just want to be able to show that you can reproduce the energy that's reported for those molecules because then I think a more example was between number nine we had whatever identical results silently between number eight or nine or nine or ten it changed the way they interpreted their own torsions completely fine the reason we find out that suddenly if it impacts their results exactly anymore, completely undocumented some they don't read, some they don't and the point is not to blame Amber but the point is if they had an internal test then it would also be clear to them that we no longer passed internal that we have now updated our regression tests having on that we would have also been able to oh they've updated their regression test because they changed their interpretation good, then we need to change their interpretation too as a community when we hear someone publish a new force field we ask them where is the test suite it won't be available in your favorite tools in the end to us there and test suite which should have a specific meaning for us to agree upon we found something that was of more importance and also quite easy is to identify the massive bias in space on their dynamics are these fit for the sort of simulations that the community wants to be able to do are we able to agree upon simulation protocols that are able to be implemented within the bias-based style models is there something as tool and application developers that we could actually implement as a higher level thing that we currently have at Hock file formats and non-descripts they've all worked 20 years done on models we should give them feedback because they are people working in this done model space that very few people are working on the straw men that they are proposing implicitly and the fact that they've got a well-defined data model is something that we should either criticize if we can or agree with if we can but that is a very easy thing to do as an application developer I'm aware of what the data model is after there's an application about it I can say yes I can implement that, good rearrange the stuff done and now we have something that's very easy for them to put groomed machines around for example and we thought it was important but a little more difficult is improving the user experience in terms of being able to produce parameters of the ligands particularly out of mystic force fields there are people contributing to this space there's relatively few people contributing in coarse-grained force fields parameters of the ligands we thought that was actually more difficult because few people are working on how to get that done is less well established we are working on it good, glad to hear it I've heard you try to do this as well so we have in GrowX a tool PDB to GMX jokingly we rephrase one of the frequently occurring requests that we have as tool developers please allow me to just run my simulation I want to give you a PDB code and then I want to run a simulation so I would be jokingly describing that as an MSF2TPR to avoid the growX simulation my response to that is what do you as the scientists do if I can do that for you why aren't you doing something more honest with your time because there's a whole bunch of questions that we as scientists need to be able to think about what pronations take to make sense what are the experiment protocols actually translate to in terms of something we can actually simulate there's a whole bunch of positive text I think there's a really important point there because as we're making these tools more and more accessible to people then also people who are not thinking too much about stuff are wanting to do it and so that goes maybe to training or other aspects sure absolutely I can build an MSF2TPR and then we'll just be around I feel like it's definitely a black box by having to see the black box which does everything for you automatically what's wrong with the black box if you know how it works you don't have the knowledge of like how important or what do you call it it's not the same argument you need to know how to do the current trial you don't learn anything from setting up a simulation you learn something from analyzing the simulation well you don't know the simulation crashing I think it's ridiculous to think that there is like the scientist as an artist is somehow shaping something out of the PDB file to get whatever conclusion there actually it's not quite true but we believe presumably as a field that there's a best way to do something if you're after a particular question and that there should be a standard way of making the right decisions along the way that give you a robust simulation so in principle we could write this provided you got some expert information what's the expert information that you need to put out something that's not a random check but what would that be so that's what you mentioned in your example of method section it could be something about the experiment you're trying to reproduce but maybe only certain experiments are amenable to that kind of simulation right now and speaking of the fraternization state I think all the people trust you to lead those states should we take those up would that make that code more visible no but I mean nobody can force you to do good science if you are not willing to but there are good tools that can make you and that can help you do these good science faster and even more efficient at least in the domain and specify problems that you find but people should then know about these tools because men just don't and like when they are the tools out like it should be shared we're going back to this we probably just want to run you a few nanoseconds of the protein in water and we just so this could be a good build up there's a vision in the pH and 0.15 molar salt or whatever just creates some physiological conditions that would make a lot of people happy but one of the reasons people pay ridiculous amounts of money for modular or ridiculous amounts of money for loss of the tools which they use in pharma is because they have really good PDP to set up systems from codes and they've got the protocol so again the credit suite they put a lot of effort into taking PDP and doing it and then presenting the choices in the GUI in a way where they had to make choices like okay user which is to do and you just pick and choose you know so it's making it really simple at the moment in the academic space we really haven't tied these protocol tools together for protein setup at all we don't have a good tool that does leave funding because no one's really put effort into actually building missing messages at all it's like it's an abandoned thing I wish we had an open source lead modeler because at the moment modeler is the only thing you can use for any of the worlds I really like making models but it's the closest thing and it's that protocol thing it's like you need to go here like some open source modeler is there anything that anything in the tools category that does the modeling for you for free? you can also do it in time yes, that's true as it was said as physical there's a lot of delete modeling programs they're the same certain ones have been developed in Oxford and I shout out names for it I shout out teams the general problem with loop modeling is that it's a very narrow regime where there is a significant quality difference for very short loops everybody gets it right if they just close the loop for if any rest you just don't like it it's it right and then there is a there is a there is a regime somewhere in between where some tools do better than others but it's very relatively narrow but I mean I have a simple script actually you can ignore it for that reason it's not that important to have the best loop modeling tool ever but I would love to have a script that oh I have six missing rest users we have this implicit knowledge within the community of actually this will work but this will not work there are actually protocols which we teach our PhD students and say you do this, you do this and it's just about finding ways of encoding in shareable protocols and then you learn one little about what you've been doing when you say okay we've really also six missing I'm happy with it you know that you don't continue in a workflow yes you keep doing but I'm saying let's share those protocols of doing protein sets up and making set up so that you can basically rather than taking this this is the hidden group way where we build the liquid, the hidden group way where we do a protein that's in a public arena so one challenge to think about from a user point of view I'm very much inspired to the idea about using other tools but if we grow back we're going to we're going to kill for you to do mix that's actually why it hadn't made a whole lot of problems in the last two years because it's a kids sensing solution that tries to be able to fix things that's very different from the problem of typing something to a point of speed however the general challenge for a user of course to be able to use any some sort of tools to prepare you but then you need to run it it would also be frustration of users if they're repeatedly told though you can't do this you need to find extra program one and two and three and four and five and six and seven while on the other they also know that Linux distributions they really hate it when you somehow just aggregate and pull tools into packages because suddenly you're going to have 14 installations of that in the Linux system so maybe we need to find some way but maybe either grouping tools to make it easy to pull down these so that they can be maintained externally but when we actually install them you will have them installed so this is what my hope was was with Fonda so when we originally built it was actually for tools that existed it detected it and then did a condo install one little lab did that beautifully so we realised that condos from the Xe Broke could have and if we fixed the managing dependency problem then actually the user shouldn't see this tool doesn't exist so he pulls it from the back and we talked about that over the base before that so it's easy to start new things and I would actually I spent this part to be full this can work though all the time with our tools it pulls down 40 dependency that we don't have to write ourselves and it reduces the maintenance burden on ourselves because we can use well-developed and well-maintained tools and pin them if there's any other problems so I think it's something that can be solved as a community if we floor so I want to give Mark an opportunity to finish his things and sit down with his view again something that we talked about particularly from the growbacks point of view is that we need to work on separating our analysis tools from the preconditions analysis tools need do I need molecules made of hold do I need to cluster things from how do I do the IR the post-processing this is a discussion because as a community we now can move away from somewhere out of the car I have to fix PPC before I can do anything else with it this is about separating the data model aspects of okay I have an analysis it requires these preconditions something needs to establish my preconditions that might already be true my user might notice true user might have given me enough information to be able to establish this precondition all of these things will enable us to implement analysis tools in a transferable way so we can get more into our capability to have a data model that actually works and that we develop analysis tools that actually announce what the preconditions actually are we have like five different variations of minus PPC implementation in growbacks I mean I assume other people have similar quality signals we need to work on this we upgraded the need for documentation to be the need for stellar documentation documentation in space that is yes yes that's true no documentation the single most valuable thing we found in the growbacks project for making sure that we can have tests to go along with your code documentation to go along with your code is that nobody can change the code to other people have reviewed it and for this to be the community expectation and for that you need everybody including the PIs who has done the project have to go through the same groups you can't change the code unless you have tests on your code unless you have documentation on your code both of the developers and the users doesn't matter that stuff doesn't go in we need in the more difficult category we need general well designed input output descriptors that might belong more into this sort of corner and this comes back to some of the other more difficult issues that we've been tossing around we don't have a good way of describing a chemical species in a machine possible way that allows us to understand oh, what is in this video I need to go out and build those models with more missing residues because they're missing I need to go and choose a problem I used to have it told me all of these sort of things are very difficult for us to grapple with at the moment we also don't have general well designed output formats either but all sorts of things need to get improved in that space some of which we talked about earlier today and also in the long term category we need to start evolving high level APIs such as good down models that's a great one started in biasing space to develop a good down model and as a tool developer I would very much like to be able to build upon that in terms of expressing a pattern that they won't need to write shins around because we'll just the doubt should be in memory we should not need to write intermediate disks alright, thank you so much well, okay look at the most important side going from easy to hard at least we have something emerging okay, it starts with better docs and training and how we can get there is anybody's guess like you've been good about enforcing that you have to have documentation with your code but are there other ways we can encourage our community to do that modularity we have data model, common data model we have APIs which are pulling from up there as Marc just mentioned we have escape from dependency hell better error messages and finally up at the top we need a culture change that rewards modularity people said was hard so all these things are possible technically it's possible to write a common data model it's possible to write good error messages there's some technical schemes that facilitate that it's also possible to resolve the dependency hell if we put it in a hybrid but what's the societal way we can make sure that this is what people want to do or that people are encouraged to do this people in this room are certainly motivated but is there a way we can facilitate or encourage that development through some change that we can do later somewhere


So, Rashmi in the case of Gillette we had a company that asked designers to come up with a product that would expand their market, but I think you also have an example of an individual coming to a designer with a need which led to a new product. Yes, it is an interesting story from the world of ergonomic design, Sam Farber a manufacturer of Cookware watched his wife Betsy struggle with arthritic pain specially when she was working in the kitchen. So, he got in touch with a design firm called Smart Design and he asked them to develop a range of kitchen tools which would be easier for her to use. This led to a new line of kitchen implements the oxo good grips range. Kitchen products whose winning feature was their ergonomic handles. I know I have a peeler at home it is really good. Isn't it? So, Betsy's special needs led to the creation of a new product range which became a new line of business for the manufacturer you know for quality results like this to emerge from a design exercise the manufacturer and the designer must form an effective partnership and then they work like allies and good work emerges. For more about this, do go to the next tab.