Alright, I'm going to go ahead and introduce everyone and get the ball rolling. Welcome everyone. Good morning. If it's morning for you, if it's afternoon or evening, welcome. Very happy today to get the ball rolling and to discuss this special issue and the work that's ongoing in order to help improve the specificity and guidance around registration especially in evidence-based toxicology. Joining us today at the Center for Open Science is Paul Whaley. Paul, say hello. Editor-in-Chief at EBT Evidence-Based Toxicology. Yeah, good morning, good afternoon, good evening. Everybody, thank you for coming. Looking forward to kind of presenting this work. I hope you find it interesting. Also on the call from the COS, my colleagues. We've got Mark Call who will be demonstrating some of the key features on the Open Science Framework Registry and Katie Corker, who will be going over a couple of common issues and details with pre-registration. Say hello. Hi, nice to see everyone, glad you could join us. So, I'm going to get started with a general overview, a little bit of a very brief history of registration and some of the templates available on OSF and then I'll be passing it off to Mark to show some of the more detailed features. Paul will then be describing the special issue and what we hope to gain from this call of requesting standards and developing best practices for the field and then we'll finish things off with Katie. So, let me go ahead and get started with a little bit of definition and history. So, when we talk about pre-registration, what we're specifying is that it's a read-only version of a research plan posted to a public registry so that others can see that the work is ongoing. Now, there are a couple of qualifiers to that definition. We do know, we do allow and most registries do allow for transparent changes as the process evolves. Those changes to the pre-specified plan are allowed and some registries also allow an embargo period but generally speaking the process of registering a study involves posting it to a publicly available and viewable site. If you're coming from a clinical background, the term pre-registration might be a little bit different. Of course, it emphasizes the fact that it's taking place before the study occurs. Perspective study registration might be more familiar to other disciplines and some of the social sciences terms like pre-analysis plans are a little bit more prevalent than those are all... Some of them are slightly different but for the most part they mean essentially the same thing. So, pre-registration has been around for a long time. It's been called for in many disciplines for quite a while but it's been fairly ubiquitous especially in clinical medicine and since the early 2000s when there was a specific problem with the number of research trials ongoing funded by the federal government and constituents weren't able to access or determine where those cutting edge treatments were being administered. So that led to the creation of clinicaltrials.gov and around the same time registries in across the world in many countries were forwarding to solve similar problems. In addition to help open the file drawer, it helped increase discovery of ongoing research. And in basic research in pre-clinical work can increase transparency and ultimately what we're looking to see is a clear distinction between planned research from unplanned research. These two modes of work also known as exploration and confirmation of pre-specified analyses are equally valuable but distinguishing between the two can be quite nuanced. In exploration that's really a data driven process. We're looking for unexpected, unanticipated discoveries. We're looking for pushing knowledge into a new area and find unexpected trends or relationships. And this can be really used to inform theory. This is in contrast with confirmation of a pre-specified predicted relationship. It uses traditional tools of statistical inference, most frequently hypothesis testing, even more specifically null hypothesis significance testing. The context of confirmation is held to a higher standard of recovery. You want to be extremely sure of a relationship before suggesting that one exists. And it can have a more direct decision on impact. And we want to minimize false positives in the context of confirmation, whereas in exploration you want to minimize false negatives. If there's a chance of a relationship you want to be able to discover it but before saying that there's a strong causal relationship, for example, you want to hold that to the highest standards of rigor. So both of these modes of research are equally valuable for pushing knowledge forward and pre-registration helps make the distinction between the two more clear. So how can pre-registration help the overall research planning process? It can help improve overall planning. It can help bring additional feedback to the study creation and study design process. Importantly that feedback occurs at a very important time in the study design process in which the study, the data collection or perhaps the data analysis hasn't taken place yet. And so those changes or those bits of feedback are most useful at that early stage of research. Especially when using confirmatory tests, it helps to improve rigor because it really specifies that these were those pre-specified analyses that were not informed by the data exploration process. And it can help to limit what's known in some fields as questionable research practices, those QRPs. It can help specify that the plan is being followed precisely. Once work is registered and conducted, there are a couple of rules of thumb to follow in reporting the results of pre-registered work. Importantly, one must report the results of all pre-specified analyses. Any newly added tests are encouraged, especially unexpected trends or covariates or anything else that was done are enabled and encouraged through this process, but they should be transparently disclosed as unregistered or as unplanned analyses. And if there's any changes to the data collection or to the data analysis process that happened after the plan was registered, simply state that as such. Now, there are a couple of different ways to pre-register, and this is what we're going to be getting to in the webinar. Essentially, it's structured versus unstructured. When you use a template, it reminds it's a checklist, essentially, asking the author to specify all the important details of the research plan, the hypotheses, the data collection processes, what analyses will be used to address each research question. An unstructured registration is simply a narrative document describing what the process is. We have about a dozen templates on the OSF. Some of them are quite specific, like the recommended OSF pre-reg template. We have some that are specific to different modes of research, such as systematic reviews, secondary data analysis, qualitative methods, and we have some that are specific to publication types. Next report format is one of which the proposed study is peer-reviewed by a journal prior to it being conducted. We encourage the use of registration templates because we know that they help individual authors include all the important information needed to articulate prior to the study being conducted. In this evaluation of structured versus unstructured research plans, Bakar and her colleagues compared 53 registrations that were structured to those that were unstructured and found quite a bit of differences between the two. Essentially finding that the unstructured ones didn't include necessary details in order to fully pre-specify the analyses and authors also recommended improvements in ways to make sure that those structured ones included as many details as possible. Now I'm going to hand it off to Mark to talk a little bit about the workflows on the open science framework on OSF. Mark, would you like me to stop sharing my screen? Yes, please. All right. Thank you, David. David discussed about the concept of pre-registration, what it is and why we need it. If I can share, I'm going to show what a pre-registration looks like specifically on the open science framework platform. I am not going to go through all the small little details, but more of saying kind of like what does it look like and how do you get that process started. This is an example of a pre-registration template. David mentioned earlier that we have several different ones. The ones we're looking at is the OSF pre-registration template. It is one of our most rigorous ones and the more structured ones. And the intent is to really challenge researchers to think through the different nuances of their study to help you think ahead about what some of the challenges you may encounter. So kind of going through this, you will see here on the structure side or the left-hand side, we have an overview. That's where you all have all those different types of questions. How those questions are entered, it could be text fields, radio buttons, check boxes and all that is right through here. And some of them will also give you the option to upload files as well. So if there's something that has way too much context to put into a simple text box, you do have that option of some of those questions. And everything is structured pretty much in a section kind of format and subsections. So that way you can easily navigate through it. So you can scan through it and you'll see it through formatting. Or you can click the hamburger box up here, click something such as design plan and it'll take you right there. So everything is structured so it's easy to navigate. This also has a lot of metadata. So when the metadata is going to be here on the right, and what that really does is help others discover your research and it helps you be able to do other things such as if it's public, you will share your DOI. So every public registration has a DOI. So it's always linkable. It will also tell you when it was registered, so that way you have that kind of permanence in that timestamp saying, here is my study design. Here's one I submitted it, et cetera. It gives that kind of sense of kind of confidence, kind of reduces some of that concern that some researchers have and other things such as contributors and description of the registration. Just kind of like an abstract, except you kind of don't know your results yet. Along with that, there are other different pieces such as you can, it also be able to arch up all the files that are stored on a project or an connected add-on such as OneDrive or GitLab, et cetera. And then you can also add different types of resources. So we know that with different resources, depending on what your discipline is or depending on what your study design is, there might be other repositories that's best to house that research. For example, if you're doing EEG data, you have terabytes of information. And there might be some different types of requirements based on funders or journals or your discipline to put that data in a specific type of repository. Well, OSF, what you're trying to do is connect all those different types of objects. So you don't have to have them on OSF, you just need a DOI. So you can click on one of these types of resources. It will take you on this page where you can add that through DOI. You can add a description and all that information is right there. So pretty much since the pre-registration is kind of the first stepping stone of what you're going to do, then that's it will help be able to build kind of that structure, kind of that house that creates that foundation. And then you can add these pieces brick by brick until you create the entire house. Now, how do you actually start a registration? It's really quite simple. If you go to the main OSF page, you'll start at OSF home. You can click done here, like registries. Click add new registration. I want to assume that you don't have a project because we just created an account. Select which registration that you want to go with. Let's go with open-ended. It is our most flexible, create draft and you're off to the races. If you need any help or support, we do have help guides. Those are populated at help.osf.io, which I'll put in a chat in a second. And if you have any more specific questions that the help guides don't really answer, you can email me directly or you can email us at support at osf.io. And I will put that in the chat here in a minute as well. So with that, I will stop sharing screen and I'll pass on to my next colleague. How did you want to share your screen? Do you want me to? If you want to share, I think I've put my slides in the wrong place in the slide deck, so they are from slide number 20. Sorry about that. So if you go ahead and share screen, then I'll just say next when we want to go to the next slide. I'm just trying to get that slide up. Good stuff. Thank you very much. So thank you, David, Mark. So I guess I'm here talking about pre-registration templates for toxicology and environmental health. Next slide, please. So for those of you who can't remember who I am or don't know who I am or weren't here at the top of the hour. So my name is Paul Whaley. I'm editor-in-chief of evidence-based toxicology, which is an open science journal for environmental health. And I'm co-chair of the evidence-based toxicology collaborations, open science working group. So you've heard of our pre-registration templates in general from David and Mark, but you're probably thinking, why is toxicology environmental health journal getting involved? And I'm going to explain now. Next slide, please. So the reason really is that in toxicology, environmental health, we have the same challenges as many other fields with the reproducibility, accessibility, and so forth of research. But in toxicology, we probably, to some extent, don't really have that same rich tradition of discussion of whether the publishing practices that we're engaged in will have incentives and center structures that are working under a sufficiently supportive of our scientific goals, right? So we're kind of famously a bit slow on implementing systematic review and systematic evidence mapping methods for making sense of evidence. They were very kind of very much mainstream by the mid-1990s in medicine, more or less unheard of in our field until around about sort of 2014, 2015, really only kind of became a big deal probably around 2019, 2020. Protocols, so the plans for doing a study, making those available before doing the research, those are still very rare. Like they're rare for systematic reviews, there probably is still only maybe three dozen of those have been published in Environmental Health Journals. And pre-registrations for animal studies and in vitro studies are even rarer still. So it's just not really part of the research culture. And to some extent, there's probably some ambivalence, I think, about what the replication crisis means for our fields. No one's really done a lot of research into that in our domain yet. And then obviously then how to respond. Like we don't know what the problem is, we don't know how to respond to it. That's not really discussed. Next slide. But at the same time, we're toxicologists and environmental researchers, like many other scientists, very good at complaining about other people's studies, right? So we ask ourselves how much of the research that we're doing is really being done well enough. Particularly in research that's been done and is sort of intended for making decisions, a kind of regulatory toxicology, this kind of thing. There are lots of concerns about whether or not we really have enough information about the study methods to judge the validity and the utility of data for the decision-making context in which we're working in. And that's kind of seen as a big challenge for making research as usable and useful as it could be. There are questions about journals contributing to publication bias. We know that there's probably an over-prevalence of positive findings in the literature, even in our space. There are some interesting incentive structures which may mean that negative findings may in some areas be also over-prevalent. Generally quite tricky to get on top of. We know that authors are probably selective in the findings that they report from their studies. If they've looked at lots and lots and lots of outcomes or lots of intermediate outcomes, they might only be reporting the significant findings amongst that. We don't know how much research is being done that is basically redundant because we already have enough information about that thing. Yeah, next slide. So evidence-based toxicology is a journal set up to address this, we think, we may be wrong, but we think with the first open science journal for toxicology and environmental health were set up by the evidence-based toxicology collaboration as their official journal in collaboration with Taylor and Francis. And our mission is to publish manuscripts that utilize or advance evidence-based methods and toxicology, implement innovative open science practices as we handle submissions and hopefully demonstrate the value of open science practices to the environmental health community. All very grand. Next slide, please. To support this, we have a set of key editorial policies that are intended to improve publishing by putting science for impact and increasing transparency and accountability of the decisions we make as a journal. We've got some stuff that we're pretty pleased with. We're pleased about ourselves with. We require preprints because we think that basically solves the problem of publication bias. We do public manuscript evaluation and peer review. So when we make a judgment on a submission, we publish the judgment and its reasons as a linked record back to that preprint so everybody can see why we thought what we did and made the decision that we did. We try to base our decisions on rigor rather than the results of a study. So we try to focus on the methods, not how cool the findings are. We are aiming a very, very high open data standards. We're hoping to end up in the top 1% of top journals worldwide. And we give people the option to publish our methods before collecting their data. Next slide. Of course, improving publishing overall is very complicated and strong editorial policies help. But so does experimenting with different publishing formats. And we want to give people the maximum opportunity to innovate in the types of manuscripts that they submit so that if we are not aware of the submission type, that could be, if it became normal, beneficial to the research community, we want to give people the opportunity to make those submissions. We happen to be involved in the world's first randomised control trial of editorial policy interventions. That's going to be a slightly different submission type and you're going to be hearing more about that through Centre for Open Science because they're the people leading this and we happen to be involved. We do consider protocols for any study design. Then we're open to methods papers of any type and then the opportunity to try pre-registration templates came along and we were like, huh, this kind of strikes the philosophical hold with us, obviously. Next slide. And there's just a good fit for us because we encourage publication methods before collecting data. I think I said that three times already. This is obviously difficult to do without examples and templates showing how. So if we can publish a type of manuscript that helps people publish their methods before they collect their data, then that's only going to be a good thing, right? These templates, because you allow us to view the method as a product of a research process, that gives people the opportunity to publish and get credit for work that might otherwise have gone unrewarded in journals, maybe that only publish the results after the methods are done. And then we thought, well, how do you kick things off? Well, if you're a new journal, you kick it off with a special issue, right? So that's what we did. Next slide. So give you some ideas for templates. Next slide. The reminder of how it works. Basically, the gist of all this is that anyone with a method that someone else might use should create a pre-registration template. I mean, it really is a submission format for everybody who is interested in helping other researchers use their methods, right? So you've got a researcher, they develop a method, they conduct their study. They probably submit that study to a journal and get it published good for them. But then what they're thinking also is, well, I know that I've got colleagues who want to use this method. They're probably people in different labs have not met yet who might want to use the method. So what I'm going to do is I'm going to convert the method in my paper into a pre-registration template. You know, publish that template in evidence-based toxicology or with Center for Open Science or with both. And then a second researcher comes along, they're like, you know, the Googling on the internet or whatever, and then bing, they find this template that relates to the study design that they're interested in doing. They're like, well, this would make it much easier for me to pre-register or it gives me the recipe for the study if you like. They go ahead and they pre-register in confidence. They're covering their bases and they get all the benefits of doing that pre-registration that David was talking about and the previous researcher gets cited and everybody wins, right? Next slide. So lots of options for this, like talking in the abstract, maybe more difficult to understand what those options are. But we could think about minimum information for registering the existence of a planned epidemiological study. So that's quite easy, relatively easy pre-registration to do because you're only giving people the information they need to understand what study that you are doing and when. So they don't replicate your work by accident that they know what you're doing so they can plan around that. People will be aware of your study before you finished it, how it was publicizing it. You can plan the analysis of data. If you're doing this like you're doing data heavy stuff with cell culture things, this could be really useful just to have like an analysis pipeline template that everybody who uses the more or less the same method that you're using can use to pre-register their analysis plan. That's good for preventing kind of expectation bias and things influencing results in a way that sort of biases them towards like positive publication. And then if you wanted to aim really high you could do like a comprehensive study methods and data analysis plan for like say big kind of observational studies. And if you're seeking to comply with good epidemiology practice because you want the regulators to respond to your research then that could be kind of like a big deal thing that you're aiming at. So there's lots of levels to aim at. Lots of different ways of thinking about this as a journal.  If you want to go big and go wide, we'll support that. Like we aren't interested in prestige we just want good methods. So we'll be very open to pretty much anything that people can come with provided it's rigorous. Next slide please. See if there's checklist or guidance there's a template is basically what we're saying. So give them and regulate your compliance kind of issues. So you're doing a study you want to regulate your uptake you could interpret the guidance the regulator gives into the specific studies that you've got and that can help people really kind of crystallize what can be quite broad guidance into very specific standards compliant studies. You can do that for epidemiology. It could be really useful for animal studies where you're trying to show that your study is worth doing that you've minimized the use of animals that people will necessarily need to do an unnecessary follow-up study that would waste animals. Systematic view and evidence mapping that's very kind of structured and procedural and that lends itself well to templates. If these are necessary, desired, end up being helpful they could be very high impact publications. So I really kind of recommend that people consider doing these things. Next slide. I just say reward about eight slides ago if anyone's ears pricked up to that we will be giving the best templates or templates that we receive at the journal before the end of 2024 a cash prize at Society of Toxicology 2525. We haven't quite figured out the details like how much cash, because it depends a little bit on how many prizes. But we always want you to know that we are really keen and we definitely want to reward people who do this work well. Next slide. So thank you very much for listening. Thank you for Center for Opus Science for this initiative. I recommend you read the editorial. You submit your template to the link. You can follow the journal on LinkedIn for exciting updates about new publications when we publish evaluation reports of a manuscript. If you bit knows, you can follow that. You can, we'll place links to those. And if you really want to, you can join an evidence based toxicology working group. And I certainly recommend the Opus Arts working group as the coolest working group that we have. So I'll hand it back to, I think Katie now. Yeah, thanks very much, Paul. That's super interesting. It strikes me just in general that this is a process we would love to see duplicated at other journals. So if there are other editors out there that are interested in doing this sort of thing, please reach out and see if we can set up a similar process at your journal. Even just for people in other fields more generally, I think this gives a nice concrete example of the kinds of templates that we'd like to feature on OSF more broadly. And so we do have another process that runs alongside the process for the special issue where people can submit their templates to be considered for inclusion in the OSF. And all of the templates that are submitted to the special issue will also be considered for inclusion in the OSF. So it's sort of two parallel processes for evaluating those templates for publication and for inclusion in OSF. But it's just an opportunity for there to be even more impact for those templates to be of broader use to the community. We do wanna take your questions and have time for everyone's to get everyone's concerns heard and the questions answered. I have a couple of questions to sort of kick us off, but please be thinking of your questions while I'm asking these. And you can go ahead and put them either into the Q&A or into the chat and Zoom. So as a sort of seed to get us started, one question we might think about is why we need these pre-registration templates at all, right? So why is it the case that the pre-registration templates are providing something above and beyond what's in the method section of a paper or what might be in a protocol study by itself, protocol paper? Protocol paper. And this could be for any of the three of you or all of you. Thanks for sharing the slides as well, Paul. We'll definitely share the slides back with all of the registrants as well after the webinar is done. Yeah, that's a good question. So there's a couple of different ways to sort of articulate the benefits. So one is specific to toxicology. Every discipline is going to have slightly different norms, different terminology, different methods that are frequently used. And so the benefits of field-specific templates are that the language is most interpretable by the intended audience. We have several on the OSF right now that are very generic, but they were intended to be quite generic in terms of discipline relevance. But frankly, they were created by folks from within specific disciplines. So some of the terminology, some of the norms might not mean precisely the same thing when reviewed by a different audience. What's the difference between a registration template and a protocol? The registration template is really designed to mark somebody's claim that this study is going to be taking place by these people at this time with this population of data or of people. And it might reference, it might use a specific protocol, but it has more data, more metadata around it, articulating why this study needs to happen at this time and then using a specific protocol that might be included in the registration or some of the templates that are submitted to this might rely on a couple of externally previously published protocols and ask people to identify which one will be used. So the variation there is what we expect to see from this type of open call for templates. Wonderful, thanks. Sort of general question for Paul, and then I do see we have an audience submitted question, so I'll ask that one as well. Question for Paul, would evidence-based toxicology consider templates that are partial, so ones that are focusing on a particular issue that's relevant for evidence-based toxicology rather than say a full-fledged template? I mean, I know you addressed it a little bit at the end of your talk, but could you just say a little bit more about kind of the scale of the templates that you expect you might see? Yeah, so, I mean, we're very open to working with authors on what they think is a worthwhile amount of work and what they think is a unit of contribution, right? And we don't necessarily expect all units of contribution if you like to be equally massive, so whilst obviously big papers and massive contributions are very welcome, also small contributions are very welcome and we don't mind how big the contribution is and that isn't something that we take into account in our decision-making when we review a manuscript, like a small rigorous manuscript, as far as we're concerned, is as publication-worthy as a large rigorous manuscript, so what we care about is rigor. So if you have like a pre-registration template that's quite niche, that addresses just some aspect of, or some value in pre-registration, so like you say, like, we're just gonna concentrate on the minimum information that you need for pre-registering, so registering intent to conduct a systematic review, right, and that's it, then you'd obviously wanna show that you weren't just repeating what's on Prospero in some way, but you had like some unique value, that might be challenging given the Prospero already exists, but you might have something that makes it an additional unit of contribution and then that would be totally fine for us. You might be like, actually, we've got this complex, like all known chip toxicology study where we've got a whole method and analysis pipeline that includes some machine learning and some big data and all kinds of things, right, and biological pathways and then all sorts of stuff going on, and then we'd be like, okay, well, no, that's also equally valid, it's gonna be a big deal, probably, but again, we don't mind, we'll just take pre-registration template and we'll evaluate it the same way for everybody, which would be based on rigor, not on Skype. Wonderful, thank you. Patricia, would you like to ask your question live or shall I read it for you? You can read it for me, Katie, thank you. Patricia is teaching about research integrity and open science next week for first and second year PhD students, and she's wondering what would be a way to convince them that pre-registering their studies is good for them because of course we know it's good for science, but what is the argument considering that they still consider it something that's difficult to do? Yeah, this is a great question. So there are a couple of different ways to go about that. One would start with a sort of improved planning process, being an early PhD student, it can be quite intimidating to sort of figure out what goes into sort of a complete study and just use of the templates themselves can remind one about important design elements to include or the necessary details to add to the proposal as one gets closer to conducting their first sort of independent type of work. Using that template could be a good way to solicit feedback from advisors and committee members that helps both the writer of the plan make sure it's as fully fleshed out as possible, but then helps the advisor and the other committee members or other colleagues that they're reaching out to sort of reminds them about the different elements that should be included and can provide some better feedback there. If they want a strong reward of a promise to publish the results of the work, regardless of how it turns out, they can reach out to a journal that applies the registered report format. And after working with the advisor to sort of flesh that out fully, submit it as a proposed study to one of the over 300 journals that offer that as a publication option. That's a very strong incentive and reward to go through. So that's submitting the proposed research plan before it's even finally registered to the journal for feedback. And we haven't talked too much about that process here, but that's sort of a modification of the traditional publishing format. Another couple of benefits is that it shows progress on a research track in a way that's better than just something's in prep. So NIH, for example, encourages the use of registrations and allows that as an interim research output. So if you can identify that in progress reports and send a link to that, that's much more compelling than just saying something's under development or something's under work. You can actually sort of demonstrate that this proposed research has been fully fleshed out and is ready to go. So there's a couple of different of those types of benefits I would start with. Can I build a little bit on that? Yes, I see that very clearly. The four points you put out. The barrier we've been seeing here is how these students can convince their supervisors who are older, more traditional, not familiar. They don't know what cost is, what OSF is. Really, sorry, but they don't know. And then besides having to learn how to register something in OSF, they have to teach back their supervisors. Yeah, that's one of the big... You see that. One of the biggest kind of culture change impediments almost is just the fact that one might be being trained by somebody who's never heard of this, doesn't see it as sort of the necessary part of the process. It's not something they did. So why should they tell their trainees or even allow their trainees to do it? There's no magic bullet here. BI, sort of teaching about best practice and benefits that they have. You're encouraging the students to learn up on it so that they become their own strongest advocates. Kind of worst case scenario, kind of using the template and not even registering it, but just using it as that planning document is a way to get experience with what it is, even if you don't use it to its fullest potential and that can perhaps provide some more sort of concrete examples of those first two points. It really helped me make up a better planned document and I can share that with you in a little bit more concrete of a way than I would have otherwise. And that could be sort of the step in the right direction we want to see. So yeah, there's obviously nothing, there's no so-called magic bullet, but these are all kind of steps that we encourage folks to take one after the other. Can I also just chip in on this very quickly? So I said I work for the evidence-based toxicology collaboration. So what we do is as a research collaboration is trying to provide a supportive environment for early career researchers who want to use these pre-registration type processes, right? So if an early career researcher is working on an EBTC project, then it'll be the something that we've set up because we think it's worth doing, or something that a member has proposed doing through EBTC and through one of the working groups so they can collaborate with other people in the working group on a project. Then EBTC, we require empirical research and it's going to be like an official product of collaboration to be done on a registered reports format. So you have to do your protocol and pre-register it. So we provide a mechanism whereby if a researcher is not being supported by a supervisor in pre-registering their study, we'll provide that support instead. And if necessary, we'll mediate between the PhD student and supervisor if they want. Usually the extra free supervision that we effectively provide is pretty welcome and provided it's not costing too much or anything, then we just generally have to get on with it, right? So I think that's another way of enabling early career researchers is to steer them towards research collaborations who will support them in this type of work if they're not getting that support in their own internal labs.  So yes, I think I'm basically saying goes to the evidence based on security collaboration. Hit that join button. Yeah. Great, thank you, Paul. Yes, thank you. I would just echo that in general, even for researchers in other areas, finding a community with individuals who are also interested in doing this kind of work is also very helpful for finding support and exploring strategies for navigating these kinds of tough issues. Yeah, just a second comment. We work here with a big group that does clinical trials. So for them, it's their nature to register, right? It's easy, but there are a lot of people working now on laboratory research and big data. And these guys don't know where to register, how to register. Even the systematic reviews don't, which are not, which don't have a clinical question. What do we do? Where do we register? I'm always sending them to you guys, so. Well, we appreciate it, thank you. Thank you very much. So we have another question in the chat about exploratory research. So if I could add on or elaborate to the question a little bit, I'm sort of wondering about the value of the pre-registration process for exploratory research. And then the question that the question asker was wondering was how are templates updated if in the process of exploratory research? I'll put in a link to the update help doc in just a second. Yeah, to answer the first question directly and then I'll expand on that. There's a feature to show updates or changes or revisions to the plan in the interface and that allows transparency to the reader to say, okay, this part of the plan what was updated at this time and provides a spot up to explain the rationale for that decision. You know, perhaps data, the best way plans once you get into the battlefield, of course, go up and smoke. And so as the study is conducted and reality sinks in, if changes happen, that can be transparently disclosed. The clear aim is to still make that distinction between the planned work and the unplanned work. And so there are nearly infinite number of reasons why those changes could have to take place. It could be that the data came in, didn't meet the assumptions of the statistical test provided. It could be that the data collection process was just unfeasible for one reason or another. It could be that a moderator, the temperature or age or anything else like that seems to account for a lot of the differentiation that you see in the results. The problem with just doing that without describing why or how it happened is that there are of course so many of those reasons that noise itself will creep in and provide results that look more surprising that come at the expense of the credibility. And so the transparency that that interface provides helps give context to when and why those changes were made. Especially if they occurred before the study was being conducted, there's extremely little justification needed. The later the changes happen, so the more detailed the justification can and should be. And the baseline recommendation remains generally the same of reporting the results of anything that was specified ahead of time. And then if the sort of post-talk explanations seem to be promising, report those as unplanned tests that show a lot of promise, but that deserve follow-up confirmation or follow-up replication. And so that provides that good avenue for showing what's promising and that unplanned discovery, but providing it with the correct amount of context saying that this can't be the main finding of the study. It's something that needs to be replicated and confirmed through additional research, which is something that's very commonly said in almost any paper that we read, of course, but it doesn't have that transparency of saying why the additional follow-up is really needed. Can I just jump in there and break a desire? So I think this is Adam Betman's question, if I understand. Yes, so I think Adam's also asking a little bit just about versioning of templates, right? So like if you've got a template and then you're like, actually, oh, we've improved our method, right? We want to update our template. Yes, you have to do that. So that depends a little bit on the technology platform that you're using and how you handle versioning. So I'd say I can tell you how we would handle it at Evans Space Toxicology, Center for Open Science will have a different method or approach, protocols.io will have another approach, cell-press protocols will have another approach, right? So what we would do is we would say, okay, because we're pre-print required, your protocol is that your template is, it's first of all posted as a pre-print, somewhere either OSF or Zenado or whatever, and then you would publish the template with us as a manuscript, and then you'd do some updates to the template and you'd revise, you create new versions of that pre-print registration on the repository each time you updated the template. And then probably at some point, you'd come back to the say, look, we've done five updates now, it's been a year, we reckon our template is a lot better, and then we would re-review your template, see if it has improved. Is there any questions we need to ask or anything like that? And then probably what we would do is we say, okay, well, we're probably going to publish a whole new template here, but what you probably should do is submit as a research letter, which is one of the lightest submission types that we have, which would be like three pages long, explaining how you've got new versions, and that's a way of announcing you've kind of got version two or version 1.5 or whatever of your template, and you can broadcast that through the journal and get an additional publication out of it maybe, is one way we would probably go about that. Okay, yeah, that's helpful. Thank you both for good answers. Yeah, I'm speaking specifically, we're working on a lot of in vitro inhalation, toxicology testing, and a lot of it's very new, and we're looking at a lot of different ways to kind of get to the same endpoint to evaluate cell viability, things like that. So yeah, my question came from a point of, if we do come up with our first method, and then we put it in a preregistration template, if that does end up changing, and we find out that there might be a more efficient way, or maybe the cells perform better using a different cell culture media or anything like that, then it may need to be updated with different versions. So, but thank you. Yeah, I think just talk to the journal, all right, and I was like, to the risk of pulling back the curtain and revealing a brinkly little bald old man, like when we're off, we don't know what we're doing either, right? Because this is me. So if you have questions, it might be that we just don't know what the answer is, and then we can work together to figure out what the most appropriate format and versioning process is, and we would appreciate that probably even more so than you would, and then we can implement that for other authors as well. And partly, one of the reasons for doing this special issue is to encourage some of this dialogue between us as a journal and the researchers who are publishing with us, and just try to recalibrate this relationship, but not a journal that is like a gatekeeper of publications. We are supposed to be collaborators in research. We have an editing role, which is to make research work better. So we're providing a service to you as a submitting author of making things better for you. So if there's stuff that we can do differently, we want to hear about it, and we want to change what we do as a result, not just sit here and just like in our ivory tower, go, yes, no, Bing, whatever, right? So yeah. All right, thank you. I can just say a little bit too about the process from the COS side. It is possible for us to version templates as well. So like our existing templates conceivably could get, you know, an update and have a new version. Our current procedure is that new templates that are submitted for consideration for inclusion in OSF go through a review process that's reviewed by the committee of peers, the working group that we have that considers these templates. And then our plan is to put revisions to templates through that same review process, or if there's a major update that needs to be made, that can also happen. But one of the factors that we'll consider when deciding whether or not to implement a template on the OSF side will be about the sort of maturity of the template or sort of how ready it is to be rolled out to the whole world. So I could imagine a situation where something that's a little bit more experimental or at an earlier stage of development might get published by the journal and not yet be ready for implementation at OSF, but eventually perhaps after it's been road tested a little bit more or seen a bit more use, then it would be ready for that kind of deployment. I mean, Casey, so that's because there's a resource implication for the Center for Open Science to maintain a template. Whereas as a journal, we don't have a resource implication because we just post the manuscript which has kind of like a document template rather than like a form with interactive elements and things. So yeah, they're quite different I think. So you kind of have like a two step thing you can publish with us. It may or may not end up being a supported template depending on Center for Open Sciences capacity and the availability of resources for supporting it essentially. Like that second filter is the supported template, but you can still get a template published with us. No problem at all. Provided you hear our regular requirements. Yeah, absolutely. David, did you want to contribute something? No, you covered it well. Thanks. Yeah, Patricia was asking about the peer review process for the templates at Center for Open Science. This is a relatively new process for us as well. We've had a working group in place for a little less than a year to consider new templates as they are being submitted. So the working group has developed some procedures and criteria for considering those templates. It is modeled after kind of what you would expect for a typical peer review process of the templates are submitted, the reviewers compare the templates to criteria, there's discussion, that's some sort of a hybrid between kind of like a grant review or a normal journal peer review. The reviewers, the people on the committee are other academics, experts in the process of preregistration. So people who have either created these templates themselves or just have some expertise in the process of registration more generally. And yes, Paul is one of the committee members. Yeah, I think what I was saying, it is basically, it's more like it's kind of like a committee peer review. So you would go through places where two people from the committee are a volunteer to review one of the preregistration templates as a form that we work through to try to improve and encourage the consistency and thoroughness of evaluation. We're still trying to work out what that form looks like. You're gonna have to do something similar at the journal, work out how we're gonna do the peer review process ourselves. This will help accelerate our understanding of how to review these things. And again, speaking to myself and not for anybody else, we don't know yet fully what we're doing. So we're kind of working it out by doing it to a certain extent. So I try to say way more about what they're doing than I do, so I'm just speaking for myself. But that's kind of the situation we're in. Yeah, no, I think exploring and breaking new ground is sort of part of making progress in this area and contributing. So it's exciting for me to see it moving forward. Well, I think we are about out of time. So I will go ahead and thank our speakers today and thank everyone for the engagement and the questions. Like I said, we'll send the slides around and we hope to see your submissions soon to the special issue.


All right, so it looks like it's 2.50 on my clock. Welcome, everybody. My name is Thomas Cameron, and I am the Global Solutions Architect Leader at Red Hat. Today, we're gonna be talking a little bit about containers and security. This is an introductory session. We've got 40 minutes together, and so we're not gonna be able to dive terribly deeply into containers and security, but what I really wanna do is talk a little bit about who I am, a little bit about what Red Hat has been doing with containers, containers in general in the industry, what are they, how do they work, what containers are not, and then talk about the components that make up container security, including kernel namespaces, control groups, the Docker daemon and how it works and how to secure it, Linux kernel capabilities, SE Linux, one of my favorite topics, and some tips and tricks and some general conclusions. So to start off with, who am I? I'm Thomas Cameron. As I said, I'm the Global Solutions Architect Leader of Red Hat. I've been doing this since about 1993. I have a sort of a cool interest in security because I actually started out my adult life as a police officer. I was a corrections officer when I was a teenager, I was a police officer when I was 21, when I was 24, I went, holy crap, I can't afford to do this anymore. And so I changed careers into IT and I've been in IT ever since. I have been with Red Hat since 2005, been in IT since 93. Got all kinds of Red Hat certifications. Before that, I started out, like most folks in IT, I started out in Novell Network. Yeah, I'm kind of dating myself. Then I became a Microsoft guy and got Microsoft certified and fell in love with Linux back in about 1995 and have been doing Linux ever since. I have spent a lot of time working on security and organizations like banks and manufacturing facilities and e-commerce companies, things like that. I certainly have learned the longer I've been in IT that I don't know everything but I've certainly got some impressive scars. Generally though, just a big old nerd working in IT. So let's talk a little bit about where I've come from at Red Hat with containers. We've actually been working with container technology since before 2010. A lot of folks don't realize that. We bought a company called Makara back in 2010 because we saw that we needed to have a platform as a service offering. The Makara acquisition was eventually rebranded to OpenShift, which is our container offering today, our Paz offering today. We started doing containers except we called them cartridges using SE Linux control groups and kernel namespaces which should sound a little bit familiar if you're working with containers today. In about 2013 though, Docker really started doing some amazing work. And in the true spirit of meritocracy and open source, we kind of realized that, holy cow, this Docker thing has really taken off. We had been doing some contribution to it. We really ratcheted up our contributions to Docker. And last time I checked, and I haven't checked in probably a month or so, we were the number two contributor behind Docker to upstream the Docker project. And industry adoption of Docker is phenomenal. Docker's been through multiple successful venture capital rounds, AppSara, Cisco, EMC, et cetera, et cetera, et cetera, et cetera, including us have all invested and worked on standardization of containers with Docker. Even Microsoft has announced that they will support Docker containers. So, what are containers? At a very high level, containerization specifically Docker is a technology which allows for applications like web services or database services, application services to be run abstracted from and in some ways isolated from the underlying operating system. So for instance, Docker service can launch containers regardless of the underlying Linux distro, which is very cool. We've had the promise of software abstraction right once run anywhere for a long time. And it's kind of worked, but with containers, we're getting a whole lot closer, I think. Containers can enable incredible application density since you don't have the overhead of a full OS like you do with virtualization. And Linux control groups allow for really, really impressive utilization of the system. Control groups are not only, and I'll talk more about them in a little while, control groups are not only about stopping a process from taking over the system, but control groups are also about carving the system up into a little bite-sized pieces to get the best utilization possible. And the same container can run on different versions of Linux. Ubuntu can run on Fedora. CentOS can run on RHEL. Human sacrifice, dogs and cats, living together, mass hysteria, or at least really cool things for developers to do to roll their applications out. So what are containers not? Now, containers are not a panacea. They are not the cure for all that Ilsea, or all that Ilsea. And they are certainly not yet anyway a fit for every application. We see folks that are like, oh, we should run that in a container. Sometimes it makes sense, sometimes it doesn't. If you are beholden to third-party ISVs, for instance, if you're running big enterprise databases, or ERP applications, or something like that, those vendors may not yet, and probably won't yet support those in containers. So like I said, it's not necessarily a panacea for everything that you wanna do. Containers are not virtualization. You can certainly run containers in virtualized environments as you can run containers on bare metal machines, which I do all the time. So I do get questions periodically that you can kinda tell by the way that someone's asking the question that they're kinda thinking in terms of virtualization, that's really not what it is. So let's talk a little bit about container security. Containers use several mechanisms for security. And it's a layered approach. It's kind of the old onion idea. You got multiple layers of security, multiple ways of keeping bad guys out of it, and let's face it at the end of the day what we really want is for the bad guy to go, this is too hard to target, I'm gonna go next door. So Linux kernel namespaces, Linux control groups or C groups, the Docker daemon itself has security built into it. The Linux kernel or Linux capabilities, Libcap has the ability to limit activities or limit privileges that root processes can run, and then other security mechanisms like App Armor or SE Linux. I know SE Linux, so that's what I will talk about. So let's talk about kernel namespaces. I've had a lot of conversations with folks when you have sort of the stock conversation about, well, how do we secure it? You talk about kernel namespaces and you've got mountain namespaces and PID namespaces and user namespaces, and you get the kind of a blank nod and people are like, yes, but what does that mean? And so we'll talk about that and I'll show you some examples of what these mean. So namespaces are just a way to make a global resource appear to be unique and isolated and the namespaces that Linux kernel manages are mountain namespaces, PID namespaces, UTS, IPC, network, and user namespaces. And let's talk about how those look. So with mount namespaces, what this allows a container to do is the container will think that a directory that it has access to is, which is actually mounted from the host OS is the exclusive domain of the container. So for instance, when you start a container with the dash V, the path on the host, and then the path inside of the container and optionally read write or read only argument, you can mount a directory that exists on the host within the container. The container sees that directory in its own mountain namespace and doesn't know that it's actually on the host. So the cool thing about that is if you have any sort of shared resources that you want multiple containers to have access to, especially if you wanna make sure that that content is not gonna be modified by the containers and it's gonna be identical across all the containers, instead of trying to copy it a whole bunch of times, you just mount it one time or you make it available to the containers so that it's always the same in every container. And as an example, what I've done here is on the host, I cat var www HTML index.html. Yep, that's my silly web page. I use docker run dash it with the dash v argument and I say take the file system on the host, put it there on the container, run an instance of fedora and my executable is bash. So now you can see that my prompt changes from my user prompt to a root prompt so I'm now inside of that container. And if I cat that var www HTML index.html file, I see the same content inside of that container. Now the cool thing is, this is just inside of one container.  And depending on how I mount that, whether I mount that read write or read only and you should do it read only, then the content of that file is going to be immutable and within the container, you're not gonna be able to damage that content. So you can make content available to containers and make it read only so that the person operating the container can't do anything about it. And because we're on kind of a tight schedule, I'm gonna move a little bit quickly. So the next namespace that I wanna talk about is process ID namespaces. So pid namespaces really just let the container think that it's its own contained instance of an OS. It's its own instantiated operating system. So when you start a container on a host, it's gonna get a new process ID. Pid namespaces enable the container to see the pids inside of the container as if they were unique, as if it was a new instantiation of the operating system. So in the following example, I launch a Fedora container running bash and I run the PSAX command. And what that looks like is, when I run Docker run IT Fedora with the executable bash and I run PS, it thinks that that bash instance is process ID one. It's the first process. It's a self-contained operating system. But then if I open up another console on my host and do a PS, and actually, there's a typo on here. I apologize for that. I should have gone further up into the Docker process because I did, where'd I do it? Where'd I do it? I accidentally copied that process ID number and I shouldn't have. It's actually the bash command that we're talking about is that one right there. So when I show you that, that bash instance on the host, that's actually process ID 18596. That's just that isolation or that abstraction of those process IDs within kernel namespaces. So I also want to talk about username spaces. When you start a container, assuming you've added your user to the Docker group or however you've done it, it started as your user account. So I start the container in this example as my user T Cameron. But immediately once it started inside of that container, I'm user ID zero because of that namespacing. So you can see I run the ID command and I am T Cameron. I run Docker run dash IT fedora with the executable bash, but my ID changes to UID zero. Now, am I root on the entire system? No, I am only root inside of my container. I still need to pay attention and do smart things because I am root inside of my container and I can do all the silliness that I want to. But this is an example of that UID namespacing where a non-privileged user, not through SU or anything like that, but just through the privileges that are granted from within the container, can have elevated privileges inside of the container through that UID namespacing. Now networking namespacing or network namespaces, pretty cool capability where basically it allows the container to think it's got its own IP address. It's not gonna be in the network range of whatever your physical interface is. In fact, the Docker service will set up IP tables, masquerading rule to make sure the container can get out to the rest of the internet and in this example, let me just go to the page, I use Docker inspect to take a look at the network settings IP address within the container that I've fired up and I get this address as <IP_ADDRESS>. When I do IP add or show on my interface, my interface isn't even connected to a network so I don't have an IP address at all. That address, that network namespace address gets set up and then the Docker daemon is smart enough to set up IP tables masquerading so that inside of my Docker container, I can get out to the network and pull things down and stuff like that. It doesn't necessarily, depending on how you set it up, it doesn't necessarily grant access to that container from the outside directly because we are doing masquerading, but it's going to segregate that container off from the rest of the network stack on the host and make it safer. Interprocess communication namespacing. Our IPC namespaces, same thing with interprocess communications. Essentially, it just abstracts them out so that within a container, for instance, I can do IPCS and nothing. Inside of the container, the thing thinks I'm my own operating system, I don't have any IPCS stuff running or any processes running or anything, really. It's not doing anything, but if I go to another console, see right there, I'm the root inside of my container, but if I go to my main console on the host and I do IPCS, I've actually got page after page after page after page after page after page of interprocess communication mappings going there. So again, all this kernel namespacing is doing is isolating the container so it thinks its own unique little world and it doesn't know about anything that's going on in the host. All right, UTS or Unix Time Sharing System namespaces allow the container to think it's its own separate OS with its own host name, own domain name, and so on. So if I look inside of, or if I look on the host and I run the host name command, that is my actual fully qualified domain name for my laptop, t540p.tc.redhead.com, but if I fire up a container, you notice that it changes, I've got the root prompt there, and when I run host name in there, it thinks that it's this randomly generated, you guys have all heard the pets versus cattle analogy. This is a cattle name, right? It's just a serial number almost for the host name for that container. It thinks it's its own instance, its own network name, and it's segregated from the activities of the host. All right, so that is kernel namespaces. Again, the whole point of kernel namespaces is really just to let the container think it's doing its own thing, it's segregated from the rest of the networking and rest of the capabilities of the host so that if anybody does anything bad to it, it's isolated within that container. Now, control groups allow for some really cool, fine grained control over resource utilization on the host. So, there's a ton of really good documentation about control groups in the kernel cgroups.txt file, and really, all it allows you to do is to aggregate and partition sets of tasks and their future children into hierarchical groups with specialized behavior. This allows us to put various system resources into control groups and apply limits to it, like how much disk I.O., how much network I.O., how much memory you can use, how much CPU you can use, so you can have some really fine grained control over what's going on. In the case of, for instance, the Red Hat Atomic Server, for instance, which is our container, specialty container server, we actually set up control groups that are very fine grained for every set of containers that gets kicked off. And basically what that does is it ensures that even if an individual container is compromised, they still, whoever takes over that container, or even if somebody just has poorly written app and their container just spins out of control, you've got some Java job or some craziness going on or somebody that does a fork bomb, because someone always has to do that to prove something, does something inside of a container, we're gonna limit it to just taking down that container. So, for instance, when I run the command system control status docker.service, I get the control group and slice information. So, you can see, when I run, again, system control status docker, there's my control group and it is in its own control group, including SC Linux, which we'll talk about in a little while. So, even if a bad guy does something silly to it, you can apply rules to that control group that say no more than 10% CPU or no more than 10% network or 5% network or whatever. You can look through the sysfscgroup pseudo directory to see what resources are allocated to your containers. Now, there are over 8,500 entries in that directory, just on my little laptop. So, it's not practical to be able to dive into the depths of it, but essentially, you can look inside of there and get information about, again, memory, CPU, block device, IO, network IO, and so on in that environment. I just showed that when you go to the sysfscgroup directory and do a find.pipewc-l, there's like 8,500, almost 8,600 instances in there. All right, so, we've talked about kernel namespaces, we've talked about C groups, let's talk about the features of the Docker daemon itself. So, the Docker daemon, user bin Docker, is responsible for managing the control groups, orchestrating namespaces, and so on, so that these Docker instances can be spun up and secured. Because of the need to manage kernel functions, Docker runs with root privileges, do you be aware of that? Now, at Red Hat, we don't enable the ability to do things like, if you read the Docker documentation, they say to create a Docker group and add your user to the Docker group, and once your user's a member of the Docker group, they can run Docker from the command line. We actually don't enable that, we would rather see you actually run things as root and let the Docker daemon drop privileges. So, there are some considerations for running Docker. Only allow trusted users to run Docker. I recommend that, so let me back up. The Docker documentation recommends that you add users to the Docker group so they can run the Docker commands. That's fine, but just be aware that there are some risks associated with that. Make sure you only delegate that ability to trusted users and remember that they can do things like mount the host file systems and potentially do bad things to you. So again, we recommend that you actually only grant privileges to the users that you want to by using a mechanism like sudo, for instance, so you can create your Etsy sudoers file and say that they're only gonna have access to you certain Docker commands and things like that. If you're using the REST API to manage your hosts, make sure that you don't have any vulnerabilities exposed. In other words, keep your systems up to date. It's kind of common sense, right? But a lot of times we get something working just the way that we want it to and then we're like, don't touch it. So, don't do that. Make sure you keep your systems up to date and make sure that you're using strong authentication if you're doing things over REST. If you're going to use the REST API over HTTP, please make sure that you're using SSL or TLS. Don't expose it unless, except by secured networks or maybe by VPNs. So, Linux kernel capabilities or libcap. Really cool technology which allows you to essentially set limits on root privileges. Historically, the root user has had the ability to do anything anywhere to anyone, right? Once you're authenticated, I am root bowel before me. But Linux capabilities is a set of fine-grained controls which allow a service or even users with root equivalents to be limited in their scope. So even root users can be cut down on what they're able to do, but you can also use Linux capabilities or libcap to grant regular users the ability to have elevated privileges without having to do SU or anything like that. For instance, a user could be granted the NetBind service capability and they could bind a service to a privileged port, in other words, a port below 1024, even if they're not running as a root user. Now, in containers, a lot of the capabilities to manage network and other services not actually needed. For instance, SSH services, Cron services, file system mounts, things like that, really not needed from within the container, typically, except for SSH, you never need to run SSH in your container. Don't use SSH in your container. It's dangerous, it's silly, it's gonna get out to date and people are gonna do bad things to you. So, a lot of these things are not needed. By default, Docker does disallow a lot of root capabilities including the ability to modify logs, change networking, modify kernel memory, and the catch all capsys admin, which I'll show you some more about in a little while. So, if you look at, and I'm sorry, this is kind of an eye chart for those of you in the back, this is the libcap page on GitHub, and man, that came through horribly badly, I'm sorry. It looks really good on my screen, but basically, I'm not gonna go into it, but this is a table of all of the capabilities, all of the root capabilities, which can be managed by libcap. So, I've talked about a lot of them, networking capabilities, changing kernel memory, and stuff like that. But if you look through the libcap GitHub page, it's actually really informative. It's really cool to go through and see all of the things that you can limit, and this is actually the Docker filters which use libcap, so you can go through and see all of those by looking at that page. And again, you guys will get this presentation at the end, so you're welcome to go follow that link. All right. So, one of my favorite topics. I actually present about SeLinux on a pretty regular basis, I really enjoy it. I've got a YouTube video called SeLinux for Mere Mortals. If you don't like SeLinux, that's cool. Go watch, give me 50 minutes, go watch SeLinux for Mere Mortals on YouTube, and I think I'll probably change your mind, or at least I'll make it so you don't hate SeLinux quite as much. SeLinux is a mandatory access control system. Processes, files, memories, addresses, network interfaces, and so on. All have labels that are maintained by the kernel or as extended attributes on the file system. Everything's labeled and there's a policy which is administratively set and fixed. So, the policy is gonna determine how processes can interact with files on the file system, how processes can interact with each other, network ports and things like that. It's a really cool technology. It can be a little bit complicated, but the thing about SeLinux that I tell folks is, SeLinux is really only about two things, labels and type enforcement. So, for instance, if I have the mythical service foo, the foo service, the executable on disk might have the SeLinux label foo underscore exec type or underscore t. The startup scripts might be foo underscore config type. The log files, foo underscore log type. The data may be foo underscore data, right? It's actually fairly intuitive. When you're doing SeLinux, it's all about labeling, right? When the foo process is running, it may have the label in memory foo underscore t. So, that's labeling. Type enforcement is just the rule that says, if I explicitly allow the foo exec type, for instance, to access the foo config type files, then when the foo service starts up, it can read its config files. Then when I set a policy, it says, oh, yeah, the foo exec type can also write to foo underscore log types. Again, that's fairly intuitive, right? You want your process to be able to write to its log files. But type enforcement says, unless I've explicitly allowed it, I'm going to deny it. So, for instance, any other access unless explicitly allowed by the policy is denied. So, as an example, if the foo process running in the foo type context tries to access, for instance, the directory slash home slash t camera which has a label of user home dir type, even if the permissions are wide open, even if I have done chmod 777 on my home directory, right? We'll give you the gun and point you to your foot. We'll tell you how to do it. But SE Linux will step in and go, no. Unless it's explicitly been allowed, I'm going to deny it. So, SE Linux is really cool. It can save your bacon in the event of misconfigurations. I've seen it happen. When I talk about type enforcement and labeling, when I talk about the labels, the labels are usually stored or are stored in the format of the SE Linux user, the SE Linux role, the SE Linux type, and then, optionally, MLS and MCS labels. So, for that mythical foo service, the full syntax for the label the running process might be the user user object role foo type and then we can have MLS and MCS labels as S0 and C0. Now, when we're talking about SE Linux, the default policy for SE Linux is the targeted policy. In the targeted policy, we really actually don't care about the SE Linux user or the SE Linux role. We really care about the label, because remember, it's all about labeling and type enforcement. So, we can also ignore the MLS or multi-level security labels since that's really only used in the MLS policy, which is usually only used in like Department of Defense or CIA or places like that. We really only care about the type and the MCS label. So, think of the MCS labels really as just extra identifiers and the reason that's important is we can use this in containerized environments to provide very fine-grained control between containers. So, for instance, these are totally different labels. I got userU objectR foo type S0 C0 and then down here, I've got S0 C1 even though those are identical, except for just that MCS label. From an SE Linux perspective, they may as well be completely different as black and white or whatever. Type enforcement says that the process with that first label is different from the second one, so policy would prevent the two of those from interacting. Also, there's no policy allowing the process running those labels to interact with the file system unless it's labeled a foo config type or foo content type or another predefined label. So, if one of those processes, for instance, was compromised and it tried to access a file on the host, let's say Etsy Shadow, which has the label shadow underscore T, by default, SE Linux, if it's not explicitly allowed, it would be denied. So, on a standalone system running Docker, for instance, all the containers run in the same context by default. If you look at OpenShift, for instance, or Atomic Platform, that's not the case. Each container actually runs in its own context. You can do that on a standalone laptop machine. You'll have to tweak your Docker command to do it, but you can absolutely do it on a standalone machine. But as a, for instance, I've got three instances running. They're all running in the OpenShift context or with the OpenShift label, I should say, but you've got different contexts here. So, even if somebody were to access the Docker container process on the host, even if they compromised the Docker process and they got into one of your containers, they still would not be able to access the other containers on the machine. So, what I'm gonna do is, I'm gonna show you an example of a simulation of somebody exploiting your Docker environment. So, what happens is, on the first line, I'm logged in and I have the context, let's, we really only care about the unconfined type, S0 to S0 and C0 to C1023. So, when I run the ID command, you can see that is my SE Linux context. What I'm going to do is, I'm logged in as root. Root is omnipotent. I could do anything on the system that I want to as root, right? But what I'm gonna do is, I'm gonna use RunCon to change my running context and I'm gonna change over to the OpenShift T label with S0, C0 and C1 and I'm gonna run the bash command. I am still root. All I've done is I've just changed my SE Linux context and the funny thing is, as soon as I run bash, it goes, whoop, permission's denied to bash RC because I'm no longer in the right context. If I try to cat Etsy shadow, for instance, even though I am root, permission denied. If I try to touch a test file in the root of the file system, even though I'm still root, I've just changed my SE Linux context, permission denied. If I take a look in the home directories, even though I'm still root because I'm no longer in that correct context, I have changed over from the unconfined context over to the OpenShift context and OpenShift does not have SE Linux access to home T Cameron, I immediately get permission denied. I'm root. Well, as root, that's easy. All I need to do is just disable SE Linux, right? Nope. If I try to run set and force zero because I am no longer in the right context, SE Linux will see that and go, nope, permission is denied. So set and force failed. So SE Linux is an incredibly powerful capability. The things that I've talked about previously, obviously all really, really important. Kernel namespace is really important. Control groups for keeping compromised systems from taking over or compromised containers from taking over your system, really important. My humble opinion, not that I'm biased or anything like that, because I've never present on SE Linux or anything, but my personal opinion is SE Linux is really the linch pin to security in a containerized environment. So let's talk about some tips and tricks. Containers are, at the end of the day, just processes running on the host, right? I mean, containers are not magic. They're cool, but they're not magic. So some of the things that you do want to do in a containerized environment do have a process in place to update your containers. Follow it. It is so easy. I get it. It is so easy for a developer to come up with something and it's like, hey, I got it to work. It's working perfectly. We're gonna throw that bad boy out in production and then I'm gonna move on to the next project. It happens. We know that, but have a process in place to update your containers and follow it. Run services in the containers with the lowest possible privilege. Drop root privileges as soon as you can, whether it's web services, database service, I don't care, Bill and Ted's excellent service. Make sure you drop privileges. Use services that allow you to do that. Mount file systems from the host read only unless you absolutely positively have a good reason not to. Treat root inside of the container just like you would on the host. Watch your log files, pay attention, and don't just download any old container you find on the net. Bill and Ted's excellent container repository may have some cool stuff in it, but unless you've vetted it and you know what's going on with Bill and Ted, you probably want to be real cautious about downloading them. Please don't run SSH inside of the container. Use the system management tools of the host or use Git or something like that. Please don't run SSH. Don't run with root privileges unless there's absolutely positively no other option in which case find another container or use another piece of software. You shouldn't run with root privileges. Don't disable SE Linux. If you really think you need to disable SE Linux, go watch SE Linux for mere mortals and then send me an email, thomasatredhat.com. I am available. I will talk to you about SE Linux. Don't disable SE Linux. Don't roll your own containers once and never maintain them. Again, have a policy in place to keep those things up to date. And again, don't run production containers on unsupported platforms. It's a shameless little marketing plug from Red Hat there, but you really want to have a certified platform or you're gonna be able to pick up the phone and go help if something bad happens. So in conclusion, go forth and contain stuff. Containers are awesome. I really, I've been doing this for a long time. I'm the first one to admit that I'm pretty jaded, right? You know, we always hear every year, this is the next new big thing and it's gonna be awesome and it's gonna change everything. And after about 10 years in the industry, you're like, yeah, okay, whatever. Containers are pretty cool. I don't know if I'm gonna say that they're gonna be like a tectonic change in everything we do in IT, but containers are pretty cool. They make app deployment really, really easy. They leverage some incredible capabilities of the underlying operating system. And by design, they're pretty secure. They can be secure if you maintain them well. There are some gotchas though, as with every other piece of software out there. It requires some care and feeding, right? You gotta take care of your systems. But well maintained, containers will absolutely make your business more agile, less complex, hopefully, and if done right, safe. So thank you very much for coming. I appreciate it. We'll open it up for any questions. If anyone's got any questions, I can't see anything right now because I've got these flamethrowers in my face. But if you have a question, please go up to the microphone and I'd be happy to answer them if I can. Somebody's gonna stump me though. It always happens. Yes, sir. When you used Run-Con to change your context, would you have been able to use Run-Con again to change it back? Nope. That was a one way trip. That was just for demonstration purposes. Hey, look, here's your foot. There's the pistol. Yes, sir. Hey there, thanks for the talk. I'm Rich from Cloud Engineering at Box. I'm just curious, what work has Red Hat done, if any, around regulations like FedRAMP and PCI and container configuration? We've actually done a whole lot of work with getting the atomic platform certified for common criteria and things like that in conjunction with the folks in the DOD and folks like that. Like, as far as specific projects and stuff, I don't have that right here with me, but yes, we are absolutely aware of the requirements for that and we're working with the federal government to make sure that we are at least pursuing if we haven't already received a lot of those certifications. Okay, thanks. Thank you. Hey, just by the way, real quick, guys, if you need to reach me, I don't think I put it on my slide. I am thomasatredhat.com. You can follow me on Twitter at thomasdcameron. If you have any questions, don't hesitate to follow up and these slides will be available on the website when we get done. Yes, sir. What do you think is missing to get better multi-tenant security from two different containers on the same system? What is missing? That's interesting. There are a lot of things that we need to get better about. There are a lot of things that we need to get better about around just doing simple stuff like enforcing security, enforcing updates within containers. I think some of that glue, the plumbing around that is probably something that the industry in general is weak at. From a kernel standpoint, do you think there's any capabilities right now that you're missing or do we still need to run different VMs to keep tenants separated? I think that, no, actually, I think that with containerization we're doing a lot better. If you look at what Docker specifically and what Red Hat as a contributor is doing around Libcap, that is changing almost weekly it seems like. So I think that what we're really having to do is spend a lot of time taking a look at what capabilities are absolutely positively needed and weeding out the rest of them. I think that's probably where I'm seeing the most activity and then also SE Linux policy and doing things like SE Linux segregation on like what we do with OpenShift. I would like, I'm lobbying internally at Red Hat to make that available for every instance, every place that we use Docker. It's not there yet, but we're working on it. Thanks. Thank you. Yes, sir. So you mentioned more than once to not use SSH. Can you elaborate on that a little bit? Why not and what would you use in place of that? So the problem is, again, SSH is like an open door way to the world potentially. You know, if you secure it correctly and you use keys and no passwords and stuff like that, it's better. But the thing is, if you need to have access to your systems, do it, don't have a million instances of SSH running. If you need to log in to the host, SSH into the host and then make whatever changes, do a Docker attach or whatever. But running SSH inside of the host is an invitation for disaster because invariably what's gonna happen is you're gonna have an old outdated version that has security holes in it.  So I'm gonna pay attention to the application and I'm gonna update the application, but they'll forget to do the SSH daemon and you just eventually wind up shooting yourself in the foot. So the same could be said about any system, not just containers? Yeah. I mean, yeah, if you're not maintaining it, the thing, I think the reason that it's, I don't know if I'd say critical, but I think the reason that it's so common in containerized environments is you have app devs, and even if the app devs are being really smart about keeping their Java or Node.js or whatever up to date, they're not sysadmins, right? So they don't even think about the SSH daemon. The guy that they got, they tugged on the sleeve to set that up, he's not involved in it anymore and so that's where you see that kind of thing slide. Okay, how are we doing on time? Do I need to get down? Two minutes, okay. Hey, thank you very much for the presentation. Thank you. So to the previous question around SC Linux and security, so what additional security capabilities do you get by running within a VM or do you believe that containers with SC Linux are truly, that they do not require VMs to be truly secure?  It may make sense for you to spin a VM up for them so they can spin up those 50 containers. You've got your control group set up inside of that VM. Maybe you're even using control groups and the underlying hypervisor to make sure that that VM doesn't lose its mind. So potentially there are gonna be cases where it absolutely makes sense to use virtualization in addition to containerization. I think as we get more sophisticated with containerization, I think, don't hold me to it, but I think we're gonna see less of a need to have segregation at the VM level. I think we're gonna get to the point where you're gonna be able to have these big honking hosts and just spin up zillions of containers and apply security through C groups and SC Linux and so on so that you don't have to have those multiple layers. Does that make sense? Yeah, thank you very much, appreciate it. Thank you. Okay, one last question. Is there any development around the isolation of containers from a networking point of view? I mean, you can totally disable the communication between containers, but I haven't seen how I could be more granular, let's say, I lost some communication, but not any time. Yeah, so we're looking at it, but then there are also some third party tools like, oh, crap, I was just talking to him last night. He's a former Red Hatter and I'm drawing a blank as to his company name, but they're using Quagga. They're using Quagga to do dynamic routing and dynamic networking so that you can get all the way down to the individual container layer and set up really harsh, strict rules that say, this thing can only get out to the internet and I can only get back in and they can't see each other. So it's not just Red Hat, it's not just Docker. There's a ton of folks who are working on that. It's clearly a gap and there are a lot of people who are trying to figure it out. Open source, hopefully the meritocracy will be that the best one will rise. Okay, thank you. Was this helpful? Was this good? Okay, good. Thank you very much. Thank you for coming. I appreciate it.


Hello everyone, welcome to Windows Server Summit 2024. My name is Aakash and I hope you are excited to hear what we have to share around System Center today. Well, the cats out of the bag now. Today we are proud to announce that the next version of System Center, that is System Center 2025, will be released later this year. This is in continuation of the ongoing System Center investments that we have made a commitment to each of you. Before I jump into all the details, I will quickly run down the agenda for today. So today we have about 25 minutes to walk through these topics. We are going to reiterate what System Center means to each of us, share roadmap about all the version of System Center products, then we get into the crux of today's meeting by talking about System Center 2025 and what it entails. So let's jump into it. For those of you who don't know me, I am Aakash Basraj and I work as a product manager at System Center Product Group. I've been in System Center business for about four plus years now and I'm joined by my colleague Kartik who will talk about his favorite topic, data center management using virtual machine manager. Thanks for joining, Kartik. So let's get started. This is a quick overview of System Center products and their use cases. System Center has been the workhorse data center management tool for more than 30% of Windows Server customers over the past couple of decades. System Center's main target is for on-premises data center management and it will remain to do so for upcoming releases of Windows. What has really clicked is the close partnership that we maintain with Windows Server team ensuring support for all latest versions of Windows and its latest capabilities. Without draining further into each product, I want to address that System Center suite will be released with every new major version of Windows Server allowing you to install and manage them easily. Specifically, these six products are unmatched when grouped together and we appreciate you, our System Center users for your support and usage so far. While we continue to support and provide meaningful enhancements to these products, we want to help you transition to cloud management services such as SCOM Management Instance and others allowing you to read the benefit of Azure and its evergreen services. More on this in upcoming slides. I want to share a quick update on the existing LTSC releases that is System Center 2019 and 2022. We will be releasing UR6 for System Center 2019 to ensure your System Center 2019 environments are up to date with security fixes as the product is going out of mainstream support by end of April. UR6 will be released later this month. Additionally, System Center 2022 will continue to be supported for the next three years followed by extended support. So we will provide product updates and patches for System Center 2022 as mentioned in the slide. However, the big announcement for today is the System Center 2025 that will be released in GA version in the latter part of this year along with Windows Server 2025. We heard your strong feedback from last time where we delayed the release of System Center 2022 by six months from that of Windows Server 2022 and it impacted the users by preventing them from using Windows Server 2022. So this time around we will ensure to release both these products together with high quality. So what is new in System Center 2025? Talking about, talking from investment perspective, these are the three main focus areas. Our primary focus is for us to provide you with best-in-class data center management tool that can manage your heterogeneous infrastructure which are based on the evolution of data centers that are undergoing B to the cloud or updates to the hardware systems. As mentioned earlier, System Center 2025 will support installation and management of Windows Server 2025 servers and based on your feedback again, we are going to support management of Azure Stack HCI infrastructure, specifically Azure Stack HCI 23 H2. So keep your feedback coming in so that we can adapt accordingly. Moving to the next pillar, as you may have seen in Windows Server 2025 announcements that one of the big lever that we are pushing is that of security. As the data centers and applications evolve towards hybrid state, we want to ensure that the management tools are compliant, safe and secure. In System Center 2025, we plan to reduce the usage of NTLM and credit SSP as much as possible and use alternate authentications such as Kerberos and Open Authentication wherever possible. We plan to add support for TL S1.3 to ensure secure communication between System Center servers and management clients. Additionally, we will support Gen2 VMs. We will talk about Gen2 VMs and its benefit in upcoming slide. Lastly, as you evaluate and plan to reduce on-premises footprint, we want to support infrastructure modernization. You may have seen earlier in SCOM 2022, we added discovery of Azure Migrate, helping you evaluate cost benefits of moving to cloud while modernizing your management services. We will continue to invest in similar areas. For users who cannot migrate to cloud right away, we will help bringing cloud services to System Center via ARC, specifically ARC for SAVMM. And last but not the least, System Center, SAVMM specifically, will provide capabilities to convert your private cloud software from VMware to alternatives such as Hyper-V and Azure Stack at CI. This ensures business continuity with reliable support and modernization of data centers. That said, let's get into the finer details of what is new in specific products. We will start with VMM and my colleague Karthik will walk you through the updates. By delivering System Center 25 closely with Windows Server 20.25, we are bringing you management support for the latest Windows Server version right from day zero. Understanding our customers' preferences to manage their heterogeneous infrastructure with SCVMM, we are adding support for management of Azure Stack at CI 23H2 clusters from SCVMM 20.25. Customers will also have an enhanced experience while converting their VMware VMs to Windows Server or Azure Stack at CI through SCVMM. SCVMM 20.25 also brings significant security enhancements as all virtual machines created through SCVMM will default to Generation 2, which provides faster unsecured routing. With SCVMM 20.25, customers will also see a reduction in the number of scenarios which use credit SSP and NTLN as their authentication mechanisms. As customers' data centers evolve, so do our solutions. With the general availability of Azure Arc-enabled SCVMM from November 2023, customers can now use Azure management services like Microsoft Defender for Cloud, Azure Monitor, Azure Update Manager, etc. to secure, govern, monitor, and configure their SCVMM managed VMs through Azure Arc. Customers will also get Arc-enabled capabilities of Windows Server 20.25 like hard patching for Arc-enabled SCVMM VMs right from day zero. The end-of-life Windows Server 20.12 or two VMs can continue to be in supported state with extended security updates procured via Arc-enabled SCVMM. Azure Arc-enabled SCVMM also gives the ability to perform lifecycle operations such as start, stop, create, scale, resize, and delete your SCVMM managed VMs that are hosted in customers' data centers. All these activities are governed by Azure role-based access control enabling customers team to perform VM operations in a self-service fashion. This will serve as an effective replacement to end-of-life approaching Windows Azure back. With System Center 20.25, we are also discontinuing Azure Profile's future of SCVMM and System Center's service provider foundation, SPF, as these capabilities are now being built into Arc-enabled SCVMM. Thanks, Karthik, for the update on Virtual Machine Manager. Moving on to Data Protection Manager. With a steadfast commitment to improving workload coverage, Data Protection Manager 20.25 ensures comprehensive management support from day zero for Windows Azure 20.25, integrating seamlessly with the SharePoint subscription edition and introducing features like virtual DPM support for VMware. A DPM 20.25 brings the capability to exclude specific disks from backups in Hyper-V environments, offering flexibility and efficiency in data protection strategies. At the core of DPM 20.25 innovation is a heightened emphasis on security enhancements, recognizing the paramount importance of safeguarding sensitive information. DPM 20.25 introduces the capability to securely store the passphrase in Azure Keyword, significantly increasing data security by leveraging Azure's robust cloud infrastructure. Furthermore, it embraces the latest in encryption technology by supporting TLS 1.3, ensuring that all data transmissions are protected by the security standards available. The adoption of support for OLED between 19 further underscores DPM 20.25's commitment to security. That said, these are some of the finer details that we are looking to share currently as part of our 20.25 plans. Let me give you a little bit more information on what is coming next. So we are planning to release systems in our 20.25 by end of April in the preview version. So we plan to release virtual machine manager and data protection manager in the preview version based on the feedback that we have received and based on how the validation has happened. We will take those features or take those enhancements to 20.25 GA. Having said that, we will be releasing all the six products, which is Orchestrator, OSCOM, DPM, VMM, Service Manager, SMA, all of them we are planning to release in 20.25 GA and then we hope to continue supporting with enhancements to these products. That brings us to the end of our session today. I really thank you for the time you've shared and the feedback that you constantly give on systems and our products. For any feedback, I request each of you to drop a note to systemscentralfeedbackatmax.com and you can get in touch with me at LinkedIn as well under the same name.