Hello, and welcome to Build Your Contributor Pipeline with CNCF SIG Contributor Strategy and Josh and Carolyn. We're here to explain something that a lot of projects have been asking about lately, which is how do we get contributors? How do we get new contributors for our projects? My name is Josh Berkus. I'm a community architect at Red Hat in the open source practice office where I work on the many cloud native projects that we support at Red Hat, plus a few that are technically supported by Red Hat, but I help in my role with CNCF anyway. I'm Carolyn Vanslike. I'm an engineer at Microsoft and I work on the CNCF Sandbox Project Porter. So before we really get into advice and things like that, I wanna just stop and make sure that we're being intentional and understand what we're hoping to get out of this. When you say you wanna build your contributor pipeline, what are you trying to achieve? Do you wanna create a community around your project? Are you trying to find that next special maintainer? Or are you looking for maybe adding more people who have diverse skills and perspectives that your project is lacking? Depending on what you wanna do, this could be a pretty big time investment. So you wanna make sure that it's not just you trying to figure this out and welcome new contributors to your project. You want all the other maintainers on the project bought in and understanding what your game plan is. So before we get into attracting contributors, so let's think in general about where contributors come from, new contributors for open source projects. Why do they join to begin with? Usually though, we've got three sort of different groups that come to projects with the goal of contribution or participation. The first group that people are the most familiar with is individual users. These are people who use your project or they use your project to build other things. And sometimes they're just really excited about the project. They wanna be a part of it, they wanna participate. Other times they have something specific that they want to add or to fix that's broken for them in their use case. Usually these are what we call power users who are using it in production, who are using it in some very advanced way. However, in the CNCF, we actually get a lot of a second kind of user so what we call corporate users, which means that rather than the individual using your project, their company uses the project and depends on it. And for that reason, the company has assigned them as an employee to work on the project. Either again, to add some functionality or to fix something that specifically matters to the company or if you're extremely lucky to maintain help maintain the project in order that the company can depend on it long term. Now a third group that people often forget to think about which is bad because this group is unique in a lot of ways is what we call learners. Maybe they wanna learn the programming language, maybe they wanna learn the field of technology that your project is in. Maybe they're about to graduate school and they wanna beef up the resume. The nice thing about this is these people will contribute to areas of your project that are fairly non-commercial because they don't care. And so you should cultivate them if you can. So let's think a bit about what type of user are you focusing? You know, we think you're not where are they coming from? What are their motivations? So what's the pool of people available to us? The most obvious, maybe the first place you should look for if you haven't already is look at your existing user base. These are people who are opening bug reports with you. They're already engaging maybe in Slack. They're pretty much maybe the best well to go to to find new people. But if you're trying to look even further, you can consider recruiting people who don't actually know your project at all. The learners especially fall into this group. Another thing you can look at is, are you lacking contributors with specific skills? Like raise your hand. Everyone needs a tech writer. You probably need someone to help make your website look right, create concept art diagrams. You know, someone to help with that roadmap for project management. And what a lot of these have in common is that they aren't code contributors and they're not coming to you and submitting a polar quest. You need to go to them and think about what you can, what they can bring to your project and make sure that they know that it's welcome. So people can write documentation. They can give you user stories and feedback on your roadmap and designs, right? A guest blog post explaining how they're using your project. A lot of these things are gold, but people won't give it to you unless you say it's welcome. Now some people may be looking for that unicorn, an experienced maintainer who has already been a maintainer on another project and you need somebody who's a superhero essentially to help you out. But that's not even worth looking for. Those type of people already too busy and have their own projects. So once we've identified what kind of contributors we want to attract and where we're going to be contributing them from, we actually need to construct a framework called the contributor framework in order to look at how we're going to attract these people. And I find it's useful to divide these up into enablers and blockers. Enablers are things that help you attract additional contributors. That is things that if they are present in your project if they're good will cause more people to show up and help people who have already shown up to stay. And you can further divide these down into things that are what we call low investment and things that are high investment. Low investment example would be hey, creating contributing that MD file or having a contributor forum, right? These don't take a lot of your time and energy to create and yet they really do help people become and stay a part of your project. And then there's a second group of things that are much more high investment but often also pay dividends in being really strong contribution enablers. Like for example, if you create a very detailed contribution tutorial going through every step of the way of, oh, hey, you want to add a feature to my project then that will enable your code contributors to self start without needing to have one on one mentoring in order to contribute stuff. And thus for you to get more of them and for them to be more productive once they do join. But creating that kind of a tutorial is hard and it will be a huge time commitment from one of your existing people. One of the other things that's often overlooked as a very simple enabler which is low commitment is recognition. So when somebody does contribute make sure they get noticed in some way. Digital badges, swag, here actually I've got, you can see right here this is my Kubernetes contributor summit Shanghai t-shirt. So special contributor swag, public thank yous and public contributor listings. People actually care a lot about being recognized even if they don't say that they do. Now, there is a bunch of things that can prevent even your best intention contributors from contributing to your project. And these are things that you want to try to eradicate from your project so they don't ruin your other efforts.  Not having any visibility into the project in a variety of ways. Governance, rights on repositories, tests. Open sourcing your test framework is hard, I understand. But on the other hand, somebody contributes they get a bunch of pre-submit test failures and they ask, hey, how do I troubleshoot this test failure? And you tell them, well, you can't. Then that contributor is blocked until one of your existing people gets around to troubleshooting it for them. And that's a bad experience for them and it's a bad experience for your people. Now, one of the biggest things about enabling contributors is communication and being as open as possible about communication. So if you're having a strategy meeting or a developer meeting or a standup or whatever that should be in the open. External contributors, even if you have it's a company-sponsored project external contributors should be able to join that. Public contributors should be able to join that meeting if they want to. Now, a lot of the times they won't want to. But they need to know that it's there.  Things are ideally also published in text because you want to make contributors, new contributors part of planning for the project which both makes them feel included and make sure that they don't spend a lot of time on things that are out of scope-free. Also, please, please, please don't make attending things like daily meetings mandatory for contribution because that limits your contributor pool to people who are being sponsored by companies to work on the project full-time. Now, when we've sort of been over this we've looked at the various ways you can communicate and there are some things that reach more people for less effort than others. And that starts with anything that you can have that's both written in asynchronous. So whether it's your documentation or a mailing list or a blog or a discussion forum of some sort if it doesn't require synchronizing time zones and if it's written, so people can refer to it in the future that's gonna reach more people more easily. After that, start looking at an open developer meeting because it's something you would hold anyway and then move on to more very contributor-targeted things like office hours or end-user forums that are just specifically contributors. One of the mistakes we see people making is they create this community meeting that is specifically just a community meeting and then it doesn't work because they really don't have any content for it and that should really be handled by things like your blog. So we're going to list everything that you can do to bring in and keep contributors in your pipeline but I wanna make it clear that that's not everything you should be doing and you definitely should be trying to do all this at once. Considering your project size how many people you have to support any recruiting efforts and then pick one, try it, see how it goes and focus on tasks that make you scale. Anything that involves one-on-one mentoring, live things, synchronous things, all that, save for later, start with all that written communication for example. And then over time, you can add more because what's important to understand here is that recruiting and growing your pipeline is not a one and done thing. If you add three contributors over time they're going to eventually leave, your project will change, you need updated documentation, your recordings get old, you're always gonna be doing more things. So commit to a small amount of work and find a way to be able to do it every couple months and refresh and make another recruiting drive essentially but it is not a destination you're never gonna be done with this. So let's consider how you let people know that you want new contributors. The obvious place is the contributing guide but there's actually a lot of other breadcrumbs you should be putting out there so that people know that you're actively looking for more contributors. The first place most people are gonna see your project is actually your read me. So that's where you're gonna wanna say, hey, we want you, come join us. Another place that I've seen people do it is they have a dedicated page on their project's website that talks about wanting contributors and kind of walking people through it. Another way to signal that you're looking for people and this one's kind of fun because you get a lot of people for free is labeling your issues as good first issue and help wanted because other projects, hackathons, things like that, Hacktoberfest will link to all the issues in GitHub and go look for ones that are good first issue, look for ones that are help wanted and you can actually put up on your contributing guide, link to these and give people ideas for where they can get started. Another thing is if you're given a talk, if you're doing a webinar, anytime they're pitching your project, let people know that you want help. So I mentioned you read me, this is an example from my project porters, read me here and we're unabashedly like we need help, come contribute, work with us and we link people to the new contributing guide. So let's focus just on helping people get started with the project for the first time. And then we also call out to our existing contributors.  I also have a page on my website that says, hey, we want you to work with us. For a lot of people coming to this, they may be unfamiliar with what your project is. So lay that out. Like point them to other places on your site, point them to tutorials, let them know where they should get started, let people know what are the types of contributions you're gonna welcome, especially if you're open to non-code contributions and let people know, is there a path to becoming a maintainer if you do this? Basically put up front all the recognition and rewards for working on your project into these documents. Now in your contributing guide itself, it's kind of doing double duty here, right? Oftentimes you're talking about build tasks, how to do certain things, what are your build scripts? But at the very top, what we like to do is let people know again that you're looking for contributors. People will find your contributing guide because someone will link to it and say this is the best way to get into contributing or every time you submit a poll request, it'll probably link to it at the top. So you wanna just always make sure that welcome mat is there and let them know the ways that they can contribute and then as always, it's welcome. So let's talk about labeling real quick. There are like common accepted default labels. Like if you make a new repo, you probably have a help wanted, you probably have a good first issue label, but there's so much more to making it workable. And your goal should be, if someone picked up this issue on the weekend, could they complete it without pinging people on Slack, getting lost, asking for help? Is there enough context in the issue that they can actually just accomplish it? Help wanted is kind of a special label. It does double duty essentially. One, it's great for people who are newer to the project and kind of calls out issues that are gonna have enough information for them to do things. But on the other side, they're also really good for saying, this requires a skill set or a particular knowledge that we're lacking in the project. Maybe you're looking for someone who's a tech writer to help with docs. You're looking for someone who has specialized networking knowledge, something like that. Oftentimes, even though they're new to your project, they're not new to the domain. And so they can tackle more difficult issues than you would have expected. Good first issues is a subset of help wanted. And the idea is we're really removing all the barriers to entry. We're making sure that you completely spell out, this is how to do the task. Give examples to the code, like link directly to GitHub. This is what you should change. Go here, one, two, three, four, what should you be doing? Where is the test? If it doesn't have a test and you need to kind of make it test out of thin air, you either need to link them to something that they can copy and kind of model off of, or maybe it's actually not such a good first issue yet. Here's an example of a good first issue and what's here doesn't matter. But what you can kind of see at a high level is I have links to the code that needs to be changed. And I've step-by-step instructions saying how the change should be made, maybe any background or motivation so they understand why they're making this change and then where they can add their tests to. Now, I've had a good first issue open on my repository probably since day one from three years ago and I'm never ever gonna close it because new contributors who are coming to my project have a perspective that I will never be able to have again on my project, which is being new to it, not knowing the right way to do everything, not even knowing the domain. And this is the type of perspective that you critically need in order to console me vetting your project and making sure that you haven't left gaps in both your new contributor documentation and honestly your new user documentation. So it's kind of cool as a new contributor to be like, I'm in the best possible position to do this task. So make sure that they always know that we want them to open PRs or just ping you any way they're willing to give you this feedback about what it was like to be a new contributor or a new user on your project. Make sure you get that feedback. Two, one. Now let's take a look at you've attracted someone to your project, they're interested in contributing, you're gonna need to teach them what it takes to contribute and what's involved for various things. There's the obvious, which is setting up your environment, right? And we'll get into that, but less obvious is telling them what your project is and how to use it. Thinking back again to those learners, they may have never used your project before, they may not be familiar with cloud native. So any videos or content, blogs, doesn't matter, something that's gonna get them up and running with your own project before they try to make a first contribution is gonna save you a lot of heartache reteaching people things and it's gonna help them feel more productive from day one. And in general, you're gonna wanna focus on the experience for that first pull request and how you wanna handle them and how you wanna handle this. Some people like to take this and encode it. Like in my project, we actually say all reviewers, you're gonna look out for new contributors, you're gonna do X, Y and Z, but it could just be something informal that you talk about with your other maintainers and go, what do we want this experience to look like for a new contributor? Now, there's different ways that you can actually do this sort of intake process. And like we said before, I wanna say, first of all, the first way that you should work on is the sort of text guide version of this. It's gonna be the most generally accessible, it's accessible around the clock, it's much easier to maintain and keep current with your project's contribution processes and code and documentation. Once you've taken care of the text version, then you might wanna consider having some online videos because some people are video learners. They learn better from videos, but if you wanna gonna do that, having a lot of short videos is better from a maintenance perspective, nothing else, right? Because things will change about your project and it's a lot easier to re-record a single three minute video than to have to re-record that 90 minute full tutorial.  Because as attractive as that is when you're having, say, a project summit, it's actually the most resource intensive to reach the fewest people. This was a suggestion from Charles from LakerD. And I took this and ran with it for Porter and I found that this was actually one of the most valuable things I could have done for my own pipeline, which is I wrote a contributing tutorial that says from start to finish, what does it look like to change my project get set up, test the code, et cetera. And the very first thing on here, again, is I want to see contributors try the project and understand what it is. Like they've definitely run it before they submit their first PR. I've got a number of first PRs before where it's like, you can tell they never actually ran it, they figured out the code to change. And I feel like you're not doing anyone a favor there. We're here to learn, we're here to become better at these things. So make it clear upfront, like try it, learn it, give us feedback on it, and then walk them through. How do you set up your environment to build the code? How do you check it out? If you have any specific build tool recommendations, most projects have signing, for example, like a DCO, do not assume that someone knows how to do this or this could turn into a blocker for you, even if it's something that could be easily done. Document how to do a DCO, how to sign your commit. And then also, oops, how to fix it when you forgot to sign something. Otherwise, things I like to look at is how to build the code, try it out with manually and through the command line tool or an API, whatever makes sense for your project. And then do an actual change, like Porter is a command line tool, so we always have someone make a sample command that like prints their name, for example, show them how to test their changes in general, and then celebrate, have them tell you when they finish the tutorial. Cause again, closing the loop here, remember that they have the best perspective on improving these docs. Talk to them after they've done this and you're probably gonna find something new that you need to change, probably every couple contributors. Now, one of the things we talk about in terms of contributors is you don't wanna just get new contributors, but you want to advance them, right? Because one of your goals eventually is to have more maintainers for the project, because that's the only way that the project is really going to grow. And moving from new contributor to maintainer goes through a series of steps that we like to call the contributor ladder. And for this contributor ladder to work, it's something you actually want to publish. The different sort of project roles and what the qualifications for each are and what the responsibilities of each are. So, add that in, consider some things that you maybe don't have yet as a way of recruiting people, like for example, maybe you don't have a docs maintainer, but you would like to have one. Well, putting that role in your contributor ladder is an opportunity for somebody to say, hey, I could do that. Also, it's important to write this down in paper because it's a good way for you to agree among all of your existing contributors how people earn trust and how they advance, because if you're having that argument in with the contributor at the time that they want to advance, it's a bad experience for everybody. Decide it in advance. Now, we have examples of this. We just finished the contributor ladder template, which is a part of our project templates, but in advance is exactly very simple, right? We have contributor and how you become a contributor, maintainer, how you become a maintainer, admin, how you become an admin, depending on the roles, right? Other roles might include reviewer, doc's maintainer, organization member for how people get credentials on GitHub. And we have some examples in the template of how you define those things. So we've given you a lot of examples and you've seen screenshots of what various documents are that would be really helpful to have in your project. Now, as part of our SIG, we've created templates where you can just clone this entire repository or cherry pick files that you think would be useful. And it covers things like the read me in this contributor ladder, which you may not be familiar with. We also have created guides. Now, basically all the information that we've given you here in this talk, you can find in text format in some of these long form guides on our website. For example, the contributor playbook will go into details on how do you find and recruit people into your project and attract them. Whereas the contributor growth framework is real stories from maintainers of successful projects and what has worked for them. And reflections on maybe what hasn't worked so well too. So if you have other questions or if you are excited about this or you want interactive help on advancing the project, attracting contributors, this is what SIG contributor strategy is here for. We have two different meetings for the asynchronous option. We have two different meetings, one every first and third Thursday of the month at 10 30 AM Pacific time. Alternating meetings are maintainer circle meetings. The other meetings are general and you are welcome and encouraged to bring your project and your particular problem and put it on the agenda for that meeting. That is completely invited. For specifically attracting contributors, we have the contributor growth working group meeting every second and fourth Tuesday at 2 PM Pacific. And that is for more contributor growth problems. Any way you're welcome. And around the clock, you can join us on CNCF Slack in the SIG contributor strategy channel. Now we will move on here to any questions you have during the session.


I now open this meeting if I may, thank you very much, and welcome members to the second meeting of the Delegated Powers and Law Reform Committee. I'd ask members as ever to turn off their mobile phones. I've moved to agenda item 1, which is both to welcome Elaine Smith to this committee but in accordance with section 3 of the code of conduct. I invite you, Elaine, to declare any relevant interests. My interests are as per my register, but I don't have any interests specifically relevant to this committee. I'll now move to item 2. It's proposed that the committee discusses item 5 in private, which will be an opportunity for the committee to discuss its proposed business planning. The members agree that we consider this item in private. Agenda item 3, instrument subject to negative procedure. Agenda item 3 is consideration of instrument subject to negative procedure. The Food for Specific Group Scotland Regulation 2016 2016-19. Those regulations omit to make further consequential amendments to the food intended for use in energy-restricted diets for weight reduction regulations 1997, the 1997 regulations, which are required as a result of the changes to the 1997 regulations introduced by Regulation 6. Regulation 6 amends the definition of relevant food in the 1997 regulations by leading the reference to food that replaces one or more meals of the daily diet. The effect of this change is that partial diet replacement weight loss products, as opposed to total diet replacement weight loss products, will no longer be classed as relevant food and will no longer fall within the ambit of the 1997 regulations. Only total diet replacement products will now fall within the ambit of the 1997 regulations as relevant food. Regulation 2 and 3 of the 1997 regulations make further references to partial diet replacement products, which, following the amendment made by Regulation 6 to the definition of relevant food, have no legal meaning or effect. The regulations omit to make further consequential amendments to the 1997 regulations, which are required to remove those further references. That has led to a lack of clarity in the 1997 regulations as amended, in that redundant text is left on the statute book that should have been removed. Scottish Government accepts that the amendment made by Regulation 6 has not led to sufficient clarity in the 1997 regulations and is undertaken to further amend the 1997 regulations at the earliest available opportunity. Elaine Smith Thank you, convener. Could the committee welcome the Government's commitment to amend the regulations, but also ensure that the committee tracks the implementation of that commitment? I think that that is absolutely fine. We welcome that. Does the committee agree to draw those regulations to the attention of the Parliament under the general reporting grounds in relation to the emission of consequential amendments to the 1997 regulations? No points have been raised by our legal advisers on the national health service, free prescriptions and charges for drugs and appliances. Scotland amendment regulations 2016-195 is the committee content with those regulations. Thank you very much. Now we move to agenda item 4, which are instruments not subject to any parliamentary procedure. No points have been raised by our legal advisers on the Education Scotland Act 2016, commencement number 1, regulations 2016-192, C15, the act of sederant, sheriff appeal court rules 2015 and sheriff court rules amendment miscellaneous 2016-2016 slash 194. No points have been raised on inquiries into fatal accidents and southern deaths, etc. Scotland Act 2016, commencement number 1 and traditional provisions, regulation 2016-2016-196, C17, and no points have been raised on the criminal justice Scotland Act 2016, commencement number 2, order 2016-2016-199, C18. No points have been raised on the act of a journal criminal procedure rules 1996 amendment, number 3, instruction of representation in the High Court 2016 slash 201. Is the committee content with these instruments as no points have been raised? Thank you very much. And we will now move into private session, as that is the end of that part of the agenda.


Call your shot. Bryce, for sure. He's just special. He's just, the word generational gets thrown around way too often, but he is generational as a human being. Like who he is, what he does, what he brings to the table. And as we all know, being a quarterback is up here, right? Tom Brady may not be the most athletic guy, but he was the best up here. He was the smartest guy. He prepared the best and he was ready the best. And that's the type of thing that Bryce does. And he's got that kind of Steph Curry almost, even more like Chris Paul facilitator point guard to bring it to basketball as a cross reference. But yeah, he's just, he's got it in factor. And he's done it and he's done it at the SEC. And man, that Georgia defense is pretty much an NFL defense and, you know, he did just fine.


I came across this taiaki stall in Japan and noticed something different about it. They had all the regular fillings, but then there was this. They were selling actual Magikarp taiaki. And you could see them being made right in front of you. No, you don't need an old rod to catch one of these. It even came in special Pokemon themed packaging. And on the back it had a song about Magikarp. Or Koi King, as it's called in Japan. Now this one was a lot chunkier than normal taiaki. And even though the majority of it wasn't actually filling, as you can see right here, it was still good. And where else can you say you ate a taiaki Magikarp?


Hi everybody. I want to give you a bit of a background on myself. So I studied machine learning and that's what my PhD was in, and then I switched a bit to computational biology. So I come into this field from computer science and what I do is mostly develop computational methods, machine learning methods to try to integrate the data to kind of get more informed opinions about patients and patient cohorts that we study. So today's objectives are, first I'll talk quickly about the single data type analysis from the kind of computational perspective, and this is always good to do for data exploration purposes, even if you are working on data integration. I will mention three most common data integration methods, most commonly used that I've seen, and I will talk briefly about feature selection and classification type problems. So this is the available patient data. It might not be available for every single patient for every disease, but across several diseases, it's definitely all of this is available. So DNA, as expected, mRNA expression, epigenetic data. So epigenetic conclusion, DNA mutilation, but also the epigenetic marks, microRNA data and protein data. So this is all omics, and a lot of this is now available for large sets of patients, especially in cancer. On the phenotype side, where you can derive different kinds of phenotypes, there's all kinds of clinical information that includes the tests, questionnaire data in neuropsychiatry, if any of you work there, then for sure you've seen a lot of questionnaire data. Imaging, of course, especially in neuropsychiatry, and anything related to right now, anything related to autoimmune disease, especially inflammatory bowel disease, you would see diet data being collected and also informing kind of decision. So this data is not just sitting in hospitals. This data is publicly available, some of it, and the best, to my knowledge, such a repository is the cancer genome atlas, where for the same set of individuals, so this is cases, for the same set of individuals, you will have exome, SNP data, mutilation, mRNA, microRNAs, and in overlap, if you take an overlap of all of this data, so not all of the data available on all patients, but over 500, this data might be available on over 500 patients. So this is large enough cohorts to start kind of playing with and understanding different integrative methods, so that's your goal. So why do we really integrate patient data? Hopefully integrating patient data will help us to make more informed decisions, something that we can help doctors to integrate. Apparently it's been analyzed and published that human capacity to analyze multiple variables at a time is about five variables as a limit, so we can put together five variables and make a decision based on five variables. We are talking about thousands, right? So this will have to be done computational and that's why we are doing this. So let's start with a single data type analysis. So let me go slowly and I would really like kind of an interactive session, not just talking at you, but please ask me questions because the goal is to understand kind of the kinds of analysis that you can do with this data. So this is a paper that was published in 2005 in PNAS and it looked at 20 glioblastoma cancers and gene expression on 18,000 genes, right? So what did they do? They collected gene expression. They looked at most variable genes across their collection and they performed hierarchical clustering. I'll mention, I'll go briefly through what the hierarchical clustering is. So here they identified clusters. Now they identified clusters because they were looking at heterogeneity of this cancer and they were trying to identify patients, homogeneous subsets of patients that they had in their hands. When you're looking at disease versus healthy individuals, it's the same kind of problem. You're trying to identify genes that are most differential between one group which is disease individuals and the other group which is healthy individuals. So in some sense this is similar. So you can skip this clustering step and the rest would be the same. Then they identified the genes associated that were most different between the clusters and corrected for multiple hypothesis correction. I will not go through, I will not talk about, is it part of the workshop? I don't remember the correction for multiple hypothesis. It will be mentioned probably, not directly. The point is that if you are looking at hundreds, if you are looking at thousands of tests, it's very likely that you will identify some of the things by chance, right? Some of the things that you've tested will come up significant by chance and that's the correction is very important when you're doing that. So they have identified these genes and you can see that on the picture on the, what is it left for you? Right for me, there are several genes that pretty clearly identify a set of patients. So they're mostly red for the blue group and mostly green for this orange group. So this is a simple analysis. It's still being done. But there are a few problems with it, right? So what if you have more data? If you have more different kinds of data modalities, they will tell you something else about these patients and these might not align with those groups that you find from the gene expression. So how do people handle it? How did people used to handle it before? So the way it was done, for example, in this 2010 paper, this is a classic also glioblastoma paper. They had 200 glioblastoma patients and they took essentially also mRNA expression and they also clustered it in kind of a similar way. They identified this four clusters and I think each signature was about 80 genes for each of the clusters. So you can see it here in red. Can you see? Perfect. So this is kind of this four or five clusters and they've identified the different types and based on the genes that were correlated with those clusters, they have decided that these are pro-neural, neuro-classical mesenchymal groups. So the problem here was that in this particular analysis, it was all based on gene expression. So gene expression essentially was driving the groups. They've identified the groups based on one data type and then they looked at mutations. If the mutations correlated with the clustering that they found, they said, okay, these mutations agree and they inform our clustering. If they don't, they don't inform. So maybe they've added a few genes to refine their clustering, but the clusters were defined by this one single type. And what was interesting is that they actually had mutilation data available to them when they were doing this analysis and they said that the mutilation data was just not informative, right? And the reason for that was that gene expression and mutilation data did not correlate very well on this subset of patients and it produced different types of clustering and they didn't know how to reconcile that. So in 2012, there was a contrary evidence also in glioblastoma. It was a glioblastoma study where they looked very carefully at the epigenetic data. And they found this, this is one of the ways that they've confirmed the IDH1 subtype. If any of you have seen glioblastoma data or have heard about it, IDH1 subtype is basically the only subtype which is really well defined in glioblastoma, where this particular mutation causes a huge hypermutilation across many sites across the genome. So even though the proportion of patients on this kind of x-axis is patients, so the full set is 210 patients, the IDH1 subtype is a very, very small subtype, maybe 10% of the patients, maybe a bit less. You could still identify it because the signature in mutilation was so strong, right? So this is what happens when we start integrating data and start looking at the data from the perspective of a single type of the data stream which is driving our analysis. We can miss this kind of things. So I will tell you more about the type of different types of approaches. The first and most common by far in, for example, the TCGA papers and other publications is to concatenate on cluster. So you can imagine what this is, but I will go in a little bit of detail, no problem. So through this analysis, then there is a slightly more, well, there is a more sophisticated method called eye cluster, which is kind of a latent factor type analysis, and similarity network fusion, which is the method developed in our lab. So the concatenation is simple, right? You have patients as rows, you have your measurements as columns, so you have here your gene expression, you have mutilation. You just group them together and treat them all as kind of one vector per patient, regardless of that type. So the problems with that that we found in our own lab were that if the structure within each of the type of the data is different, you are washing it out, right? If there is some kind of correlation between genes that you don't want to lose, if there is some kind of difference in the structure of the measurements within the different types, you actually lose that and wash it out.  So that is also a problem. So what they do is concatenate the data and then you do hierarchical clustering. So many of you have probably used hierarchical clustering. This is approximately how it works. So here you have six, let us say individuals, right? So this is already a correlation matrix. You go from kind of patients by features to patients by patient matrix by just either multiplying them or performing correlation on these different feature vectors. So here you have six by six patients and you look for the smallest distance between these two patients. So it's represented here also in a kind of in space and also as a graph. The green tree that you see is the most common way to visualize hierarchical clustering. But the point is that D and F here are the closest ones. So you merge them first, you identify the mean and then you look at the difference and you compress this matrix and you look at the next smallest difference. The next smallest difference will be between the mean of D and F and E and those are the ones that you merge second and you can see it here. And also you compare, you do it for all pairs essentially and A and B are also small so you merge them as well. Next, et cetera. So you basically go through and compare all the pairs and compress the matrix. As you go, at the end you have only two meta groups, two means left from these two groups in this example and then you merge it. This is how you get. So to decide on a number of clusters here in this particular clustering, what people do is most commonly actually they're arbitrarily cut by eye. So here you have this kind of diagram. So what people would do is cut it into two clustering because that looks like most reasonable from the dendrogram. Yeah, this graph is called the dendrogram, the tree. Okay, another measure very commonly used is a silhouette statistic. An eigengap is used for spectral clustering. So if you have a graph that you are trying to cluster, you're trying to look at the difference in the eigenvalues for that graph and the spectral decomposition. This is if you're using spectral clustering. There are many more. There's a thesis that was published that looked at the different scenarios, simulated different scenarios and compared the matrix, all the matrix like maybe 15 of them on those different scenarios. So basically there's not one metric that is good for every single scenario. So some of them are good for kind of in a Euclidean space and some of them are good on a manifold where it's not so straightforward, right? Where if you compress the clusters, they look very close and 2D space but further away when you consider high dimensions. So basically there are many examples and there is not one good metric. And if somebody tells you, I found your clusters, you can ask them how they have found them and which metric they used and you can test other metrics to be sure. Can we have access to that PhD thesis? Yeah, it's online. Yeah, I just, yeah, search for the clustering. This is just, yeah, you can just search for Jan and PhD thesis 2005. I think it's from Waterloo. Yeah. So here I want to tell you a little bit about Silhouette because it is kind of very, very commonly used if you, especially if you don't use the hierarchical clustering or to verify whether you like, you know, the clustering, what kind of clustering that you get from Silhouette. So the Silhouette statistic was originally published in 1987 and the idea behind it is it tries to figure out if the points are closer to each other in your cluster versus further apart between clusters. So this is the difference between points within a cluster versus across clusters. This is what the B and I stand for. So AI is the average distance to all other patients for the individual I within a cluster and BI is the average distance to all other patients in other clusters. So, for example, you can see in DNF, all of these points will be very close to, in our previous example, will be closer to each other than to this cluster, for example. But it's not always the case. So Silhouette statistic, it ranges between minus one and one. If it's one, it's an excellent assignment. If it's minus one, it's a bad assignment, a really bad one. Maybe you want to do exactly the opposite. And zero is borderline, but zero means also that there is not much agreement in your cluster. It's basically saying that your points are as close to each other within a cluster as to the points in the next cluster. So this is an example of what Silhouette value plots look like. And basically if it's positive, it means that all these points are closer to each other within a cluster. And the negative part means that within this one cluster, in this three cluster case, within one cluster, these points are closer to other clusters than to the other points in this cluster, on average. Right? It's on average. So this is actually, this is a very important point about clustering. What clustering methods do is they place everything into a particular cluster, right? They assign everything. It doesn't matter if the data cluster is naturally or it doesn't cluster naturally. It just will, it will produce clustering for you. You'll say I want three clusters and it will find three clusters for you. Now Silhouette allows you to evaluate how good your clusters are. And this is, this is a point to always keep in mind. What can, yeah? Do you find what a pattern is in this case? Is it just all the features? Are you referring to? The pattern here is just a point. Okay. Just a point. Yeah, like in this example, it's just a point in space. Yeah, pattern entity object. And some of the... It can be, but in this particular case, you are talking about each individual point in your clustering. So one way to see how robust your clustering is, is to do this idea called consensus clustering. So this was published in 2003, but people use it, have been using it for a long time. So basically what you do is you take a set of your features or samples that you want to cluster and you sub-sample. You say, what if I take 80% of my patients? Will they still be together in the same cluster or not? And you do it a thousand times. You take a different 80% or 70% of your individuals and you cluster them a thousand times and then you construct a consensus matrix. And the consensus matrix basically tells you whether the individuals clustered in the same cluster if they were sampled together. So how often they belonged to the same cluster? So here there is an important point that the number of the cluster doesn't really matter, right? You can permute the labels of the clusters. It doesn't really matter. Cluster one and one case, cluster three and a second case, that doesn't matter. What matters is whether the individuals belong to the same cluster or not. So you look at all pairs of individuals. You construct a consensus matrix by being something actually something like this, I would say. So I imagine that these numbers are how often individual A appeared with individual B in the cluster. So suppose 70% of the time that you've sampled them together, they were clustered together. So that's a pretty good and accurate measure of the fact that this is a pretty stable cluster. Now this matrix doesn't work too well because normally you would have all the values between zero and one, right? You cannot cluster together more than 100% of the time. But the point is the smaller the number, the less evidence you have of the fact, the less robust that pair is. And sometimes it's good to identify those individuals that don't cluster together well with anybody and just say these are my outliers, these are not the individuals that should belong to a cluster together and maybe consider them separately, right? So this way you identify the cores of the clusters, something that you are very fairly certain about. Yeah. Does this work equally well on all sample sizes or would it work that matter? Well, the smaller the sample size, the less variance you will have when you sample, right? If you have 10 points and you sample them 80% of the time, you get 8 points. So there is a huge overlap in your samples. So it's less of an evidence. You will have less of a confidence than if you have a much larger space. Yeah. Okay, so this is the most standard approach, the concatenation and clustering and using consensus clustering improves the robustness of the results. So I would recommend always to always do consensus clustering. And I think there is a package in R also that does some consensus clustering. It takes much longer, of course, because it does the clustering procedure a thousand times. But it's worth it for the stability of the result. Okay, so the next type of tool is this eye cluster. So this is more of a latent kind of a factor model. So if you are familiar, this is great. So what this approach does, and I will briefly mention it because it's also using TCGA papers and other type of papers. And I've seen it in a lot of different publications recently. So what happens is that they're trying to identify some kind of a latent space, some kind of a latent embedding, which is common among all data types. So let's say we believe that our patients, for example, cluster all in the same way, should cluster in the same way, given any type of data on my patients. And if that is the case, what they're trying to identify is an optimization problem where they try to identify this latent variable Z, which represents that latent embedding. We don't know what it is, but every one of the types of data that we have is informative of that. So what they do is they identify the common Z, and this W1, W2, and WM, these are the projections of your original observed data, say copy number data, how that projects onto this latent space. This is what they do, and this is what they find. They find this latent clustering in the latent space. The problem with this approach, even though in their paper, they kind of say that they don't just look at the similarity. It actually ends up being that they are looking at the similarity of the data sets. And if you have some complementarity, it is a problem. So this is kind of what we looked at so far. Eye cluster is also a problem because the complexity of the method depends on the number of the measurements that you use, number of the features, so you can't use your 20,000 genes or all of your methylation probes in the eye cluster. You have to select about 1,500 features. How you select that is magic. And you select 1,500 features usually by most variable ones or something like that, maybe t-testing with respect to some held out group. This is not necessarily part of the method, but you have to pre-select your features and then do the clustering in the latent space using those features. So this is part. There are many steps in this pipeline. The feature pre-selection which is different between the different types of analysis and that, et cetera. And I already mentioned that these methods don't really take the complementarity of the data into account. So seeing that we have developed a different methodology. The methodology is this called similarity network fusion. It consists of two steps. The first step is to integrate the data in patients, construct the patient similarity matrix similar to how you do for hierarchical clustering. And then we fuse this multiple matrices using nonlinear approach. So this is the first step. You go from the patient by say mRNA or gene expression and construct the similarity just according to the gene expression. So, for example, the darker spots here, they mean that these patients are more similar according to the gene expression, to the genes that you have in your sample. Now, this matrix, if you just look at the correlation, this matrix is likely to be a full matrix, no zero entries. But you can actually sparsify it. And if you do sparsify it, so there are two ways to sparsify matrices. Generally, one is to cut by small values. So if you have 0.01 correlation, maybe you don't care about that, even 10% correlation, you don't care. The way that we sparsify the data here is by k nearest neighbors. So we keep the 20 most similar individuals regardless of their value and we throw away the rest. And that helps us with numerical stability when we combine the matrices. So you do it for every single type of the data. So you have these two similarity matrices. For example, one is from gene expression and the other one from DNA methylation. There's kind of a legend on the bottom. And the second step is to iterate. We iteratively make them more similar. So how we do that is based on graph diffusion. So we basically take a matrix, we multiply it by the matrix. We want to make it more similar too. And this is kind of like starting at a node in one graph and continuing in another graph and coming back to the first graph. So what this procedure does, and we do it iteratively until it converges. So this is the similarity. So what this procedure does is it helps you to get rid of the noise, for example. If you have some edges here that are very weak. So this indicates weak correlations. They're not supported by most of the data. So in that case, they disappear as you make the matrices more similar. On the other hand, for example, you have a very strong correlation here between these individuals, but it's not supported by all of the data sets. But because it's so strong, it actually permeates all levels. And we can keep this complementarity that is introduced by the mRNA-based kind of similarity into the other types of data. So this probably converges. And at the end of the day, we have one matrix, one network corresponding network, which is actually supported by all the different data types, all the types of data. So are there any questions as I'll present some of the results using this approach? Yeah? So we don't actually use other methods as part of this method. Oh, you mean like types of measurements? Yeah, yeah, it's definitely, I mean, whether you keep weak correlation or not is numerical, right? So there is no particular threshold that tells you, if it's 10%, we lose it, if it's 20%, we keep it. So it is possible that in one measurement, one type of measurement, we have the information, but we have 10 types of measurements. And the nine types of measurements do not actually support that similarity, and of course we lose it then. So it definitely happens, yes. But I guess it is the question of how much support there is across the different types. So if there is a very, we definitely have the situation, I'll actually show it in a real example. Let me show a real example. I will tell you that it doesn't always happen. So these are the TCGA types of data. We looked at five different cancers. So this is the glioblastoma that I've been talking about. 215 patients, slightly different cohort. We had three types of measurements, mRNA, methylation, and microRNA. And this is the number of measurements in each. It was in 2014 when we did the analysis.  So we have five different cancers here, and we also have controls, so healthy individuals for whom some of the data was measured for these cancers. Actually, not necessarily healthy individuals, but the healthy samples from the same individuals. Obviously, not healthy individuals. Nobody does biopsy on healthy individuals. So this is case, so this is glioblastoma, and this is the patient by patient similarity. This is the full matrix, and this is kind of the corresponding graph. The topology doesn't really matter in the graph because the topology was taken from the fusion matrix. But it makes it a bit easier to compare. So you can see, and this goes back to the question that was just asked, you can see that there is some similarity according to mRNA, which is not really here in methylation, but some of it is here in microRNAs, for example. And there is a lot of similarity according to methylation between those two clusters, which is not supported as strongly by a gene expression or by microRNAs. So a lot of this would go away. But what you can see is that each of the types of data provide different structure of how patients are similar to each other. And each of the different types, they also correspond to some degree. So every type has these clusters. But mRNA, for example, in this particular case, did not yield such a strong structure. So maybe here there is some structure but such a strong structure as the other types of data. So by combining, we hope to take advantage of all of the evidence that we have. And this is the fused matrix. It looks quite a bit cleaner, because of the off-diagonal noise, the entries that were kind of the similarities, they did not correspond to each other in this different data type. So they kind of went away. But you can see that the clusters are similar to each other, to some extent. But what's important and interesting here, even if this type of data, this data set is clustered, you can see within clusters. Sometimes you will get a disease that looks like one of these clusters. And there is no intrinsic clustering to this particular cluster, at least not according to the data. But what's important to notice is that the colors, each edge is colored by what type of data supports it the most. So green is microRNA, pink is DNA methylation, blue is mRNA. What's interesting to note is there are pockets of this microRNA supporting the similarity between patients within cluster. They're also very heterogeneous. So for example, there's a whole pocket here of patients who are also similar to other patients but are similar to each other according to microRNA and DNA methylation. There is some signature between these patients. So when we start looking at patients as a whole, we can't necessarily distill it down to five variables that we can profile for these patients. It's a heterogeneous disease, and even if there are some more better-defined overall groupings, it's still very heterogeneous within each cluster. So we looked at the clinical information, and I think you will have an example of that in the workshop. For example, this is the survival, and this is subtype 3. In subtype 3, it's here in blue, and this is the IDH1 subtype that I talked about. So this was actually very interesting. Of course, I'm not a glioblastoma researcher, but I had a meeting with another jobado who was visiting here, and she came into my office and I had this plot, and she said, this is IDH1. I said, what is IDH1? This is a check if this is IDH1, and every single patient for us, we didn't incorporate the DNA, the mutation data here, but every single one of the patients for whom we had the mutation data was IDH1 mutant. So this is primarily due to the mutilation, the very strong mutilation signature that we had. And in glioblastoma, there is this very nice relation in leukemia. The relation is more complicated. So there is also IDH1 mutations that do not lead to hypermetilation of almost the whole genome. But the point is sometimes you can recover biological signal if it permeates other types of data that you have collected, even by proxy. So the IDH1 patients, they're usually younger, but they have a better prognosis. So what was interesting is that subtype 1, this big subtype 1, was the only one that seemed that actually responded in some way to thymazolomide. Thymazolomide is the standard of treatment in glioblastoma. The other subtype did not actually seem responsive to thymazolomide. When you look at the data on three different levels of DNA, they still seem to map into the same clusters or the same genes that were mapping into those clusters. No, no, they actually weren't. We did look at that. When I'll be talking about feature selection, I can actually, given that there's almost time, let me talk about that. I can come back to the rest. So this is the feature selection. So this is where we looked back. We have our three clusters, and we looked back at mRNA expression, DNA methylation, and microRNA expression. And this is the standard, the t-test analysis that you would do, the pairwise t-test. So features this cluster versus not, and looking at each individual gene, saying which genes seem to be most responsive. And this was actually interesting because we did this analysis with a t-test, and this is the figure we submitted with the paper and the reviewer said, well, it looks like you don't have signal. What is this figure? And even though it looks like, to us, I mean, heat maps are intrinsically hard to interpret, right? You can see different people see different patterns. But we started thinking about why we are not picking up stronger patterns. And so what we thought about was that this pairwise test, they're missing something. And what they're missing is, these are not the genes that are corresponding to the kind of the three-way analysis. And even further, what would be nice, which we actually didn't do, would be to see whether the patients are similar according to this particular gene in a fused matrix, right? Are they as similar according to this gene as they are in a fused matrix? So even within cluster, you kind of want to keep preserving this similarity of identifying genes that capture the full similarity across the patients, rather than just between the clusters. But even when we looked at kind of this three-way, and we used normalized mutual information, you could see a much better set of genes. So yes, there were a few genes, I think on the order of five to ten genes that actually corresponded between methylation and mRNA. There were genes that had abnormal methylation in the promoter and had a different... differential gene expression, but like five out of this set of genes, 250 genes. So the answer is that these are... this is different biology that's being recovered from each of the individual sets. So what were the initial subtypes of clusters based on which data that we saw? The initial ones in this analysis, all of it. This is the... all three types of data, all of the data. Yeah, I skipped through this, but this is kind of PCA plots that are very useful to visualize the clusterings, so you can see how much the clusterings are spread, and so in BRCA it looks like this is a different cluster. And also one thing that I want to bring up to you, if you work in cancer and you work with survival analysis, if your clusters are very small, for example in kidney renal carcinoma here, this one cluster is very small, so if you move one of the individuals, the p-values go crazy. They just go crazy because p-values are very sensitive to the kind of the size of support that you have, and you can see here... the reason we discovered it was that we started looking at the minus log 10 p-value according to the different number of features, and at some point the p-value just dropped. We added maybe 50 more features and the p-value dropped, and we said, well, this is very unstable, how come? In other cancers we don't really depend on which features we select, we didn't pre-select features for this analysis, and in kidney it seemed to matter in some very strange way, and what we discovered was this really tiny cluster that made the p-values unstable. So this is also something else to keep in mind. All right, and I have a couple of minutes. Yeah, this is NMI. You can look it up. This is what we used for the analysis to identify the features. It's an information theoretic measure, but you can actually use cross-call Wallace if you want to do the analysis. This is what we used in that paper. So what I want to mention now is... especially in our cancer work, what people want to do is build classifiers. I want to identify the subtypes. You say, I have subtype 1, subtype 2, subtype 3. There's a novel subtype. How do I create a clinical test that will tell me, okay, I want my 50 genes. I want to test this 50 genes, and I want to be able to label new patients that come in as subtype 1, subtype 2, or subtype 3. This problem is called classification. And the way that people kind of do it is they take the features that you've identified, either through titas, cross-call Wallace and MI. You identify features that were significant from here, from this type of analysis that associate with the clusters. Then you build a classifier, let us say, Random Forest is a very common and well-loved classifier, and it works really well. And I have a question for you. It's a quiz. What is the problem with this approach? So you take, you've identified your subtypes. You take the features that seem to be associated with the subtypes, and then you build a classifier using those features. What is the problem? Yes, overfitting. Overfitting. In machinery, it's called a leakage. You have a leakage problem. You use the same data to create the clusters, and then you're using the same data to create the classifier. And it's a big problem. So the problem that it creates is with generalizability. What is generalizability is that if you have a different cohort of patients for the same type of disease or outcome, your results might not generalize because you've overfitted to the data that you have built your classifier on. So how do we fix this problem? One way to fix this problem, which doesn't solve it fully, is to take a subset from the original data, run SNF just on that subset, and then identify the features and train the classifier just on a subset of your data. At the same time, you can learn SNF on the full data and see if the classifier that you've learned is actually predicting the same classes as if you had the whole data. Now, obviously, when a new patient comes in, you don't have the original SNF, so you will just be left with a classifier, and that's what you ultimately want. So you can compare the labels. So this is saying if I had all of the data in the world that I needed to have to get maybe the best subtype labels, does my classifier predict those labels as well? So, of course, there is a bit of a leakage here as well, right? But you don't have any other kind of data. So this is a better approach than the original approach proposed, which some of the people use. It's very important to try to separate your training and your test set to know how well your method generalizes. Very important. And I think it's a very common problem in published literature, where people just report the training results. You actually don't know how well it will perform on the cohort outside of the hospital. So this is a better solution. It's not ideal, but if you don't have any other data, this is one of the ways that you could do. So with that, there are some advantages to the networks. I propose it as an exploratory analysis of your data. So what we do is we kind of look at our original data, each individual type of data separately. We compare what kind of classes you would get from each individual type of data. Then we combine all of the data together. One other thing that is useful with networks is that you can visualize the networks and see, well, does it look like there are real clusters or does it look like one big block? Because if you just cluster and then look at the results of your clustering, you don't actually have a sense of, well, how bad it might be, right? So the idea is that you actually have a way to visualize your data for your exploratory analysis. And then at the end of the day, if it looks like there are some clustering, you can cluster and you have some certainty about this particular clustering. This approach scales very well. You don't need to pre-select features. It also scales very well with respect to multiple different types of data that you are trying to incorporate. So one of the plots I had that I skipped was showing that actually the time it takes is about the hierarchical clustering. So the biggest time sink is to construct a similarity matrix. Once you've constructed a similarity matrix, it actually runs really fast. All right, so for the future, I think the simultaneous feature selection and data integration would be great. The kind of a problem with it is that depending on the question you are trying to answer in the end, you want to put a different objective for what kind of features you care about. For example, you're trying to build a drug response predictor. So depending on the drug, you want to incorporate different pathways, right? So your network, the final network will be different depending on what you use to compare the patients. Some patients will respond in exactly the same way, according to one of the pathways, and this information will not be informative at all if you are trying to answer a question for a completely different drug that has nothing to do with this pathway. So that's something important to keep in mind. And that's it for me. Questions? So you can certainly do cross-validation for your classifier as part of learning the classifier. But it doesn't remove the question of generalizability because you've just selected the features from exactly the same data. So it's really a question of do you have any data leakage? Do you have any information that you are using in your classifier that you've already seen? You're already pre-selecting the features that will be used in the classifier. If you have a held-out set that you have not looked at, that's fine. Yeah. So in terms of the measurements, being completely different scale. Yeah, so I would recommend that if they're on a different scale and you don't want to kind of standardize the variables with zero mean, I recommend to separate them into different types of different matrices because for this method, it doesn't really matter how many matrices you combine. So to separate them and to identify similarities according to the variables of the same scale that are comparable because otherwise numerically, the bigger scale variables will dominate. Yeah. So we... Yeah, concatenation, they just normalize. They either standardize the variables to zero mean or one variance or normalize them to be between minus one, one or zero, one. Yeah. So this kind of works. The important step here is also to remove the outliers. So what we've done in some of these data is we removed the outliers and re-imputed the data. And the reason for doing that is if you, for example, scale something to be between zero and one and you have a huge outlier then your outlier will be at one and everything else will be at zero, like completely numerically indistinguishable. Right? So you kind of want to keep track of the outliers. And actually, I think in the workshop you will... No, there's no outliers in the workshop. Okay, that's great. No outliers. I think at some point we definitely encountered... They're outliers, but they're not outliers. We definitely have encountered where we construct a similarity matrix and we just see two individuals are similar and nobody else. And then we say, well, this doesn't seem right.  And so once you remove those outliers you actually get them much better. So you're actually combining the data. So the iterations for the structure is more similar and I just am not really sure... How? Yeah. So I can try to give... It depends on the background so I can try to give some intuition. So you know how if you multiply matrix by itself it gives you all the random walks from that node of the second degree, right? The second order neighbors. You'll get all the second order neighbors with that. So basically if you keep multiplying matrix by itself it'll give you a random walk from that node to nth degree if you multiply to the power of n, right? For that same matrix. So we use that same idea except we multiply by another matrix the second matrix and you come back to this matrix. So its multiplication goes like this. It starts here. It multiplies by this kind of matrix. It goes the random walk. It's not... The random walk is not the best illustration because it's hard to visualize it directly but the math is the same. You kind of you start with this matrix. You say okay what are all the walks that I can do from this node in this other matrix and at the same time I want to preserve my second order. So it's a second order walk that's perturbed by the other evidence. And actually if you have... I didn't talk about this but if you have multiple matrices what we do is we multiply by the average of all the other matrices. And because we sparsified the matrix by the K and N it's relatively stable. You can actually do that. One thing that happens here of course if you take random matrices and you start combining them together you'll get the random matrix back. And we've experienced it once where we combined one matrix with something that really didn't correspond so we were trying to combine cytokine and gene expression data and they didn't correspond which was very surprising to us. But then we started looking at the cytokine data and we had exactly the scale and outlier problem and so we went back and said we need to reprocess the cytokine data. We went back to our collaborators and we actually started working with our original data before it gets transformed and then we were able to combine it beautifully. So the way we do it is we look at the two matrices and we look at the NMI so basically are they likely to produce similar looking clusters to each other or not? For me, you look at the formulas and say that's exactly what happens but I don't have the formulas on these slides unfortunately. But it's in a paper if you want to look at the Nature Methods paper. Disease types or data sets you've seen where this method doesn't work that well because the signal isn't as strong and all the different data types is it only for diseases where there's a really strong signal possible? So the places where it doesn't work is so it's much harder with DNA data. If you take genetic data and you compare patients according to their genetic data you just get ethnicity back and ethnicity is usually not what you're looking for to try to infer the biology of your disease, right? Though it could be correlated with the biology it's not usually the kind of signal that you want so we've had a lot of kind of struggle with how to properly incorporate DNA data on a SNP level for example. You take a GWAS data and you have 500,000 SNPs or a million SNPs and what do you correlate? So there you have to pre-select so basically the underlying assumption of this is that the features that you are combining are informative of the signal you're looking for in the data. So if they're not informative overall like a signal on GWAS maybe, I don't know there are different ideas about how the what the complex diseases can be attributed to but if you imagine that there are 10 genes that are relevant and you're looking at 20,000 then you're not likely to identify the similarity which is relevant to the disease. So the underlying assumption is that the features that go into computing similarity are informative of the disease. Largely. You can have some noise obviously but largely informed.


Now, let's take a look at some low-cost solutions because this is really what this course is all about. We want to talk more about solutions than about the problems. So, how do you make a society healthier without much money? That's a really big question that the leaders of countries wake up every day and ask themselves. And really, if you boil it all down, it comes down to being better access to public health services. So that basically just means clean water. It means safe food, decent housing, which this is a little bit harder. Being able to offer housing on a wide scale to an entire country is not as easy as checking your food and providing safe water. So, decent housing, adequate sanitation, which can be provided at low cost in some situations, but not all. And then improved education, especially in the area of health, which is something that we at Alversity are working at. So improved education. Alright, and we're going to talk about all of these different things. There are all big discussions. There are no secrets here. People have known these things for a very long time. But we're going to be talking a lot about all of these different points and these different solutions to lessons coming up in this course.


I'ma put it like this. The only thing stronger than a man's pride is a mother's love. Every man and woman will swallow every pill known to man to stay alive. But a lot of men or women won't swallow their pride. Yeah, we on boss talk one on one, one on one. Yeah, we gon' talk. When you think about that LA area, like I said, it can become dangerous real fast. Pop Smoke, I believe he was up there when he got killed. He had shown himself being somewhere and somebody's seen it. I believe it's what I heard now. I don't know, cause I wasn't there, but just why is it so violent when it comes down to these rap artists and the things that they're doing? Is it because of the money or what? Well, you know, to me, they're in a world of their own where they get to interact with each other and they don't take it too far to where they're trying to live up to the image. And you know, I'm gonna put it like this. The only thing stronger than a man's pride is a mother's love. Every man and woman will swallow every pill known to man to stay alive. But a lot of men or women won't swallow their pride. Did they get up in here? You have a lot of money involved. They don't nail each other and it's a camaraderie to where it doesn't make no sense. To where you have to go and kill off good talent. Oh, look at all these talented artists that's getting their lives destroyed over image and acting and talking about something that really doesn't adversely affect them. Cause usually when you're in the rap game, you so far away from what's going on that you got to get your stories from somebody else. That's why a lot of times you're seeing with street credibility dudes around them how else they gonna know the stories to tell the same what they do. And I just think it's a shame that they sit and target each other, argue with each other, then take it up measures to the extreme to where somebody's parent is crying, somebody's daughter has lost their father. And it just, it really needs to be resolved. They need to have some type of dialogue, create some type of dialogue and understand them that's what's acceptable. If you got a beef, sit up and talk, you don't have to air it out on the internet. But once again, this social media, this what they call it when you trolling and looking around, it's has changed and we're abusing social media as well as the entertainment. We're abusing it as our race. Don't nobody do the things we doing to each other then broadcast it. You can go on the internet and see that your child has been killed quicker than you can getting a call from the coroner. That's not acceptable. It's not.  I don't make, no, no, no, no, no. It's rules to this game, man. But we don't apply them because everything cool. But that ain't how they go for everybody, man. That ain't how that should be, man. Yeah, we on boss talk one-on-one, one-on-one. Yeah, we gon' talk.


Hello and welcome to the International Daily Roundup with People's Just Patch where we bring you some of the top stories from across the globe. Let's take a look at today's headlines. Extradition trial concludes a chance to remain in prison until verdict. Trump administration to cut refugee admissions under the guise of COVID-19. Iraqis take to the streets to mark one year of anti-government protests. Zimbabwe government threatens to replace tracking teachers. Massive protests by health workers across Germany. On Thursday, the extradition trial of WikiLeaks founder Julian Assange came to a close. The trial ends after nearly four weeks of testimonies from nearly three dozen expert witnesses. The testimony has covered a range of aspects of the case from Assange's mental health to the political nature of the charges he will be facing in the US and the possibility of torture and mild treatment he is likely to face if extradited. The defense will have four weeks to submit their closing arguments to the court. The prosecution will have two weeks after that to respond to the defense and submit their closing arguments. Judge Vanessa Barreczar is set to deliver the verdict on January 4th, 2021 until January. Assange will remain in the Belmarsh prison because he was denied pain. The final two days of the trial involved a marathon run of testimonials submitted by the defense team. On the final day of trial, the presiding judge refused to accept some of the testimonies over prosecutions, objections and the defense had run out of time. The submissions that were rejected by the court included those that would have dealt with current superseding charges against Assange based on which the extradition process is ongoing. The rejection of the statements was despite the fact that the trial ended a day earlier than the original schedule and the defense was only submitting testimonies that were already agreed by both parties to be up for submission. Among the testimonies submitted in the last two days, there were several anonymous statements from employees of UC Global, the company that is accused of spying on Assange on behalf of the US government during his time in the Ecuadorian embassy. The anonymous testimonies revealed discussions between US intelligence officials and employees of UC Global on the possibility of kidnapping or even poisoning Assange. Outside the old Bailey courthouse, where the trial took place, Assange's family, colleagues and supporters held a small celebration commemorating the 14th anniversary of Piccaleague's domain registration and also delivered short and final statements to those assembled. In an announcement made by the US Department of State on Wednesday, the administration is expected to allow not more than 15,000 refugees into the country in 2021. The new cap on the number of refugees admitted will be 3,000 less than the 18,000 mark set for the current year. The reason given for such a measure is the ongoing COVID-19 pandemic. The government justified the move as an attempt to supposedly prioritize safety and well-being of Americans, especially in light of the ongoing COVID-19 pandemic. The announcement was made late in the night and just less than an hour before the deadline to notify the US Congress of the proposed changes. While the changes are supposed to be cleared by the Congress, it is seldom ever challenged by the legislature. The new admission bar will be the lowest in US history because of the repeated annual cuts since the beginning of the Donald Trump administration. This year alone, the US admitted only around 10,800 refugees until March before freezing all refugee admissions because of the pandemic. The latest move also comes hours after an election rally by Trump who is seeking re-election in November this year. Trump flashed out at his opponent Joe Biden's campaign promise to gradually increase annual refugee admissions to over 125,000. Trump claimed that Biden and the Democrats want to flood the country with refugees at the cost of US citizens. The move to cut down on refugee admissions is in line with the overall policy of the administration to curb immigration. Since Trump took over presidency in 2016, the administration has heightened betting for asylum seekers, expanded punitive measures against undocumented migrants, built new detention facilities that have separated migrant families and used COVID-19 to freeze a range of these hours and refuse admission programs. On Thursday, thousands of Iraqis took to the streets to mark the anniversary of last year's anti-government protests. The protesters reiterated their demand of putting an end to widespread political and economic corruption, better employment opportunities and reforms in the country's political system. The protesters vowed to revive the protests which went on for months before they were stopped due to the outbreak of COVID-19 and subsequent imposition of lockdowns. They also denounced the massive state repression including the killing of over 600 of the protesters by the security forces in those protests. Thursday's protests were held in several southern cities in Iraq, including the capital Baghdad, Basra and Najaf. Protesters tied portraits of all the people who were allegedly killed or went missing during the protests. The demanded punishment for all those responsible for the killings and freedom and safety to protest. Last year's protests forced the government led by Prime Minister Adil Abdul Mahdi to resign. After months of political uncertainty, his successor Mustafa Al-Kadimi was appointed in May this year. Kadimi promised investigation into the deaths of protesters in strong action against corruption. His government also announced early elections for the Iraqi parliament in June next year. The last elections were held in 2018. Protesters have also been demanding an end to the political quota or Muhassasa system in Iraq, which is seen as the root cause of nepotism and corruption in the country. The Muhassasa guarantees a proportional representation of all major ethnic groups in the country and fixes crucial government posts according to the sects. As per the Iraqi constitution, the Prime Minister of the country can only be a Shia, the president occurred, and the speaker of the parliament, a Sunni. The system was introduced in the country by the American-led Occupational Administration, forced on the 2003 invasion. The protesters also opposed all kinds of foreign interventions in the country. Some protesters issued an ultimatum to the government to take effective measures by 25th October or be ready to face a national strike. Expressing its support and solidarity with the protesters, the Central Committee of the Iraqi Communist Party issued a statement on the 26th of September. The statement stated that the party glorifies the martyrs of the October uprising. Despite the government's threats of replacing them, teachers in Zimbabwe's public schools are continuing their strike, which began on Monday. The striking teachers are demanding a restoration of their salaries, which on average have reduced from an equivalent of $550 to a mere $30-35 over the last two years. At this wage, unions claim teachers are incapacitated because they cannot afford the cost of travel. Tapsen Sabanza, Secretary-General of Zimbabwe Teachers' Association, told People's Dispatch that their teachers are willing to work, but don't have the means to go there to work and subsist. However, despite being given the prior notice by the unions, the government decided to go ahead with the reopening of schools on Monday. Students had to be turned back because there were no teachers available. 98% of Zimbabwe's 136,000 teachers are currently on strike. According to reports, the Unemployed Teachers' Association has declared its unwillingness to let its members be used by the government as a reserve force to break the strike. The association instead demands employment so as to start filling the shortage of 90,000 teachers in the public education system. The Progressive Teachers' Union of Zimbabwe has called for a meeting between the government and the United Front of Teachers' Unions over the salary cuts. On Wednesday, healthcare workers from various hospitals across Germany went on a warning strike on the call of the Worthy Trade Union. In Berlin, 300 employees from hospitals and geriatric care facilities held a demonstration at the conference of health ministers demanding better working conditions and higher wages. The protesters in Berlin organized a photo demonstration as well by raising 30 banners with photos of 12,500 clinic employees across Germany printed on it. Worthy is demanding an increase in salaries by 4.8% for federal and local employees. They are also demanding an increase in training pay and a maximum of a 45-hour week for emergency service staff. Workers in other sectors are also holding warning strikes, demanding fair contracts, increased wages and safer work conditions as part of a week-long action called by the union. Sylvia Buehler, who is responsible for the health system in the Worthy Federal Board, stated that even in the corona pandemic, healthcare workers are making their demands visible and acting together. With the first nationwide photo demonstration, we have found a creative way to take the protest to the streets in a corona manner. Dialink has expressed solidarity with the health workers for tests in Germany. Party leader Bornrexinger has stated that the hospitals urgently need more nurses. These need to be paid higher and working conditions considerably need to be improved. There has been a spike in COVID-19 cases across Europe these days as of now in Germany there are around 27,353 active cases and 9,581 people have died due to COVID-19. And this is all we have for this episode of the International Daily Roundup. For more such stories and videos visit our website peoplesnisfiles.org. Subscribe to our YouTube channel and follow us on Facebook, Twitter and Instagram. Thank you for watching.


Just what's your instant analysis of what happened and what you liked and which one improved on? It's basketball, so I don't want to overcomplicate it. They came out in the second half and played very aggressive. We had a great control of this game going into halftime. And then in the third quarter and fourth quarter, they got a lot of our 50-50 basketballs. And then some few calls went the other way. But yeah, that's basically what I saw in the second half. You guys need to start the game better and start the second half better. Is there anything specifically that you've seen that has affected how you guys start and start the second half? No, I'm not even going to overcomplicate it, which you guys hear. It just starts with just defensive pride and using our IQs for the best of our ability. We have a lot of long guys in terms of length out there on the perimeter, a lot of teams getting in our paint, offensive rebound. And that's the game right there. It's effort. And it's just positioning and body positioning. And when you're not in the right body position in basketball, you're going to get beat. And Giannis kept beating us to the spot. He kept getting to the free throw line. And it showed for it tonight. I don't think anybody else, like no one else really did much except for Bobby in the first half. But I feel like we kept the rest of the guys at bay. And then Giannis got it going and just made it difficult for us to stop his rhythm. Once he gets going downhill and he starts getting to the free throw line, and we're sitting there almost every play looking at the refs, then we're going to get beat. So tonight was just a learning lesson along the journey. And we take one. Cary, get in there. Speaking of learning lessons, you talked about it. The last few days we've been in regression. And you were talking about it.  What can you guys do to get him to shoot it more? I just think if we're going to come in and be honest as a team, I'm not trying to just be critical over what Ben's done right or done wrong every single night. He's a basketball player. He's a professional. He has the skills to be a great professional. He's done it in the past. And he hasn't played in two years. So you guys keep coming here asking me, what about Ben? What about Ben? And it's just he hasn't played in two years because I'm from Jersey. But we stay on him. And you just stay on him. And we're just here to give him positive affirmations while he's out there and just let him hoop. And then while he's hooping and getting back into it, we're going to have to really gel as a team in order to learn how to win as a group. It's not an individual individual's job to go out there and be Superman. He doesn't have to. There's no such thing as made up anyway.  And I'm serious. Everybody wants to be a hero out here in basketball, but then it's a team game. So we're just developing. And I was a little frustrated after the game, rightfully so, emotional about competition. But it's the long season ahead of us.  I think we've had a guy score 30 on us every game. So that in itself just shows you our attention to detail, our IQ, and our body positioning is not there yet. The only thing on this one is just that the mic caught you in like a third quarter to say, shoot it, Ben. And I think people heard that. And they're seeing the team just wanted him to do more. That was the only difference in this case. Yeah, I mean, when I passed it to him, I felt like he had a layup at the rim. And I looked him out of eye. And I was like, shoot it, Ben. And of course, again, it's just a clip. It's a full game that we can look at in dissect. And that's what I'll do. This is a big picture thing. We want Ben to be aggressive every single play. And we want him to get assist every single play. We want him to rebound. We want him to play against the best player. We want him to do all the things that we know he's capable of. But at this time, he's going to have to work himself into his own confidence and feel good about himself. And I'm not going to say I'm being patient or humble about it. But the reality is that we're just going to keep having to try this experiment every single night until we get the right recipe. I like the recipe. Are there games in that five minutes? I don't think there's about four and a half minutes stretch.  We haven't got a chance to see much of that this year. Are there things in that stretch that can help? And help you guys? Yeah, I mean, the game is moving in a direction where a lot of these guys that are six-ten and above are able to do it all. And as we saw with Yanis today, I think Bud called maybe 50 pick-and-rolls for Yanis tonight, or ISO for him. And Ben's able to guard. But especially when we have him at that small ball five, we're able to switch a lot of pick-and-rolls. We're able to switch a lot of things, which I think works to our advantage. So when we see that lineup out there like that, I feel like we have a great chance of pushing the basketball on offensive end and defensively just being sound.


Now, we look at torque and angular acceleration. And we need to review a few things that we've already seen, specifically the equation for torque, the relationship between angular and translational quantities, and the moment of inertia equation. Now, I've got previous videos on each of these three. And you might want to take a look at those just to remind yourself if you're not quite sure. So we'll start with torque. To have a torque, you needed a tangential force that was offset from the axis. And as an equation, that was torque, the Greek symbol tau, the tangential force, and the distance to the axis r, sometimes called the radial distance. Now, because we had a force, we know that force causes acceleration. So my tangential force could be written out as the tangential force equals the mass times the tangential acceleration. So in my torque equation, I can put my MAT, my mass and tangential acceleration, in place of the tangential force. I'm going to look more at this tangential acceleration. We had a relationship between the tangential acceleration and the angular acceleration, such that tangential acceleration was equal to angular acceleration times the radius. Or in this case, the radius is the distance to the axis of rotation. So I can place this into my equation, where now I'm replacing that A of t with alpha r. So now I've got my equation here. And the first thing I'm going to do is rearrange the order of some of these terms. So I haven't changed the equation itself. I've just regrouped everything else out in front with my alpha behind. Now, I've got this quantity here. And from our review, we'll recognize that that is the moment of inertia, where I can use the simple capital I to stand for that MR squared moment of inertia part of the equation. So then I've got a final equation that the torque is equal to the moment of inertia times the angular acceleration. Now, I did this assuming I had just one force, but I could have more than one. I could have more than one torque acting. In that case, I just need to make sure I'm working with the net torque. So if I have the sum of the torques, the sum of the torques net torque gives me an angular acceleration. Just like our sum of the forces caused a linear acceleration. Now, let's take a look at the units for this. Moment of inertia has units of kilogram meter squared. And angular acceleration is radians per second squared. I can put those together to get kilogram meter squared per second squared. Remember, our radian is a placeholder to let us know what kind of angular units we're really supposed to be working with. Now, over here on the left-hand side, my net torque is going to be a Newton meter. And I can simplify this, or rather, expand it out if I remember that my Newton is a kilogram meter per second squared. So I have a kilogram meter per second squared times a meter. I see that these two things are equal to each other. So I have the same units for torque and for the combination of moment of inertia and angular acceleration. So that's the basic connection between torque and angular acceleration.


Do you truly in your heart of hearts truly believe that this is a human being this without a doubt without a doubt? Yes, this is a dolphin fetus So let me without a doubt a dolphin fetus human being this is a human fetus Look how similar they look what does this prove well to the world that only ascribes value based on appearance This means that a human fetus is no more valuable than a dolphin fetus But God does not ascribe value based on appearance, but rather human life is worthy of respect Dignity and of great worth to God because we are created in his image beyond that It's heartbreaking to see how excited this girl gets she's bought into the lie that abortion provides freedom at the heart of it Though abortion sinks us into further bondage bondage to sin and to guilt I'm praying that she comes to know the true author of life and freedom