Okay, so hi everyone. So we are about to start the second lecture of this workshop, module two, which is name finding of a represented pathway engine list. So this lecture is going to be focused a lot on statistic and theory, but after lunch we are going to try out this concept into a practical lab. So it's going to be easier after lunch. During the lab you will be recommended to follow our steps, but you will also be free to try with your engine list. So the reason we teach you the concepts and the statistics used by these enrichment tools because there are many tools that are available and many tools are constantly being developed and they appear and the other tools are becoming deprecated. So we think that if you know the theory then you will be able to choose the right tools that is the best fit for your data. So also sometimes it happens that you work with a model organism that is not very common and in a few cases you need to develop your own custom tool. So if you are an advanced archscriptors and you understand the theory that you probably will be able to build your own tool. So this is the list of all learning objectives for this current module. So we are going to learn on how to select the appropriate enrichment test for our data. We are going to learn how to compute Fisher's exact test, which is the test that we use when we have a defined engine list. When we have a wrenching list we will use some other statistical tests and one that we are going to see is the minimum hypergeometric test. And we will also talk about multiple test correction and we will learn how to calculate the Bonferroni and the force discovery rate. So here is how module 2 integrates within the workflow. So earlier in module 1 Gary talked about the different genelists that we get from our mixed data. So we get our raw data and then we analyze and normalize the data to get a genelist. And then we would like to functionally interpret the genelist. And the other elements that Gary mentioned earlier is annotations that are constantly gathered into pathway database. And the format to store pathway database is a gene set. So a gene set we can take the example of the cell cycle. So the cell cycle gene set we know that 500 genes are involved in cell cycle. So the cell cycle gene set will be the name of the gene set, cell cycle, followed by the list of the 500 gene names of the genes involved in the cell cycle. So then we have two elements, the genelist and the gene set. And these two elements can talk to each other and connect to each other only if we use the same gene identifiers in the gene list and in the gene set. So if you use gene name for the gene list, then in your pathway database you use also gene names. So the gene set should have the format of gene names. If you use ensemble ID in the gene list, then make sure that your gene set is also the format of ensemble ID. And then the pathway enrichment is simply to test if I have another representation of a pathway in my gene list. And then for the other modules of the workshop, we are going to visualize the result of the pathway analysis. So here is a workflow that Gary presented in module one. I just wanted to present this again to recall that our gene list can have different origins. So we do different omics experiment. We use different statistics to analyze our data.  So I wanted to mention that the pathway analysis is really the third step. So it's really one you did all your statistical analysis. And module two is within these red lines. And the two tools that we are going to see for this workshop is going to be GSE and G Profiler. So the order of the lecture will follow the steps of an enrichment analysis. First, we will describe that there is two major types of enrichment tasks. Depending on the type of gene list we could get from our omics experiment. So we could get a defined gene list. And if we are able, we can get a ranked gene list. So depending on whether we have a defined gene list or a ranked gene list, the statistics are going to be slightly different. But they will all result by a p-value associated with each gene sets. And because we don't have just one pathway, so one gene set, but multiple pathways that are stored in the pathway database, we need to correct for multiple hypothesis testing. And we will see the Bonferroni and the fourth discovery rate. So we describe two types of enrichment analysis based on whether we have a defined gene list or a ranked gene list. So a defined gene list will be a fixed number of genes, could be 200 genes, 500 genes or more. And a typical example would be a list of mutated genes. For example, genes that are frequently mutated in a set of patients. So that would be like the typical example of a defined gene list. And a ranked gene list is a list of all genes in the genome that you were able to rank using a scarring system. So this scarring system should be a value derived from your genomics experiment. And a typical example comes from RNA-seq data. So you have two classes, two groups. You compare your two groups. And you can use the p-value and the log-fold change to score all the genes in the genome so you don't remove anything. You score all the genes from the top, upregulate it to the top, downregulate it. Why is the distinction important? Because the statistical test is going to be different. And if you are able to do a ranked gene list, you may have a bit more power. So if you have the choice to do a ranked gene list or a defined gene list for RNA-seq, for example, just go with the ranked gene list. And you are going to use the appropriate test for that. Yes? Yeah, we are going to see. I have a slide with the formula. So if we have a gene list, the question that we are going to answer when we do enrichment analysis is, are any gene-set pathway surprisingly enriched in my gene list? And if we have a ranked gene list, then the question we are going to try to answer with enrichment analysis is, are the any gene-set pathway ranked surprisingly high or low in my rank list? So and you see, so here, again, for the gene list, the test is the Fischer's exact test. And for the ranked gene list, we will see the minimum hypergeometric test and a modified Kolmogorov-Smirnov test for GSE. OK, so we are going to start by the test that is very simple, is the gene list enrichment test. We have a defined gene list. So given a gene list, given a gene set, are any of the gene sets surprisingly enriched in the gene list? So we are going to see, again, where the gene list can come from and, more importantly, how to assess surprisingly in statistics. And then we will see the multiple correction. So this is a classic design, which is called the two classic design for gene list. And so we have two groups of samples, so that we name here class one, class two. So that could be cases versus control, treated versus control, or that could be two subgroups, for example, like in a cancer type. Within each class, we have multiple measurements that we did independently. So we've collected different samples, different measurements that we call the biological replicates. So each group contains the biological replicates. And then we do a statistic to calculate differential expressions. So for example, of RNA-6, the data follows negative binomial distribution, and the tools that exist to calculate differential expression are, for example, HR, OGS-6. And then you get your list of genes with the associated corrected p-value at FDR. And you select the ones that are differentially expressed, significant. You use a threshold. You use an arbitrary threshold, for example, FDR 0.05. And in this case, you get two gene lists, one for the genes that are higher expressed in class one compared to class two. And you have a gene list for the genes that are down-regulated in class one compared to class two. So that's not the ranked. And actually, that's a classic example to obtain a gene list. But if you have this design, we will see that you would use the ranked list. So this is just an example to get the defined gene list, but we will see that it's preferable to use a ranked gene list. But this one, this design, the time-course design for gene list, in this case, you cannot generate a ranked gene list. What you do is you have different time points for your measurements, and you use clustering algorithm to get the genes that get the same profile over time. So then for each profile, you'd get a gene list. So in this case, there is no way to do a ranking. So you get three different gene lists, and you are going to do your pathway enrichment independently on the three gene lists, and then you can compare the results after. So how does a simple enrichment task work? So here in pink, we have our gene list corresponding to our hits, so our differentially expressed genes. And in brown, we have the background. So the background is an important concept. So we call it the experimental background. And it's all the genes that your genomics experiment, or your experiment, could measure. So if you are doing a whole genome experiment, then the background is the whole genome. One case where your background is not the whole genome is when you do a custom macro array. So let's say you do an immune array where you just put your immune genes on the array, and then you have different conditions, then your experimental background will be all the immune genes that you put in the array, and the gene list will only be the ones that are differentially expressed genes. That could be also if you use a custom array, and on your array, you only put the transcription factors. So the experimental background would be all the transcription factors that you wanted to measure, and the gene list will be only the ones that are differentially expressed, or hits. So for RNA-seq, if you use RNA-seq, it's like a whole genome experiment. So usually, you use all genes in the background. You could restrict to only the genes that are expressed in your cell, but in reality, it is very difficult to put this boundary and see which one are really expressed or not. And then, so when you define your gene list and your background, then you test each gene set that you have in the pathway database, and the output result is a table. So it's always a table with gene sets as rows and with a p-value associated with each gene set that tells you if the result is due to random chance or not. So what is exactly assessing the p-value in an enrichment test? So it's assessing the probability that the overlap between your gene list and your gene set is at least as large or larger that you would obtain by randomly selecting a gene list from the background. So if you take 500 genes randomly from your background, do you get this overlap? So one way to calculate this is to do random sampling. So you could do 1,000 random sampling of a list of genes that are the same size as your original list. So your original list is 200. And 1,000 times, you pick up random genes and you calculate the overlap with your random gene list and the pathway that you are testing. And that will help you to build a null distribution. And compared to this null distribution, you will see where your observed data or your observed overlap is. And it's very far away from the mean that you will see, well, it's very difficult to get this result by random chance only. So you could do random sampling, but it uses a lot of resources. It's time consuming. And in this case, we know what the distribution is. So for random sampling like this, the distribution is the hypergeometric probability distribution. So we can use directly the Fisher's exact test, which is using this hypergeometric probability distribution. So that's why we don't have to do random sampling. The easiest way to understand the Fisher's exact test is to take the examples of marbles or balls. So let's say your background would be like 5,000 genes or balls. And the majority of the balls, 4,500, would be red.   I mean, blindly.  So I think that intuitively, you can think already that you were lucky. It's not easy to get these results. But to do it, we are going to do it the proper way. And in statistics, we need to do hypothesis testing. So we are going to work with a null hypothesis, which is, myelist is a random sample from the population. Or we are going to work with the alternative hypothesis. We have more black genes than expected in myelist. And so the only thing that we can do is to reject or fail to reject the null hypothesis. Because we know more about the null hypothesis that we know about the alternative hypothesis. Because we can kind of figure out what is not to be enriched, but it's very difficult to understand what is it to be significantly enriched more than by random chess. Do we need three black balls?  So if the null hypothesis can explain the data, we don't reject it. The result of the draw is random. If the null hypothesis cannot explain the data, we reject the null hypothesis. And it means that our result is equal or larger than expected by chance only. Therefore, the p-value is the likelihood that the results represent the null hypothesis. So if I have a p-value of 0.05, it's under my 0.05 threshold. So I reject the null hypothesis. I say, well, it's not random. But it means that I still have a 5% chance that it occurs by random chance only. So if I have, so the p-value range from 0 to 1, so 0 is the best p-value. I have 0% chance to get random results. And 1 is 100% that I'm making a type 1 error. So as I said, we could build the null distribution from random sampling, but we don't need to. And we apply the hypergeometric probability distribution. And for example, here in this histogram is the probability to get 0 black ball out of the 5. Here is the probability to get 1 black ball out of the 5, 2, 3, 4, and 5. So you see that the probability is decreasing and is getting closer to 0, because it's very difficult to get 4 or 5 black balls in one row. And so the p-value associated with the probability that 4 black balls, my result, is the sum of the probability of getting 4 or more than 4. So it's the sum of the probability of getting 4 or 5. So now if we adapt our example to a gene set enrichment that could look like this. So the black balls could be a gene set, the gene set that we are testing. And this on the right could be our gene list. Now we have in our gene list 4 balls that correspond to the one gene set that we are testing apoptosis. We directly calculate the Fisher's exact test. We get a p-value of 0.01. It means I have a one, I reject the null hypothesis. So I feel I think it's not by random chance only, but I still have a 1% risk that although it looks significant, I can make a type 1 error, meaning it's still random. So sometimes if you want to do, like for example, if you want to calculate the Fisher's exact test using an R function, you still need to give the R function some values. And the way to do it sometimes is just to draw a two by two contingency table where you put the size of your gene list, the size of the gene set that you are testing, the size of your total background, and this number x, k, m, and t, so t is total, m is the size of the gene set, x is the number of genes that are in my gene list and in my gene set, and you put this number into the Fisher's exact test function. This function, this formula here is the cumulative distribution of the hypergeometric function. It's just to remind me to tell you that this is a lot of permutations and the permutations, they contain factorials and actually it looks like an easy for a computer but it's not. It takes a lot of resource for the computers to do that and until now the computers were not able to calculate the Fisher's exact test from this exact formula and they will do approximation. So some approximation would be to use the binomial distribution or to use a square or to do a Monte Carlo simulation but nowadays the computers are powerful enough so I think that the tools that are developed now and in the future they are able to use the exact Fisher's exact test. And sometimes like in the enrichment tools that we are using, it's not called the Fisher's exact test, it's called the hypergeometric test and all the binomial on the car square. So when I'm not sure and I have two choice, for example the tool is saying use the Fisher's exact test or the hypergeometric test. Then sometimes I guess the hypergeometric test is going to be an approximation but the Fisher's exact test is going to be like this formula so I choose the Fisher's exact test. Isn't the Fisher's exact test the hypergeometric test? Yeah, so like the statistician, they will say a Fisher's exact test that use the hypergeometric probability distribution and sometimes the bioinformatician, they make like a shortcut and they call it the hypergeometric test. Which is an approximation. And very often because the computers they were not powerful enough, it was actually an approximation of the Fisher's exact test. So just to remind you that in our case, we ask for over-representation or over-enrichment of pathway in our gene list. So do we have a lot of genes in this pathway more than random chance but if you're interested, you could also look for under-enrichment. So is my pathway less represented by random chance? It's very rare but you could do it. You could do it for example by doing a two-sided Fisher's exact test where you could look for over and under-enrichment. And that the background population is very important if you are not doing a whole genome experiment and the results can be drastically different between a background that is set correctly or not if you do a custom microwave. And that the Fisher's exact test is applied on one gene set. So overlap between your gene list and one gene set. But what we are going to do is to test many, many gene sets that are in the pathway database. So it's independent Fisher's exact test. So we do one Fisher's exact test, another one, another one, another one. So at the end, we need to correct for multiple hypothesis testing. So the recipe for gene list enrichment test. The step one is to define your gene list and your background. Then you select your gene set and the pathways and you run your enrichment test using the Fisher's exact test and you correct for multiple testing. So that was the gene list enrichment test. Now, if you remember, we spoke about the right gene list. So if we have a rank gene list, it's recommended and we apply different statistics. So there are a few that are available, one which is very simple and we are going to see is the minimum hypergeometric test which is used in the tool G profile that we are going to see in the practical lab. And then we are going to see a modified comograph Smirnov test that is used in the method called GSEA. And then here it's listed the Wilcoxon rank sum which is the same as the Mann-Whitney U test. So two different names for the same test and the difference between the Wilcoxon-Mann-Whitney and the comograph Smirnov is that Wilcoxon-Mann-Whitney is assuming a normal distribution. Whereas the KS test is non-parametric. That's not true? Oh, Wilcoxon, yeah. Okay, okay. Thank you. So in the classic two-class design which is the best example for RNA. So we have two groups and now instead of defining two different gene list we are going to generate a rank gene list. So here is the formula that we talked about. So if we use HR or DEC we always have as output the p-value of the log-fold change. So the log-fold change we just extract the sign of the log-fold change so plus and minus to indicate if our genes have a higher expression in class one compared to class two or if it's a negative log-fold change we indicate if the expression of the genes is lower in class one compared to class two. So sign of log-fold change we just transform in a plus one or in a minus one. So positive means up-regulated, negative means down-regulated in class one compared to class two. So minus log 10 of the p-value it just transforms a small number so p-value close to zero in a high score. So let's say I have a very small p-value like 0.001 it will transform into a score of 30. So then maybe it's my top significant genes and you will be ranked number one. So it does look complicated but actually just let's say this is my top genes is up-regulated and I know it's a positive log-fold change, a very small p-value then it will be the number one in my rank list. Then I have another gene that is really down-regulated in class one compared to class two then it will be at the last position of my rank list. So then all my genes that I could measure in my whole genomic experiment will be ranked by from the top up-regulated to the top down-regulated with the non-significant genes in the middle. Yeah? So in this case, do you still have the background? No, no, because we are going to use a different statistical test and so we don't need to define the background but I would say that usually we restrict to the cells to the genes that are present in our experiment, you know, RNA-seq, we remove the low counts. Yeah, exactly. Even if they're not part of the up-regulated data? Exactly, the non-significant genes, they will have a lower score and they will be in the middle of the rank list.  Yeah? Can you explain the full change? So you would take the ratio from class one to class two and log it and then you would get the p-value from the test? So, yeah. Okay, so in the case of the... Just to give you an example of the RNA-seq, so in the case of the RNA-seq, the output of the results is from a GR and DS-seq which are using the negative dynamo distribution so it's not a t-test but we still have like a log full change that would say the way to... It's like a ratio, for example, in average in class one for my gene one, I have an average of recounts of 200 and then for my class two, I have an average of 400. So if I do the ratio 200 divided by 400, let's say I have a negative full change, log full change. And the log full change is in the RNA-seq, it's really calculated from the HR and the DS-seq. And yeah, if you had microwave, microwave being continuous data, then you could use a t-test and then in this case, you could directly use the t-value to rank your genes from top-up-regulated to down-regulated. Yes? So the value, no. We don't need it because we think that the p-value is a better estimate and so we just use the sign of the log full change. It's really to rank from top-up to top-down-regulated. We could remove the sign of the log full change and just do the minus log 10 of the p-value and that would rank the genes from the top-most significant to the last significant and we will not look at the direction of the changes. But usually, this rank gene list, we like to separate the genes that are up-regulated in class one compared to class two to down-regulated to facilitate our interpretation. So from the biological interpretation, it's different if a gene is up-regulated compared to a gene that is down-regulated. So that's why we do this distinction, but we also could do that from most significant to non-significant. Yes? Yeah, so that's the idea of the p-value that we'll look at the amplitude of the changes between the two class and also the variation within each group. So that's why we take this into account compared to log full change and the log full change usually just take into account the amplitude of change between two groups but not the replicability within groups. So the p-value should take into account the between and the within groups variability. So for the genomes and the genes? Yes. For this one, we actually pick all the genes and we generate the right genes. Yes, so people are used to do gene list with an arbitrary threshold. And what we say is when you have RNA-seq, try to avoid it because an arbitrary threshold is arbitrary. It could be that you too permissive when you choose a threshold or you too stringent and you are going to lose information. And we don't know really in the data, we don't know where to put the threshold. And so if you use a ranking list and a statistical test that use a ranking list, the statistical test is going to work on the threshold by itself. So it's probably going to be better at finding the natural threshold. So I just showed the two gene lists because this is what people used to do, but we recommend to do a ranking list when you can, for example, RNA-seq for sure you can generate a ranking list. There are other cases like the time course experiment or if you have a list of mutated genes where you cannot get a ranking list. In this case, you have to use tools that take as input a gene list. And the key value for the genes are actually the adjusted key value for the genes. So for you... Actually it's a good question for the rank list. We don't use the adjusted key value. So the adjusted key value, we would use it to select genes, but we don't use it here because it has, the way it's calculated and we are going to see this in a few slides, it has a lot of ties. Like you can have a lot of genes that have the same corrected key value of 0.4. So then 0.4, 0.05, so you have a lot of ties when you get to the adjusted key value, which, for example, the FDR. So that's why it's not the best way to use it in a ranking list. So the fourth discovery rate, the FDR, you use it when you want to select genes. Right. So I have 5 ranking genes, I have FDR.0, 0.05, and yeah. So how do I know? So if I run a differential expression analysis, I get a list of genes, right? Yes. So the genes are very, very long. Yes. And to each one of those genes, I have a key value and an adjusted key value. And an adjusted key value. But the key value can range from 0 to 1. Yeah. So, but the genes that are not so difficult significantly can have 1. Yes. So what's the kind of for the key value? So in the ranking list, there is no get-off. You take all genes, if you have 15,000, yeah, 15,000 genes, you rank all your 15,000 genes. 15,000 genes to begin with, like the background, okay? Yeah. So all the background will be, okay, got you. Okay, so the recipe for ranked list enrichment test is rank your genes, then select your gene set, run enrichment test, and correct for multiple testing if necessary. So here, we speak about the multiple correction after the gene set enrichment. So the FDR of the gene set, not the genes. So then the same as the same diagram as we saw for the gene list. But now instead of having a pink and a background, we have a ranked gene list. And what we look is if our gene set is more at the top or more at the bottom of the list, we use different statistics, and then we have a table at the end, which is the same, with the gene sets listed and the p-value associated with these gene sets to see if it happened by a random chance only and the probability of making a type one error. Yes. So it means that those genes haven't changed that much. Like a small change is what we consider as part of the gene set. Here? Yes. So for this gene set to be, let's say, enriched in my data or significantly enriched in my data, many genes in this gene set should be at the top of the list or the majority of the genes in my gene set should be at the top of the list. And if this gene set enriched significantly enriched in my down-regulated genes, then it should be at the bottom of the list. So the majority of the genes that are in the gene sets, so for example, the cell-cycle genes, I should have many of these genes at the bottom of the list. So, yeah. I'm thinking of genes like transition factors, like a slight change causes a lot of changes downstream. Yeah. So if you don't see, maybe, you don't see such a huge change, but it's important that small change is significant. Yes. Yes. It depends on the rank of the list. Yes. And GAC is going to be a good method for that.  So for example, GAC is going to calculate a cumulative sum. So even if we have a small change in gene 1, but plus a small change in gene 2, plus a small change in gene 3, is going to increase your enrichment score. So it's going to get significant because of the addition of the small changes. So in this case, it's going to be significant. So one goes for the rank geometry test that we are going to see in G-profiler. So what this test is doing is going to calculate a Fisher's exact test, but multiple times. And so it's going to, we're going to have our rank list, for example, rank from most significant genes to less significant genes. And each time we get a gene study that is in our gene set, a Fisher's exact test is going to be calculated. And each time we are going to get a p-value for this Fisher's exact test, and a minimum hypergeometric test is retaining only the threshold with the lowest p-value. And because we just do multiple gene sets, we need to cry for multiple testing. So here's the graph to try to explain it. So the rectangles, the black bars are illustrating the rank list. So for example, the first black bars here is my gene number one, is my most significant genes, and then from the last significant genes. And here is the gene set, the pathway that we are testing. And the red bars are all the genes that are in my rank list and in my gene set. So now for the first red bar, the test is going to calculate a Fisher's exact test. And we are going to have a p-value. So it's going to calculate of one gene that is overlapping between my gene set and the first five genes of my rank list. And I get a p-value. Then it's going to go on. And at each point, calculate a Fisher's exact test. And maybe here is going to be, well, I have 10 genes in my gene set. And it contains the top 20 of the gene list. So 10 out of 20 gets a p-value. It could be that this p-value is better than the p-value at the beginning because you got more overlap. So you got more overlap between your gene set and your gene list. So this is the way the minimal hypergeometric test is working. So this is the black bar. This is the rank list. It's bar is a gene. It's bar is a gene. This says, let's say, so this one is a rank list that contains 200 genes. So that's not the full whole genome rank list. That's the differentially expressed genes. And we are going to see in GIPO file. So let's say you've selected 500 genes. But you can rank them using your p-value from the top significant for the best p-value to the last significant. So you have a gene one, which is the best. And then this gene number 500, which is the last significant. And then you are testing this pathway, gene set, that contains maybe, I don't know, 30 genes. So 30 genes that are in the gene set and in my rank list. This is my overlap. And now the minimum hypergeometry test wants to see what is the best threshold to stop and get the best p-value to say that this gene set is in reach in my list and will tell you the threshold. It will tell you, well, the best threshold, the p-value 0.001, I will get it when your gene list has the size 150. The more genes you get, the more likely you will have overlap with the genes, right? You could think the more genes you get, you have more overlap. But you need to think also about all the genes that did not overlap. So at the beginning of the genes, you may have two genes that are overlapping between your gene set and your rank list and 10 genes that did not overlap. So it's not so good. And then you continue, and then you may have 15 genes that overlap and 20 genes that are not overlapping. And then the ratio is going to increase, and you are going to get more overlap compared to non-overlap.  So that even if you increase the size of your gene list, so here, you are not going to increase the ratio between the number of genes that are overlapping and the number of genes that are non-overlapping. And that's what the minimum hypergeomics test is trying to do, is trying to tell you the threshold where you have the most enrichment. And you do it when you choose permissive threshold.  So you want to put more weight on these genes that are on the top of the list. So that was the simplest method. But one very popular method is the GSEM method. And this worked best, so in this case, with the whole genome. So when you can rank your genes from the whole genome, so not your 500 genes. But the whole genome. So you choose a modified Kamogorov-Smeonov test, and it works by calculating your running sample. So now, the black bars will be all your genes in the genome, ranked from top up-regulated to top down-regulated. So again, the red bars will indicate the genes that are in this gene set that we are testing. And where they are located in our ranked gene list. And so we will start at gene number one. And gene number one, the running sum is going to be 0. And then gene number one is not in the gene set we are testing. So the running sum is going to stay 0. And then gene number two is still not in the gene set that we are testing. The running sum is staying 0. But then at gene number five, the gene set is present in our gene set. So the running sum is going to increase. And if you look at the graph, you can see there is a higher density of genes in this region. So that's why the peak here of the running sum is going to increase a lot. So oh, genes in the gene set, genes in the gene set, genes in the gene set, genes in the gene set. So it increases a lot until it reaches a point. And now we see a decrease. And we see a decrease because the density of the red genes is decreasing. So now we don't have any... these genes in my rank list are not in my gene set. So it's decreasing, decreasing. I have a little bit peak here because we have this red gene here. But after that, it's decreasing, decreasing, decreasing. And it's going up again and decreasing, decreasing. And here I have no genes in my gene set, so it's a complete decrease. So that's the way the running sum is working for GSEA. And so the maximum point here is called the enrichment score. So usually we will see when we run GSEA, we get a first result which is the enrichment score and the normalized enrichment score. So this is another example for GSEA. So I have here the rank gene list. So the genes are ranked from top up-regulated to top down-regulated with the non-significant genes in the middle. And I have three gene sets testing. And the gene set number one contains eight genes. And we can see in this red bar that the eight genes are more located at the top of the ranked list. So it's going to be a positive enrichment, an enrichment to walk the up-regulated side of the rank gene list. And on GSE plots, it looks like this. So now my rank list, you flipped it vertically. So here is my gene number one. This one is my last genes. This is the side of the top up-regulated genes. And this is the size of the down-regulated genes. And the black bars are all the genes that are in the gene set that I'm testing. And the name of the gene set is here. This gene set number two contains also eight genes. But it's not significantly enriched. So I don't have a GSE plot for that. And it's scattered randomly throughout the rank list. So you can see a gray gene at the top. You can see other gray genes at the middle of the rank list where the non-significant genes are located. And you can see a few at the bottom of the list. So it's not significant. There is no direction toward the up or the down-regulation list. And the third gene set, number three, contains eight genes. And we'll see that it's enriched towards the down-regulated side of the rank list. It will look like this on a GSE enrichment plot. And here is the zero. So it's negative. It's a negative enrichment score. Yeah. So the bars here, are they in the gene set at our rank list? Yeah. Yeah, so exactly. So that's all the genes that is contained in the gene set that we are testing. And that tells you the position in the rank list. That contains all the genes. Usually, they are being filtered out by GSE at the first step, so we don't see them. And there is also one thing to mention about GSE. We say it's a modified Morgarov-Smirnov test because it has a weight. And the weight is to want the genes that are at the very top of the very low end of the rank list because we don't want to have pick in the middle. And we use the expression value. So in our case, it's the minus log 10 of the p value to get this weight. So at the beginning of the list, when the running sum is increasing, it's increasing more. For example, it's increasing from 0.2. And when we reach the middle of the list, then as it uses the score, like your expression score, the sum is not going to increase a lot. It's going to increase of 0.001. So it's a way for GSE to avoid pick in the middle of the list. So we always get picks that are weighted towards the top-regulated genes or down-regulated genes. And there is also an option in GSE, which is called weight. And so we can, if we are not sure, and we say, well, I'm not sure GSE maybe is going to take into account genes that are not very significant, you can upgrade this weight and say, I want to put more weight for the very significant genes. So you can do by changing the weighted parameter. Do you know when I want micro-scoring GSE, I don't get to know the old things. And even though there's the possible to get the original micro-scoring? Yes, you can change the parameters. And instead of setting top 20, you set, for example, top 200. But what I do sometimes, I select my gene sets that are significantly enriched at my FDR. But I'm really interested in this gene set. So I basically extract this gene set from my pathway database. And I rerun GSE on this only. I do it when I do like a figure, something like that. But by default, I use top 200 instead of the top 20. The only thing is when GSE has finished to run the analysis and it creates a report, it takes maybe a few seconds more to finish GSE. Because sometimes you have like 500 gene sets that are significantly enriched. And you cannot see these plots. And it's very, very important for quality to go back and look at these plots. Yeah? I think you already mentioned at the beginning that the overrepresentation analysis is more popular. And I think I've seen also a lot in comparison to the GSE analysis. Which one? The overrepresentation analysis, the rich analysis that you do at this time. But has anyone compared, having the same data set, say, OK, I'm going to do with this one. What best is playing my research? So if I understand, so you ask someone compared like you have differential express data, you do your two discrete gene lists with your arbitrary threshold. And you do an over enrichment analysis. And you compare with GSEA. So yeah, we do all the time. And that's why we recommend a ranked gene list. Because we get more results with the ranked gene list. Because it's very difficult to set up an arbitrary threshold. And also, it is related to exactly what you mentioned with the transcription factors. Sometimes you have very, very subtle changes. And a gene has just a little change. It's barely significant. But all the genes in this pathway are changed a little. And you are going to lose this information if you use an arbitrary threshold. But GSEA is going to get this information. And this pathway is going to be significant in GSEA. Or let's say you use the arbitrary threshold. And you get eight genes in a pathway that pass the threshold. But two were just under your arbitrary threshold. They were just ranked just below. You missed it the first time. Your gene set did not get the significant. But you use GSEA, then you get these two additional genes. And it's going to be better. The only way I don't use GSEA when I have arenasic data is sometimes when I have kind of noisy data. And I had to clean the data, maybe using the law for change. Or maybe by using additional information, prior knowledge. And I have maybe merged with, like, mutation data. And then I have no choice. I just have a gene list. Sometimes when it's very noisy and you say, well, I don't know about the other gene, but I just trust this top 50. Then I just stop here. But usually ranked gene list is always better. Can you use the GSEA with a clean data? Oh, yeah. So let's say. Oh, yeah, for sure. It's recommended. This one is very, very noisy. And I just select my top 30, top 50. And I say, well, the rest maybe yes, maybe not. But usually, like 90% of the time could be clean or less clean. I use GSEA. Even for noisy data, GSEA is better. It's just when it's extremely noisy that you don't want it. Let's say you have a gene list from your experiment. And you want to clean it, filter it according to your law for change. Yes. Can you give the filter to GSEA? You could. Instead of using the sign of the law for change, you also could use the law for change value. I never do it. But if you have a good argument to do it, you could, because you are doing the scoring system. So if your argument and your calculation is valid, then you can do it. There is no one way to do your scoring system. So I give to the algorithm a list with a p-value and local. You do it before GSEA. So the scoring system, you do it on your own. So what you give to GSEA is two columns. One is the gene names, and the other one is the score. So the score, I gave you the minus log 10 of the p-value, multiplied by the sign of the law for change, by example. That's what we do. But if you had a reason not to use this score, but another score, then you could use it. So we don't recommend this option. I know it's possible with the classic GSEA that you can input a matrix of rpkm, and GSEA itself is going to calculate the differential expression using metrics like a spin-off to noise ratio or t-value. But we don't think it's the appropriate way to calculate differential expression in RNA-6. So because of the count data that are discrete value and not continuous value, so we think it's better to use the negative binomial distribution in HR in DSEC and then to create our own ranking list. When you get the output of HR or DSEC, even if you don't script in R, you could do it in Excel. And we are going to show you example in the lab practical. So but now the running sum that we just explained here was the peak, which is the point where the cumulative sum is the highest is just a score. So it's not a p-value. So how do we go from the ES score to the p-value? So here GSEA, which had the beginning new statistics, now is using random permutation to get the p-value. So each time we test the gene set, we set the permutation to 1,000. What GSEA is going to do is to replace the genes that are in the original gene set by random genes. And it's going to do it 1,000 times. And each time it's going to calculate an enrichment score for this random gene in the gene set that we are testing. And from all this random enrichment score, it's going to build the null distribution. And then from the null distribution is going to see where is our observed enrichment score for the gene set that we are testing. And for example here, so when we do permutation like this to calculate the p-value, we call it a perical p-value. And here it says, well, we got an ES score that was for we got only four results randomly that were equal or higher than my random score divided by the total number of permutation. So that's called the gene set permutation in GSEA. And this is the only permutation we can use when we use a rank gene list in GSEA. So we have seen the hypergeometric test for discrete gene list. We have seen the minimum hypergeometry test that is available in G-profiler. When you have, I would say, like a list of 500 to 2,000 genes, but you want to put more weight on the most significant genes of your defined gene list. And then we have GSEA where we can rank all genes in the genome using a score derived from our genomics experiment. So now, because we use to test multiple gene set, we need to speak about the multiple test corrections. So back to our example of red and black balls.  And the results we got is four black balls and one red ball. And we got, if you remember, we got a very, very low p-value for that, like maybe 0.01. So we just had 1% of chance to get this result randomly. So it's very difficult and unlikely when you do one draw to get four balls. Yes, but that's when you do one draw only.  So even if it's unlikely, you can get these results. That's why we need to test to correct for multiple tests. So now if we adapt this to the gene set enrichment analysis, then the squares are one gene set. And we don't have many squares compared to the round balls. So it's very unlikely to get these squares. But if we test again and again all these gene sets containing these square genes, we increase our chance to get it. So we increase our chance to make type 1 error. So that's why we need to test for multiple hypotheses. So one intuitive way to do it is basically just to get our original p-value and multiply by the number of tests we did. So we tested 10,000 gene sets. Then we could multiply our original p-value by 10,000. And it actually exists. And it's called the Bonferroni correction. And the Bonferroni correction is known as the most stringent method. So the Bonferroni correction, we say that we are controlling for the family-wise error. And it means that when we select the pathway under corrected p-value of 0.05, we say that the probability of any one of them to be a type 1 error is 5%. So usually in the enrichment tools, you will see this Bonferroni correction. And you can try it. But what's happening is that it's so stringent that usually no gene sets pass the threshold of Bonferroni 0.05. So there is another way to correct for multiple hypothesis testing. It's called the force discovery rate. And so the force discovery rate is the expected proportion of the observed enrichments due to random chance. And the method to calculate an FDR is called the Benjamin-Horberg method, actually the one that we are going to see. And so at the end, we obtain a Q-value. So now we are going to see the calculation of the FDR. And that's the last step of the lecture. So first, we have a gene set and the nominal p-value, which is the original p-value that we got from the test. And the first step is to calculate the adjusted p-value. So we ranked our gene set from the most significant, so the lowest p-value, to the last significant. So here, the last significant has a p-value of 0.99. And we tested 53 gene sets. So to calculate the adjusted p-value, we multiplied the nominal p-value by the number of gene sets that we've tested, divided by the rank. So here's rank number 1, 2. So here it's important to understand that the more gene sets we are going to test and the more pessimistic the adjusted p-values are going to be. So yes, we need to correct for multiple hypothesis testing, but by increasing the number of gene sets here, we inflate our adjusted p-value. And then the last step is to get the Q-value. So to get the Q-value, we start from the bottom of the list and we see which one of the p-value is the lowest. So here, 0.99, there is nothing below that, so we add 0.99. Then we go up and here we have the adjusted p-value of 1,0.04. But because below that we had a value that was lower than this, we put 0.99. And we go up and up like this. So for example, here we had 0.99, but the adjusted p-value is 0.053, so we put it. But here we have also this 0.053, but below that we had a lower number, so we still put 0.04. So now the adjusted FDR, the Q-value, we rank from 0.99 to the best one, which is 0.04. And you see all the ties here, so that's why we don't use it when we get the rank list. How did you get from the 0.053? So here, for example, so here we had 0.053. Because we had here maybe 0.99, 0.053 is less than 0.99, so we put 0.053, because we know it's more significant than the other one that is below. And here, so just one rank above this one, we had 0.04. But it's lower than this one, so we can keep it. And then we go one rank above this one, but in this case, it's higher than 0.04. So it doesn't make sense. So we take the smallest value and so on. So we always take the value equal or smaller. Nominal p-value. So now, if we want to select ginsets that are enriched under the FDR 0.05, then we get here the threshold. And we report this top number for ginsets as significant under the FDR threshold of 0.05. So again, this slide is just mentioning that the more ginsets you are testing, the value are going to be pessimistic. And if one way to filter the pathways could be to remove, for example, a pathway that contains two or three genes, because sometimes the pathway that I base, we have very small pathways that are not very informative. And we also have big pathways that contains more than 500 genes or more 1,000 genes. And if you remember what Gary said about the gene ontology, which is very hierarchical, so if you used the gene ontology as your pathway that you base, you may have parent terms like a cell cycle or regulation of cell cycle, which maybe yes, but maybe not the most informative. So a way to filter a pathway is to remove the small size and the big size of the pathway, and it's going to reduce the tests that you are doing. OK, so we did the future exact test, and an orange gene list, and a multiple test correction on the bone pharaony. So bone pharaony, very stringent. If you get results with the bone pharaony, you are confident to report this gene set. If nothing passed the bone pharaony, then you can try the FDR, which is usually what we do. And I think we did all the objective on module 2. And so what we learned is that the typical output of an enrichment analysis is always a table. And the tables will always contains all the gene sets, significant or not, and always a measure of the size of the overlap between this gene set and the gene list. So if it's a defined gene list, that could be the number of genes between your gene set and your gene list. If it's GSEA, that's going to be the enrichment score. If you have a big enrichment score, large value, it means you have a large overlap at the top of your list between your gene set and your gene list. And then the p-value. And you always have to get an adjusted p-value. It could be bone pharaony. It could be FDR. But this is the corrected p-value that you are going to use to select your gene sets. So it's the same. Like it's a tabular format. Yeah? You mentioned before an enrichment score, but also an enrichment score. Numerous enrichment score, yeah. It's normalized by gene set size. Yeah. So the enrichment score is when you test only one pathway. So let's say you've decided to test only cell cycle. You will just look at the enrichment score. But if you test multiple pathways and you want to rank the gene sets, then they have to be corrected for gene set size because some gene sets are smaller than others. So it's a normalization that GSEA is doing to correct small pathways and big pathways. So if you're in GSEA and you test more than one gene set, you should use the normalized enrichment score. Just to compare gene set one and gene set two. So usually we have a table as an output of an enrichment analysis. What we need to know is that the gene set, they are usually frequently overlapping. So some genes in one gene set are going to be in gene set two. But using a table is not very easy to identify the function and the gene set that are related. So that's why we usually need to go to network visualization. And so many tools are available to do enrichment analysis. Some are web based. So you just connect to a website and you copy and paste your gene list. Some are within the site escape. So we'll see some site escape application can do the enrichment analysis. Some tools are standalone. So you need to download it on your computer. So you probably did it today with GSEA. And some are available through our packages. So depending on what you prefer. And I like this Omic Tools website. There's a free registration, but you can search and give you a list of tools. So now we are going to see G Profiler and GSEA as example. It could be that depending on your data or depending on your model organism or personal choice, you want to choose a tool. So when I look at the tool, I try to answer a few questions that are correlated to the concept that we've seen today. And a few questions are, so does the tool cover your model organism? So that's the first question. Is there a good choice of gene sets path with database? So we usually like to test different sources of path with database. So some tools just use go or just react them. But we like when the tools are using multiple sources of the pathway database up to date. So sometimes the tools have been, has been developed like a years ago, but there is no one to maintain it and the pathway database are not updated. And that's not, we are going to lose information if we don't use up-to-date databases. So which statistics? So now you know, is it a gene list and or a rank list? Usually tools have two options, like the basic gene list, where I just copy and paste my gene names. Sometimes like intermediate, where I could put my gene list and a score beside it. Or sometimes they have like a full rank list statistics. And then do I like the output style? So we know it's a table, but sometimes like some, some tools like offer some network visualization or like better tools. And then because of this workshop, one question is, can you connect the output table with tools like cytoscape? Because as we will see it's very, it's easier if you can visualize as a network. And many tools, if you can download the results with the gene set name, with a p-value, then you can use the generic format, for example, of the application that we are going to see this afternoon, which is called enrichment map. So usually most of the tools, you can adapt it for the use in cytoscape. So these are topics that are not covered in this lecture, but if you want to know more about some issues with the enrichment test, we could speak about correlation between gene set and dependency of genes. There are some tools that are using the topology of the networks. And so we don't cover this, but these issues are existing. And if you want to dig more into this issue, then I can recommend this link of the protocol. Yeah, and if you want to know more about the Fisher's Exact Test, I like this little video of the MNFs. Actually helps me a lot. And the different colors of the MNFs would be the different gene sets. And the whole bag would be our background universe. So I think I put a link to the video and I think we are on a lunch break.


And then I give the floor to Siegfried. Yes, so welcome to our session on four years of OER info. I am my name is Siegfried Fahre, and I work at the DIPF. This is the Leibniz Institute of Research and Information and Education in Frankfurt, Germany, and I am the project coordinator of OER info. And we are very happy here to discuss with you our experiences and our learnings in the four year journey we did with funding of OER in Germany. I will give you a brief overview of our talking parents and by that I will introduce my fellow panelists. This is how OER info came into being, which is presented by Jan Moymann from the University Library Center of North Rhine Westphalia situated in Cologne. Maybe Jan you can give us a big wave, so we know where you are. Everybody. And then I will just give you a brief overview on our goal strategies and the architecture of our OER info project. And then I will give the word to Gabi Fahrenkruck of the agency, Juran on Constantin. She will tell us about working with the OER community in the OER info project. And Juran himself of the same agency will introduce us to the OER camps and the networking he did in the OER sphere. Then Daniel, then again Jan will tell us and give us an insight into connecting OER policies. And Daniel Otto of the University of Duisburg-Essen will follow with an outline of our transfer strategies. Then I will talk about our information tasks, mainly our OER portal, and that will it it as you can see OER info is a joint effort by many organizations and diverse educational fields and how different initiatives were molded to one big project OER info Jan will tell us now. Yes, exactly. So I'm going to give you a very brief history of OER info and Siegut already mentioned that I will today especially speak about a policy perspective and so that's not completely true for the history I'm going to give you now. And but I will try to focus a little bit on on the policy perspective here and I'm also trying to focus a little bit on the pre history of OER info because when I reflected on presenting the history of OER info, especially from a policy point I thought that the pre history might have been perhaps even more interesting than the history of the project itself. So I'm going back actually to 2011 and I identified two starting points for for OER info and and it's quite interesting to see that one was a big scandal, which was triggered by the publishers who tried to survey teachers in Germany with software and this cost quite a lot of resistance on the side of of the teachers and out of this came a movement of teachers who who thought about how can we handle resources better and how can we avoid being surveyed by by publishers when doing our job. And so this was actually, I guess the starting point for the bottom up movement in Germany. And at the same time, and we saw 2012, the UNESCO World Congress and the declaration on the we are, which was obviously more button top down approach. So two things came together so that we had a very luckily I would say coincidence of forces coming from bottom up and from top down and declaring. And as far as the World Congress is concerned I would like to remember Fred Mulder who had a saying at that where is Germany because Germany was missing at the at the World Congress, and this was distributed into Germany and so there was a perceived lack of presence and and so I think policy made makers had the feeling of not being a had enough and and having forgotten something, and they tried to to catch up with with movement in other countries. And I think this I mentioned this because I think it's something which worked out for policy making and it might be something like a strategy which could be used to motivate policymakers in other countries. So after this, we saw four years face of discussion. And, and this is quite interesting as well became because also you can see here, impulses came from from many different institutions and and we had, after the UNESCO Congress, the starting of experts hearing by the Federal Ministry of Education. And, and there were several other others following by it so experts were being invited and and the topic was discussed. In 2013, we saw the EU initiative opening up education, which brought new impulses from from another multinational institution I already mentioned UNESCO. Additionally, there was the EU and also the OECD, which all was pointing into the direction of our medication. And so I think that that germ policy making felt forced to to get deeper into this topic. And so there were initially three studies funded on on OER I think one was of the international situation. And the second one was on legal aspects and the third one was on a meter data which is quite interesting because it's a specialist topic which was very addressed early in Germany. And, and I think it's it's characteristic for we are info that there has been somehow a strong focus on on on national infrastructure from beginning on. In 2013 we saw the first OER conferences, and this was connected to a bar camp. You're on had already started bar camping back in 2012. So this is actually the origin of the bigger point, big part of the bottom up movement which was being supported by the bar camps. And so 2013 we saw the first conference, which was provided and run by the media at that time, which was another important player who pushed the topic and and there were much discussions going on on these conferences and so this I guess was another important thing to build up pressure and proceed with the understanding of the very complex topic of open education open educational resources. In, I think it was 2014 we saw the announcement of the Hamburg open online university so there came individual states doing first lighthouse projects on OER and announcing getting into the topic so things started to get into practice. And then politically it was quite interesting that the topic was addressed to a rather low hierarchy working group which consisted of members of the federal ministry and the Kultus minister conference which is the representation of the states. And so it was a little bit out of focus and under the radar, and they really provided very good very good work and provided a very good short but good report and which, which recommended the introduction of OER. So and this actually, so you could you switch the slide please. This actually led us to the, to the core funding line, which started 2016 with the phase one and and we had two pillars of these funding line. The one was a central information site which is OER info and then probably some of you know it. And the other one was to 26 train the trainers project so we had a very broad approach focusing on capacity building on on building up personal capacity to handle we are, and this I think was quite important to see that it happened in in every educational sector so so one special thing about OER info is that it addressed all at the educational sectors. And so there was a little bit of preparation there, and the first phase ended by the end of 2018 so there were I think some few of these train the trainer projects, which proceeded a little bit longer but but to simplify you can say and of 2018 is the end of phase one. And then the project was continued but only for the OER info information site so and and also for for jointly which which another project coming out of phase one, which supported community building, but these train the trainers project have not been continued in phase two. And yes, maybe you have already heard it but I can say it right now here and introduce it as an important information. OER info is currently not being continued. So they, I think there are still being discussions going on on how the project can be continued, but it's a quite interesting thing to see that despite many successes which we will see in the presentation here. And there's currently no decision to continue this work and I think it's something which which depends on the. Yeah, if you're dealing with policy making you have to to accept that there are many irrational things taking place and that things are not going planfully and rationally as you hope, sometimes. And with this I'm coming to the end of the short history of all your info and give a give back to see good. Thank you. And as Jan pointed out, the situation in Germany before 2016 was that the subject matter of OER was far from being implemented in educational practice, and this was practically the goal of OER info to, yeah, to accelerate the uptake of OER in all educational sectors. To achieve this goal, we developed a strategy that consists basically of three pillars. The first one is information. As Jan mentioned before at the core of OER info is the OER info platform, which provides different groups of audiences with information on OER. We have information on licensing and on finding OER on how to produce a worksheet, etc. The information is presented in very different formats as text and video and audio and the entire content is of course under CC licenses and can be used in other contexts. Basically, this is meant to help others help themselves as a means of getting to know more about OER. Then the next pillar is the transfer pillar. That means basically that transfer of know-how into different educators via workshops and events. Daniel will talk about this in a moment. And third, the networking, which itself can be maybe divided into networking community and policy. The networking is taking into account different stakeholders to bring together early adopters, OER experts and of course an interested audience in formats such as barcams. Joran will talk about that. And then we have the community work targeted at activists and leaders and OER doers mainly via social media. And third, of course, is advancing OER by knowing what's going on. So the collection of all kinds of information and we collect all kinds of information and try to weaponize them so to speak for policymaking. This is the third strand, which is basically done by Jan and the OER world map. The OER info architecture, Jan just talked about it briefly, consisted of two faces, one from 2016 and 2018 and the second one from 2018 up until now. Seven institutions are at the heart of the project, working closely together, implementing the strategy. So come from different backgrounds and five of them are here. The institution addressing vocational training and the institution addressing adult education and the institution that addresses the schools are not here today, but we're trying to talk about their experiences as well. So I think our strong suit is working across educational fields and doing both addressing target audiences such as teachers, but also fostering intersectional exchange. We also have to network and initiatives the OER camps, which will hear in a moment from your own and the initiative jointly, which is about community building but also it's a big platform that provides OER coaches with materials. And from 2016 to 2018, 23 individual projects were also funded with the aim of capacity building in different institutions like universities or among teachers. And this entire program is funded by the Federal Ministry of Education and research and as Jan already told the sad news where info ends at the end of December. And we are looking for means to continue the project or at least continue the website. We are very hopeful still hopeful that we can achieve this and we are in for care will continue to exist. And as a secret, there's a question from a Roberts. Could you say a little bit more about why the OER info will not be continued. Is there a clear reason for that. There are several reasons for it. And one is, as you're pointed out, in Germany, the education system is quite complicated you have the Federal Ministry, which cannot really target schools, but can work on a, yeah, on a rather general level or general level. And then you have the ministries of education in the States, and they basically have the power on what's going on in education. And those two sections are always fighting or not really working together. You can have good examples of that during the corona crisis. And, and they could not you know, they can't be united on the subject of OER. So, the federal government, which is our side, the something and now the states are doing other projects in all we are so this is I guess is one. One reason why we are it's not continue to be continued at this point. And then another reason is the corona crisis. The federal government is very much preoccupied on dealing with it and unfortunately we could not make them see that OER is a solution or a motor or very big part in in solving the crisis but for them it's just a subject on top, and where it's no personal resources or no money resources to deal with it just right now. And Robert, you suggested there's too much grassroots. I cannot say that for my point of view. I would not say that this is is is one of the reasons why it's not funded, because in the or info project. The building server where I work and also the FV we are on and the, the, the vocation training and the adult train we are all institutions of the government so and no really grassroots organization and Yeah, well, I think so think the broadness of our education sector is rather an asset than hindrance. Yeah, well, and we still think that there might be funding to come. Okay, I will give over to Gabi. Hi, I'm going to find hope for from your and a consultant and we've been working for. I've been working for the OER info blog editorials team since the launch of information to tell OER. And, and as we have already heard from secret. The aim of the project was to build a web platform around the topic of OER and the task of the blog editorial team was to compile knowledge about OER for the community and to support the exchange of expertise within the community. To ensure a sustained and broad visibility of OER and to create a contact point for the German OER community via this platform. A few facts and figures about our work may help you to get an overview. Jörn had already been operating the predecessor of OER info, the OER transfer stelle since 2014. And therefore when OER was launched there was already some content on which we could build. Since 2016 we have been producing and publishing this content and these formats for OER info. It was more than 500 blog posts, 44 podcasts, over 180 videos, mainly interviews and explanatory videos and then web talks, newsletters and so on. The content in the blog was accompanied by articles in the social media channels of OER info. On Twitter and Facebook, for example, we referred to the blog post. For this purpose and also for the exchange of information among the OER community, the hashtag OERDE was introduced at the beginning of OER info project. The hashtag refers specifically to German speaking world and it is intended for community exchange discussions, announcements and questions on all aspects of OER. Right now the hashtag OERDE is well established and widely used. In order to achieve the project objectives it was necessary to present the current state of knowledge on OER and the German speaking world, but also internationally. This and we provided ongoing reporting on current issues on fundamental questions and we also reported from international conferences such as the OER global. The aim was to present the diversity of the various initiatives and approaches and thus to support an exchange between the initiatives. We presented the various OER projects, their respective objectives and also introduced the people behind the projects. So much for what we have done over the past four years. In regard to the project goals I can say that with constant and up-to-date reporting with media partnership of the OER comes which Jörgen will talk about later. And with the fact that we have provided many different formats, we've been able to reach a great many people with the topic of OER. The information stelle OER is perceived in the educational landscape as a contact point, even though much more work is needed to make OER mainstream at some point. We have created a central contact point for the community which should be further developed in the next step I think. With the information stelle OER where to be continued at some point in the future, I believe that the few information service should be supplemented by an advice center for questions and issues relating to OER. And this brings me to the question of what could have gone differently or what could possibly be done even better in the future. Well, it has been shown that full of information such as that provided by OER info is well suited for getting started. However, when specific questions about open materials, creative comments license or other topics need to be clarified, people often do not know who to turn to. There are a lot of contact points that offer advice and concrete assistance on OER. In the best case, the information stelle OER should have been help center. For example, by referring to experts or by pointing out the places on the internet, which could offer answers to respective questions. I know that OER librarians are already active in some libraries, but some universities in Germany also have coordinators for the topic of OER. So it would be conceivable to have a scenario where there is a contact point for questions within the information stelle OER. OER could then pass these to onto libraries or universities where OER experts work and who could who would then help with specific questions. Well, have the objectives set for the community being achieved? I'm not sure. Since providing advising and information on OER as an ongoing task, the question does not seem to me very helpful. But yes, because it's true that OER info has brought together knowledge on OER, has supported the exchange of expertise and has ensured that the topic has a lasting and broad public visibility. And the contact point for the Germans speaking OER community was also created, but the provision of advice and information on the range of OER topics should be an ongoing task. But what remains of it now that financial support has ended and it is completely open whether OER info can be revived and at some point in the future. Without a permanent information stelle OER, all effects will fizzle out. Even if the website remains available on the web, much of its content will soon be outdated, I think. So, that was my part. Thank you. And now I hand over to Joran, who talks about the networking aspect of about four years of OER camps. Joran, muted Joran. Okay, so I'll try to give you a five minute version of the OER camps in Germany. For anyone who's interested in not only the five minute and the bright version, there's a recording of a talk I had earlier with a 25 minute version on the OER camps. I share the link with you on the chat in five minutes from now. Okay, I brought you mainly pictures from OER camps. So this is how it looks like this is one picture from the first OER camp in Bremen in 2012. And it was mainly about people coming together because there were not such a thing as a community until then. So in Bremen in 2012, I think it was maybe 70 people and that was probably like 90% of all the people interested in OER in Germany at this time. And it was just thinking when when Jan was talking about that he just arrived from Paris, I remember, and gave us the insights from the UNESCO point of view. And for all of you who do not know about the bar camp idea, the OER camp. So this is a bar camp. So it's about people coming together and sharing their own questions and their own experiences. So this is a picture from the last in real life event we had earlier this year in Hamburg. And I really like the picture because it's very much about what bar camps often look like. So it's very much sharing and talking to each other and working on your own questions and even your own resources. So the bar camp begins with the people coming together in a plenary session and I don't know if you can see it at the end of the room there are people lining up. This is normally the first hour of a bar camp. Everyone who wants to provide a session which is bar camp language for a workshop lines up and steps forward and says I want to provide a session on this topic on this question and it's not unusual to only have a question. I look forward and I say, Hey, my name is you and I'm currently working on this question and I'm looking for other individuals in this room interested in the same question and then it's not about finding many as many people as possible but finding the people with the same question. So this is what the schedule then looks like. We have parallel sessions and rooms and it takes as much rooms as there are people providing a session so it's really a simple format it's not rocket science but it really is helpful when you have a group with at least I would say a third of people experienced in the field so that they can at least bring in their own questions and their own thoughts it's not a format only for 100% newcomers to the format. So the sessions then can look like this looks like drinking coffee and it's pretty much about drinking coffee but with a common question. It can be more than three people so this is also a very typical picture from a bar camp people sitting in a circle and discussing a question. It's also okay to to tell people more about your own projects about your own experiences so it's also okay to to have a hands on workshop people and I know this is somewhat tinkering around workshop I don't know what was about. Or even lecturing people so this is one of the great pioneers of open licensing in Germany power climp I think his work on the problems problems with the NC license is has been translated into 20 languages, something like this. But it's also always the idea of people coming together to share their thoughts and their questions. I also brought you some numbers and figures on the OER camp only one slide because it's only the five minute version. So, in the last eight years we had 25 events so far at the beginning before there was the public funding we had like one event per year and it was very, I don't know, in a circle or hard core. So the core of the community event. We could broaden these circles so that people interested in OER but not so much that they would go to Bremen for a three day event are coming together. We had like an at the end about five or six events per year, more than 3200 participants. Those participants were the ones doing 480 sessions so we differentiate between the sessions and the workshops the workshops are formats that we pre plan so like in a normal conference. And we also mix the format of a bar camp with other more traditional formats like workshops. And I think one goal that was really important to me is that we, which always more than 50% newcomers in the last two years even 75% newcomers per event. And that's also a crucial number when it comes to how many old timers you need an event to continue the growth of a community. There are some spin offs so we learned that many people were at least as much interested in the format of learning because it's a format of open education. And so we provided all the materials, the resources that we developed for organizing OER camps as openly licensed resources. We published a book and had trainings on how to OER camps so when the funding now will finish. We hope that there will be some water sustainability because people learned how to organize their own OER camps or their own bar camps. I skipped the times of COVID-19 because it's not too exciting that we did online bar camps or online events and webinar series. This is one last chart I want to present you because it's really typical for OER camps. This was the question we had when we evaluated the bar camp participants, and we asked them which field of education would you assign yourself to. And you see that it's really, really a broad distribution so it's approximately around the same size for school, higher education and further education. And it's important to know that these people were visiting the same event so it was not an OER camp for school or not an OER camp for higher education. But these people were coming together in the same events and sharing across the sector of borders. And of course they could have their sessions and they could say I have a session specific on, I don't know, early childhood education resources. But I think it was an asset that they were sharing their experiences over these more traditional borders. Yes, should ring in a minute for now. I like the idea of Siegfried being like like our anchor woman. So I'll give back to Siegfried. What is Jan again? Jan, yes. Sorry, I muted, I was muted twice. And I hope you can hear me. I'm missing now a picture which I implemented on the slide by maybe it wasn't an earlier version. And so why I'm speaking here about policymaking is that we had the OER policy, the OER world map implemented in OER info and policymaking is a quite complex topic and it's connected to control of systems. And I think it can be understood or it has similarities with the steering process. And I think that the OER world map has been implemented into the OER info project was another quite typical and characteristic thing of the OER info project. There was an approach of thinking about how all these activities could be quantified and summarized in order to support political goal making and seeing and go control. There's a monitoring aspect connected, which comes in with the OER world map. And I think that the interesting thing is that we, by the way, published two major summaries of the work in OER info and about the data we collected for OER info, which are summarized and available. I will post the links just in a moment. These publications have been named OER Atlas and we planned actually to do a third one at the end of the project. And currently I'm not sure if we will manage to do it, but certainly this would be quite interesting to provide these quantifiable summary of the OER info project. But maybe just some lessons learned from our experiences working with the OER world map as a part of a central infrastructure. And I think that our initial idea that data collection would happen by itself turned out to be untrue. And what we could see with OER info is that it works pretty well if you have somebody who takes care of it. So, especially in the first phase of the project, we had a dedicated editor who just concentrated on the OER info and data collection that worked pretty well at that time. And another lesson learned is that it's working quite well if you are connecting it to important events like the publication of the Atlas or like conference. So, there is actually, it's working if you take care of it and it's not expensive. So, I mean, since the technical platform is there already, you just need to find somebody who takes care of data collection. And I think this is an important insight, especially for future policy and OER policies and initiatives, since they should collect and have these data we collected anyway. And so it's just the thing that you decide doing this and that you keep care that maybe you find a student editor or something like it who keeps care of the data collection. So, and another thing was that we, that there was no obligation to use this data and to contribute this data. So, we always had to ask people, could you please send us the data and could you fill it out? Could you enrich it? Which was okay and worked fine in many cases. But I think another thing which could be helpful for implementing a control system and database control system like the OER Worldmap to an OER initiative is to make it obligatory. So that you say I'm having this major project or this program and I expect every project to share the data so that we have a reliable source of information of this. So, yes, I think the last thing, technical things I said are quite evolved and just for people from other parts of the world. And what you can see on OER Info is the OERD A CARTA. And this is actually can be implemented for other countries and states quite easily. So, it's made, we are mainly implementing an iFrame into another website. So, if you're thinking about doing something like this, it's not expensive to use it. And the major task is in collecting the data. And just the last remark, we have the OER recommendation and the recommendation requires or expects countries to monitor the activities. I think that the experience made with the OER Worldmap in Germany are quite encouraging that it's possible to go in this direction and proceed in this direction. And I think that the real potential will come if we find several states using a similar data monitoring system and so that they can compare activities in different countries. And I'm not expecting that this will happen. But my opinion at the moment is that it would be possible if there would be the political will to do it and if there was enough decision and commitment to stick with it, then we could do it. And nevertheless, we only focused on quite easy to find things concerning educational monitoring. So, we concentrated on institutions, on projects, on events, on things which are quite tangible and you can see it and you can point to it. If you're thinking about OER monitoring in a wider sense and if you're thinking about pedagogical effectiveness of open education, what does open education change in the heads of the people? I think that's the thing which is even much harder to measure and to be honest, I currently have no clue how this could be done. But the good thing is that there are still some open questions and with this I'm coming to the end of my short presentation. Is it good? Yes, there is a question in the chat. Robert asks, can these data be of use in the context of the OER recommendation for nations to report about their progress at implementing? Yes, I mentioned this already in the talk. I think it can. So I think it's a good starting point for building up a monitoring system for education and I think it's quite reliable. We have just developed the OER Policy Hub. I'm just posting a link to a video which we produced for a global conference here on the OER Policy Hub. So we are currently trying to starting with policy documents and then linking it to activities which results out of these policy documents. And I definitely think that the OER world map is capable and could contribute to setting up a major monitoring system for open education. But for sure, if we really want this to be successful, we have been funded by ULIT quite fine, but nevertheless we were a small team. So we had maybe around four or five employees working on the map and none of them was full-time working. So you see, if we really want to take care of it and use it as a means to address the OER recommendation, we have to adjust the resources for this, make the team a little bit bigger. And we need committed local initiatives who are willing to take this way, who are willing to support this approach by providing an editor. This is the second question from Igor. He asked, to the best of my knowledge, there is currently no harmonized approach developed to assist countries with reporting on progress in the context of the OER recommendation, correct? Yes, I'm not aware of a harmonized approach concerning this as well. I think it's an interesting thing, if you think about it, that maybe collecting these data and reporting is kind of a scary thing for many of it. And I'm quite skeptical if it can be compared to open education and if it might be too harm to open education as well. If you look at quality making, for instance, in the health system in Germany, everything is being measured down to minutes. You have two minutes for doing this, two minutes for doing that. And I think we should try to avoid this happening for education as well because open education is about creativity, it's about having trust to other people. So we have to take care that our monitoring process and our management process do not, how do you name it, drosseln or take away the air for people to work in the educational sector. And I think that maybe it's a good idea to, if the open education community takes a proactive approach in this way and so that we can provide measuring systems which fit to our own procedures and own processes so that they are not being given by people who do not understand what they do and who do not see the effect which these approaches can have to open education. Okay, with regard to the time I would give Daniel now the opportunity to talk about the transfer strategies. Yes, I muted myself. Okay, I did not forgot it. Yes, I'm going to talk about our transfer activities and first of all my name is Danny Otto as you might know I'm from the learning lab at the University of Newspork Essen. And I was responsible for the transfer to the educational sector of higher education, but I'm today also speaking on behalf of my colleagues. There, who were responsible for vocational education, adult education and school, and I'm going to take a little bit of what we've done during these two years of the second funding period. And this mainly concerns transfer mainly concerns increasing the adoption or slash use of OER and all the educational sectors. And what we did in terms of measures is to organize workshops, give lectures, further trainings, just to teach interested people, for example, school teachers or lectures at higher education, make them aware of OER teaching them to use OER giving information of what OER is and I'm just trying to briefly reflect on some of the main impressions outcomes we have gained during these two years. And first of all, for the first, for the first funding period, there was a little bit of a problem that we did not really evaluate what we were doing. So that was not kind of an objective evaluation with the leading research questions. So interviewing, for example, people who joined the workshop and for the second period wanted to do that a little bit differently. And we designed a survey and distributed it at all the measures we made, for example, and workshops we gave and lectures we gave. And we were mainly interested in learning more about the motivation about our participants. And we use that by applying a concept of attitudes, so investigating, serving the attitudes of the people. So learning more about their motivation for joining the events. And I don't know if I can go to the next slide myself or somebody because I'm here with my... Or Zigi, can you go maybe to the next slide? I don't know why this doesn't work. I cannot. Okay, thank you. Perfect. Yes. So we surveyed all the participants of our measures. Of course, not everyone participated, but at least 207 persons completed the survey we distributed. And as I said, we use the concept of attitudes. And this is mainly just briefly consists of three parts. So first of all, behavioral components. So just knowing about the intention people have to use OER, a cognitive component. This is more about the knowledge and beliefs that the people have and an effective component, which is more about the feelings and emotions. They connect with the topic of OER. And of course, we could observe that we could state that the people who came to the workshops to third indicate concrete intention and behavior to use OER in their educational sectors. So we surveyed all educational sectors. So this was really the socially high on the approval of the people. Interestingly, we can see that for the cognitive components, so the knowledge and beliefs about OER. So meaning, for example, if OER facilitates innovative teaching, for example, if OER increases the use of OER in education, if it facilitates legal issues, for example. This was rather moderate and also the level about OER, so they have to self assess their level, was also quite moderate. What we can state, however, is that especially the effective components, so the feelings and emotions they had for the topic of OER. So that can be mainly described as, for example, the urgency of facilitating cooperation and sharing of material, the need to cooperate with others. These were rated really as high by the participants. So if you could go back to the slide before. That would be excellent. So we could say that the results show that especially the values and ideas behind OER are the main drivers to engage in it and not the concrete beliefs about it. So they say, although it's time consuming to use OER, for example, in my context, I believe that the basic idea behind OER is very good and is desirable. So maybe you can just conclude that very shortly. And what those people would need to increase their engagement in OER is especially incentives on an individual level. So incentives could be, for example, for higher occasion project funding or recognition of their activities, for example, on their teaching record. But also, and I think this was mentioned by Jan before, structural measures like policy, for example, to support those who are willing to lead. Otherwise there won't be any increase in OER adoption. The second problem we observed during the measures we did was the content problem. So we still have a problem that although meanwhile several repositories exist, there's not enough content in it to fat those who are interested in engaging in OER. So for example, if we have workshops, a lot of people are coming and saying, okay, this is really interesting. And the Turkish teacher, can you give me material, for example, for the context of this teaching and honestly, we have to say no, there's not enough content there. And maybe we can spur them to say, okay, more content needs to be provided. So that is really a problem. At the moment, there are a lot of federal initiatives to set up repositories and just fill them with content. I think this would be very important for the next years. And I think also there was a mention before we have the problem that we have that we are federal restates, so we have 16 different states who all have their idea of how to obtain results for OER that also makes it complicated. And this is one, one major step could be a meta search engine. We're working on that in another project. Kind of the Google for OER. I think this would be very good. Yes, I hope because I got several minutes, several messages that I only have one minute left or three minutes left. If there are overall questions or questions concerning transfer, please shoot. Thank you very much. So if there are any questions, please post them in the chat. So I have one question. One thing I know from Germany is that you always use German terminology. But here, you're talking about OER and about open educational resources. Is that the deliberate choice or not. Sorry, I don't know if I got the question right. Can you, can you maybe reformulate. Okay, I know that Germany always uses German terminology for everything. Okay, you mean. And here you use the English term OER open educational resources. Yeah, we also have a translation. Yes, you're sorting it. But in this case we use the term OER and I don't know if that was to our advantage because maybe it would have been better if we have used or established German translation. And did not. And in my personal opinion, I would say, and that's a bit problematic because these three. Yeah, OER it is not really. It is not really, I don't know intuitive and if you talk to people about OER they first of all say okay what's that. I don't understand it and yeah. If we do it again maybe we could think of German translation would be better in my opinion but I don't know my colleagues. I think it's the same way. I see interesting comments from Jörgen. So, we ran out of time. So, I would like to thank the panelists for this interesting overview of the German activities. I think it was interesting to see and I do hope that you find the funding to continue all your info, because I think it's a really important for the future of open educational research in Germany to continue your activities. So thank you very much. And Igor you can stop the recording.