Welcome to this course on data structures and algorithms. Last time we had seen some basic operations on graphs and written a C plus plus program. At that time, we had observed that in real life there could be problems particularly because of error in inputting data. Today, we shall see how to address some of those errors and in the process how to write real life programs. To recapitulate, we had used a data structure which was actually a vector list and there was a list associated with every node. We had agreed that the first element of each vector list will be the starting node and then subsequent elements of that vector list will be the destination nodes from each of these nodes. So, this way we will be able to represent all edges which emanate from node 1, node 2, node 3, etcetera, etcetera. We had also written functions to create graph nodes, to add an edge, to remove an edge and also two functions to print outgoing and incoming edges. We had said at that time that there could be problems in real life such that we may try to add an edge which already exists or we might add an edge where either the source node or destination node is not present in the graph. Exactly similar problems could occur during the removal operation. We shall look at two of the problems today and we will modify our computer program to address these. The one problem that we are addressing is when somebody is trying to add an edge which already exists. Other problem is its inverse we are trying to remove an edge which is not present at all. So, let us look at how we will have to modify the functions. The class graph remains as it is with a important difference. You remember that add edge and remove edge where functions which were declared as void function. We now change these to bool function. Why? Because each of these functions we will have to now return true or false depending upon whether they succeed in doing the normal operation or not and why would they not succeed? Well it is because of the real life problem that we just mentioned. For example, if we want to try and add edge which already exists then add edge function will fail and it will return false if we declare it as bool. So, this is the modification that we do and of course, the functions themselves will have to be rewritten appropriately. So, let us look at the add edge function. We begin with the assumption that the vector destination is not found. So, we set false to the bool found destination. Now, we have the same iteration that we had seen earlier using the iterator vector iterator. Again for each iteration we start with the vector iterator dot begin for that particular vector list and now we will look at all the elements of that vector. First we see the source. If the iterator is equal to source, we have found the source node otherwise we will have to go to the next list. Now, if we have found the source this is where we have to write code to add the edge. Let us see how that code will look different from our earlier code. First we increment the list iterator pointing to the second element in the list which represents the first edge. We set up a for loop till the end of this particular list incrementing the list iterator one every time. All that we check is whether we find a destination. If we find a destination that means the edge exists. If edge exists we set found destination to be true and break because we cannot add the edge if it already exists. So, we get out. If not we continue this entire iteration. If we do not find throughout this test whether the destination node exists already then the found destination is set to check whether it is false. Please remember we may come out of this loop either because the normal completion of the loop or because this break. So, we have to test whether the found destination is true or false. If it is false then we know that no edge exists and we have to add a edge. This is where we use that simple function pushback which was there in the earlier program also and we return true that we have added the edge. Otherwise we simply return false saying that edge already exists. So, please know that for a real life problem where somebody is trying to add an edge which already exists. Instead of this simple statement that we had used in our last program we have to write so much of code. This is the travesty of fate in real life you have to perpetually check for errors that may happen and write lot of code to ensure that those errors do not cause any malfunctioning of the program. Exactly the same thing will have to be done in the remove edge. So, we start with a bull set false to the found destination as before. Again we iterate through the vector list and for each list we identify whether we find the source. When the source node is found now please note what we have to do we have to remove an edge. So, if the source node is found we must locate the edge to be removed in that particular list. That is where we will write code to remove the edge. Let us look at that code. You will recall the last time the only statement we had was vector iterator dot remove destination and that is because we presume that when we find the destination that is to be removed and we were sure that we will find the destination. But now we are not so sure. So, we will iterate and when we iterate through this particular list if there is a destination we have found the edge from source to destination. We set found destination to be true and remove the edge and return true. But if found destination is false that means somebody was trying to remove an edge which did not exist that is why this function will return false because the edge is not found. The main program will look exactly like what we had earlier except for the fact that whenever we call functions for adding or removing edges these functions are now Boolean function. That means every time we invoke a function to add edge for example, we will have to check whether this returns true or false. If it returns true we say edge added from it to be otherwise we say edge already present. We will have to do it for each of the edges that we add in exactly the same way after printing outgoing and incoming edges when we try to demonstrate removal of edge. For example, the removal also will return a Boolean value and we will have to check that whether if it is true that means the edge has been removed we will say the so otherwise we will say edge not found. We will repeat the same statement for the demonstration on second edge that we had added in our program. In real life the program will not be as long as this because I would typically use an iteration just as I to collect all the vertices for example and to collect each edge from the user through an iteration. Similarly, when I do that the edge addition or edge removal will be called iteratively. The program will not be very long the point being made is that introduction of the Boolean function requires that for adding or removing edge when we want to display an appropriate message we will have to check the Boolean value returned by that function. After this we again have the print outgoing and print incoming as a part of the demonstration and that is all the main program is. As in exercise I would like to suggest that try to modify functions to handle the problems if for example the source or destination does not exist while adding an edge or the source or destination does not exist while removing an edge. More important and this is a harder problem the node being added to the graph already exists right at the beginning you remember we added nodes 1 2 3 4 5 arbitrarily because these were fixed nodes that we have. But assume that you are getting the nodes to be added from the user in an iterative fashion and by mistake a user prescribes the node which already exists. So, the node being added to the graph if it already exists well we must not add it. As I mentioned these are the kind of problems which make the real life code for our program a bit longer, but always much safer and that is the sign of a professional program. In this session we have looked at some of the real life problems that we may face while capturing a graph both for adding edges and removing edges. As an exercise we request you to try and write code which will address more such problems which have been listed in the graph. Thank you.


And we're live now. All right, welcome to ESMACON 2023 and controlling for publication bias challenges and future directions. This session is being live streamed to YouTube. Automatic subtitles should be available shortly after this event, and we work hard to get these mainly verified as soon as possible. If you have any questions for our presenters, you can ask them by the ad ES hackathon Twitter account by commenting on the tweet about this session. If you registered for the conference, you can also comment and chat with other participants on our dedicated Slack channel. We'll try to answer all questions as soon as possible. We would like to take time to draw your attention to our Code of Conduct, which is available on the ESMACON website at esmaconf.org. Now, without further ado, let's get to the topic of our panel. So, typically our goal in systematic reviews and meta-analyses is to provide a comprehensive, unbiased synthesis of all the available evidence that we have in a research field. And I think that most people would agree that this goal is seriously threatened if we have reasons to believe and we often have reasons to believe that some results are systematically missing or underrepresented or maybe even distorted in the published literature. In practice, controlling adequately for such broadly speaking publication biases plural in meta-analyses remains challenging. And our goal was therefore to bring highly experienced field expert together in this panel who would help us shed light on the state of the art in controlling for publication bias and associated problems such as questionable research practices and how we can implement these methods in practice using R. This is why I'm super, super happy to introduce our panel for today. First, Maya Mathur from Stanford University. Due to time zone conflicts, Maya has nicely agreed to pre-record her answers which we will be playing first whenever we introduce a new question. The entire interview with Maya has also been uploaded on a supplementary webpage for this discussion, which can be found at protaglab.org slash as Macon 23 will also post that link. And this webpage also showcases some representative work of the other panelists today. Now to the other panelists who are with us here today which are Pranishek Bartosz from the University of Amsterdam. Nice to have you here. Met page from Monash University, Robbie Van Aert, Tilburg University and Wolfgang Fichtbauer University of Maastricht. Again, we are super happy that you could all join us today. Lastly, I also want to introduce my co-moderator for today which is Eve Blesen from Shakhty Berlin. And Eve, why don't you start us up with the first question? Hello, everyone. Hello, Matthias. As Matthias said, we have five distinguished panelists today who will discuss an unknown number of questions on my of, I could say one of my biggest nightmares publication bias. I would like to open the panel with a very broad question on the different flavors of publication bias. Namely, how is publication bias typically conceptualized? All right, we'll start with Matthias answer. So a typical conception is fairly broad. There's a lot of different flavors of publication bias. So I think the most general definition we might give is something like selection where studies are more likely to be published and make it into the meta-analysis if they favor a desired hypothesis. Now, we can already see that there's kind of a lot of more specific conceptions that are within that umbrella. So first, what do we mean by a study favoring a given hypothesis? Do we mean it has a point estimate that's positive, let's say? That would be, for example, what funnel plots are looking for? Does it mean the study is not only has a positive estimate, but also it's significant? That would be more like certain selection models or maybe does it mean that the result is significant in either direction? So what we mean by favoring a desired hypothesis depends on sort of the context of how we think publication bias operates and this really does have important implications for what methods are going to be unbiased in any given setting. And then a second source of ambiguity with that definition is when we say selection, what do we mean by that? So sort of a traditional conception of publication bias is one where there's sort of a collection of studies that investigators produce and then the publication process, let's say journal editors choose among those results which ones are gonna get published and hence get into the meta analysis. That's a form of very clean kind of missing data where we have a well-defined population of results that are candidates to be published and then among those only some get published depending on their results. I think especially in relatively recent years it's become increasingly clear that's not the only form of selection that happens in real life. There's also potentially p-hacking or other forms of selective reporting that are happening within studies. So we can think of it as investigators will perhaps fit multiple models or let's say analyze different subsets of the data within their study to produce a single, let's say like candidate result to publish or multiple. This is, it sounds similar to publication bias but it actually manifests very differently and it's actually a much harder bias to deal with because we no longer have a sort of clean like missing us at random setting where what's getting published is just a sample of what's available. Rather any given study if there's also p-hacking will be biased even for its own true effect. So it's not just sort of a, it's not just that the published studies are not representative. It's that the individual studies themselves are now also biased. So just to say that the umbrella of publication bias is quite broad and there's sort of advanced multiple definitions within it. Thanks so much. I was just gonna say, great, we can move on to the next question, right? Yeah, excellent answer. But maybe does someone have something to add or footnotes to that really great answer? I would say, I don't know if it's the case with everybody on the panel, but I think at least in medicine traditionally we've tended to use the more broader or encompassing term of reporting bias or dissemination bias to capture all those types of bias that Maya was referring to and tends to restrict publication bias to just the simple example of the publication of a study versus the non-publication of a study. And I think another helpful distinction that sometimes we've been making is what effectively Maya outlined was two main scenarios. One is which information is missing and that can either occur because the whole journal article is not published or because the article is published but some of the outcomes that were measured are just not reported at all versus the other scenario where the results that are reported have been cherry-picked based on running multiple analyses, taking multiple measures and only picking the most favorable results. So at least in my circles, we've sort of tried to encourage the use of the term non-reporting bias but it hasn't really caught on. We haven't been that vocal about it but we kind of use non-reporting bias to refer to the non-publication of studies and the non-reporting of study outcomes and keep that separate to the cases where you do have a result available that might have arisen by a pHacking another in the various activities. Along those lines, I mean, if one wants to make things even more complicated, one could just ask about the representativeness of the studies or the evidence more broadly that you have included in a meta-analysis or systematic review. So it doesn't even have to just pertain to the effect sizes or the statistical significance or which ones are being reported but you can even ask about more broadly are the types of participants in the studies that I have representative for all of the research that has been conducted on a particular topic or even more broadly to the population that I want to generalize my findings to. So I mean, that opens up a whole different kind of worms but one could even just look at publication bias just as one small piece of this much broader issue. I think this is definitely true also because pHacking also has different effects on, for example, the results of the meta-analysis. So you have a lot of different types of pHacking. For example, you have optional stopping or also called data peaking where you repeatedly add participants to your primary study till you find a statistically significant result and this hardly affects the effect size of the study. So this will also not be a problem for the meta-analysis itself, only it only has problem for the standard errors because the standard errors will be affected by this procedure and then it can in turn also bias the meta-analysis to some extent, because usually the inverse of the variance is used as ways for this meta-analysis. So this is one type of pHacking that doesn't affect, mainly does affect the results of the meta-analysis whereas on the other hand, there are also examples like, for example, outcome reporting bias, what definitely affects the result of the meta-analysis because then outcomes are selectively reported, probably depending on whether they are statistically significant or the effect size. And this will definitely have a consequence for the results of the meta-analysis. So therefore I think it's important to draw the station to on the one hand, publication bias and these pHacking behaviors. Yeah, but when you connect back to it when people want to conduct a meta-analysis and correct for the publication bias, I think they kind of also expect to correct for pHacking because they just want to get unbiased estimates right. And I don't think they are really like trying to focus on representing of the population when they do publication bias adjustment because that's more of a qualitative assessment that they can do when they read the articles. But I would say they somehow try like, want to get an estimate that's representative of the effect sizes of the underlying true effect in the literature. So I think it kind of then boils back to publication bias, pHacking, et cetera, together to some degree. Although defining what do we really mean by the true effect? I mean that in itself, right? That would require a very clear conceptualization of who are you talking about under which circumstances, right? So I mean that in itself is super difficult, right? So. Okay. Anything else I know? I mean, it's an incredibly difficult question in a way to give a really succinct definition. Anything else you want to add? Otherwise we would proceed to the next question. One more point that I think is important to emphasize because what I quite often see in the literature is that people interpret methods like, the results of methods like Aga's regression test or the event correlation test or the term of film methods. They interpret these results as evidence for publication bias. If they observe some difference between, for example, the traditional meta-analysis and the, if you apply the term of film method, the estimate corrected for bias. Well, I think it's very important to emphasize that these methods that are all based on the film plot, that they are actually assessing whether there are small study effects. A small study effect implies that there is, that you are studying whether there's a relationship between the effect size of the studies and the precision of the studies in the meta-analysis. And this can be caused by publication bias, but there are also other causes for small study effects. So it's very good, I think, to be aware of the fact that if you observe, let's say a small study effect, that this can be publication bias, but it can also be something else. And then just to make it even more confusing that the issue, I think that most people tend to ignore is that they're using a lot of these methods to detect evidence of some pattern observed in the data suggesting that there might be publication bias, but few of them go the next step to then consider, does the extent and amount of sort of missing studies and missing outcomes actually, is it going to influence my meta-analysis at all? Because there can be some cases where you find some studies that don't report an outcome that they said that they were going to and it looks like they probably didn't report that outcome because it was non-significant. But if it was, say, a trial of 10 participants, including that in your meta-analysis of 10,000 patients, is probably not going to have any impact on the pulled results. So I don't think we do the best job of dealing with that. And so we often see people saying in the literature, oh, I've got a funnel plug that suggests that this meta-analysis is biased, but they don't necessarily consider, well, is it enough to have shifted the meta-analytic effect? So that makes it even harder, unfortunately. Thank you so much for laying the groundwork for our entire discussion. We will come back to some of those themes when some tools useful, which context might be dangerous for these interpretations, whether this skews the entire knowledge we believe to have. But due to time constraints, I would suggest that we move to the next question. Matthias, please go ahead. Yeah, so the next question was a little bit, or is a little bit more practical. So of course, as meta-analysis, we also kind of want to know how to control or adjust for publication bias in R. So the question was, what methods to adjust for publication bias are there in R currently and which approaches can you recommend to novice, experienced meta-analists? And again, Maya has also commented on that, so I'm gonna play that first. Yeah, so there is a whole really great suite of publication bias methods in R. Many of my co-panelists have produced some really excellent ones, and so I'll sort of let them speak about their own packages, but so I'll kind of focus on the ones that I've been involved with. So my group has put out a few, one is called publication bias, and this is an approach that is sort of conceptually similar to a selection model that is considering publication bias that operates on statistical significance. But what it does differently is it's a sensitivity analysis rather than an estimation method. So it asks, how severe would publication bias have to be in order to explain away the results of a meta-analysis? And the objective here was to kind of obviate some of the more challenging assumptions that are common in publication bias methods, such as needing to have a large number of studies, needing to have independent point estimates, needing to have normality. Not all methods assume those, but those have been common methods and common stumbling points when we have meta-analyses that do not fulfill that. We also have a two more recent packages, which I'll touch on a little bit later around the concepts there, but one is called p-hacking. And so this does sensitivity analyses for p-hacking, which has been sort of a really challenging issue. And the second one is called multi, or I guess it's the third one now, is called multi-bias meta. And this one does sensitivity analyses that are conceptually similar to publication bias, the package, but also accommodate the possibility that studies have internal bias. So for example, some of the studies are confounding because they're observational, but then also there's publication bias on top of that. Again, I'll touch on why that's a challenging case later. Thanks again to Maya. Any other approaches? I mean, Maya has mentioned it. Many people here have contributed packages or functions. Any other approaches did you might recommend or want to mention that maybe meta-analyses can use in our in practice or shouldn't use? Well, I think at least to mention trim and fill is a technique even though it has received quite a bit of a bad reputation in recent years. So that's the sort of kind of an imputation technique where you sort of look at the funnel plant or the method looks at the funnel plant and tries to sort of impute the missing studies on one on the two sides. Then you have the pet-peasy methods which take this regression test where you sort of fit a regression line through the funnel plant to see if it's asymmetric and you sort of look at the intercept of that regression model, which sort of extrapolates to a study of infinite sample size where the standard error, for example, is zero. So that's sort of considered then a corrected effect. And then you have selection models and P-curve, P-uniform, while Robbie can say maybe a bit more about that. Those you could sort of also consider a special cases of these selection models. When you try to model the selection process in itself, well, sort of this idea that studies sort of not representative due to some process happening out there in the world. And you can try to capture that process if you have enough data and based on that adjust the effect estimate and try to de-biasing. So I think those are three general types of methods that are quite popular. Yeah, and maybe I can build on top of the answer because as it's clear, it's like so many different methods that one can use to adjust for publication bias. It can often also yield kind of different estimates. And it might be difficult to like pick and decide which one of them you should use at the given scenario and maybe we will get to that later too. But it's also relates to what we are trying to do here in at UWA and we are trying, we build this ROBMA package from my package when we try to combine the different publication bias adjustment methods together with Bayesian model averaging. So we fit all of the, all like we did the different types of models, for example, the selection models, the pet piece and we look how well are the data described by those different models and we combine the models together based on their prior predictive performance. So if the models predict the data well, then we give them more wave and then we base the overall inference on all of the models together. So that kind of allows us to kind of get the inference back together based on how well the data are described by the different publication bias methods because in different settings, different methods might perform better and we might don't know the true underlying conditions for those methods. So we can try to assess it based on the observed data themselves. Maybe one method that is not often really considered a method in itself, but I think that should be mentioned here is if you actually have the, so if you're really focused on this type of bias due to published studies being not representative because of an overabundance of statistically significant findings or positive effects, whatever, I think a method that should always be considered if possible is to compare the effect sizes from the published studies with if you are able to get them some of these unpublished studies from thesis, dissertations, et cetera, right? So, I mean, this is not directly an adjustment for publication bias, although well, if you find a difference there, you could sort of use that to maybe unbiased the published effect sizes, but in any case, I think that is a very straightforward technique that if you have unpublished studies that you can always use. And what I really like about this is that many of these other techniques, well, they are based on sort of an extrapolation, like Pat Peezy, where you're extrapolating to a study of infinite sample size or you try through really fancy statistical machinery, capture, selection processes, right? So, I mean, it's nice that we have these methods, but it's all based on sort of an unobservable process happening out there. And if you have published and unpublished studies, you can compare them and you can see what the difference is and that is much more manifest than while using these fancy statistical methods. And that's me saying this as a statistician, right? So. Yeah, maybe if I quickly jump, you can also say similar thing, I will treat these reports, right? You can also look at the studies that have very high quality of reporting, pre-registration and compare between those studies and the rest in a similar manner, which also ties back that I believe you would have no publication bias because all of the studies would be conducted in such a clear manner, pre-registered, published with clear pre-analysis funded. You wouldn't need to do all of this statistical archeology to recover the missing studies and extrapolation, right? I think another thing about these methods that Wolfgang's talking about, similar type of approaches is if say you're doing an assessment for published studies, even you don't even have to, sometimes you don't even have to look for, sorry, I think you should always try and find a registered report or a protocol or a trial registration for a study, but even some study authors themselves can essentially tell you that they've selectively reported their data because they'll say what outcomes they plan to measure in the method section and then lower the hold, some of them just don't appear at all in the results. So that raises your suspicion, okay, they'll probably select or they'll report it in a way that's not really amenable to meta-analysis, sort of just saying there was no difference between groups, not telling you the direction of the effect or the exact p-value. But the thing I like about those methods too is that I don't know if others have noticed this, but at least by focusing on how systematic reviewers deal with publication or reporting biases, I feel like they're tense, not in all authors, but there's sometimes there's a tendency in some authors to want to not claim evidence of publication bias because that would go against their theory that something works. And so it's somewhat easier to say, or maybe it's not, but it feels like it's somewhat easier to run various statistical methods that account for publication bias, report only the one that says there's no problem or report a final plot that says there's no problem to prove that they have no concerns about reporting bias. Whereas if you've sort of done this story, I guess that sort of person probably is not going to do the heavy detective work of actually identifying evidence of reporting bias, but sometimes if they have done that, then that's kind of right in their face that there's something missing here and maybe that's a bit more objective evidence for them. So who is going to look for publication bias and publication bias test results, right? So that's a nice project right there. Well, actually we did some project like that when we tried to analyze a very big piece of meat analysis and look at the results of publication bias test. And actually I have a meme like this from my class with like publication bias, detecting publication bias, publication bias and publication bias. So to comment on what Natchez said, I've seen that, but I've also seen a quite opposite phenomenon where it's almost, well, almost on vogue to actually make sure that you can demonstrate that there might be a publication bias going on. So I've seen both. I've seen also really kind of strong attempts to if anything suggests that there might be publication bias of all of these different techniques, then okay, that already calls into question the results. So I've seen this go both ways. And I think this is a practical problem, right? Because we have this wide variety just from a practitioner's perspective and it makes it easier. You have so many research degrees of freedoms to sort of start weaponizing these methods that are based on untestable assumptions. And I think this is also a little related to another question that we have, of course. I know it's a tough one, but which of these methods performs best, because this is something that I also get asked a lot. So we have these methods, so which one should I use? Which one is the best? Even maybe you've prepared this question and introduced it properly, but I think it fits you nicely. Yeah, especially because we opened up this discussion, performs best for what? To demonstrate that there is publication bias or to demonstrate that there's no publication bias or to actually find an answer to what I would say, normal meta-analysis that don't use these tools as weapons would like to know. I would suggest that we also get Maya's perspective and we ask her to answer which of these methods typically performs best or might be best suited for which specific context. So which method will perform best is gonna be very contextually dependent, unfortunately. And this is because the methods generally make just different assumptions. And so, ideally if we knew sort of the mechanism of the publication process and we knew sort of statistical characteristics of our meta-analysis, like are the true effects normal? Are the point estimates independent? What's the sample size, things like that? If we knew all of that, then we probably could choose a method relatively easily. Now, the challenge is that in general, especially when it comes to the mechanism of the publication process, we can make it educated guess, but we don't usually know for sure. And so, just as one example, if you think that selection primarily is about whether a study is significant or not, and I think that's quite plausible in many scientific contexts, rather than operating on sort of favoring larger point estimates, regardless of significance. And if you think that publication bias affects all studies regardless of size, then you are in a context where funnel plot methods will probably not work well. Not necessarily not work well, but you're outside the kind of classical case where those methods are appropriate. And so that the situation I described is one where something like a stepwise selection model, so something that assumes that the publication probability increases once a study becomes significant would probably be more appropriate. A student of mine, Maximilian Meyer, led a tutorial paper that kind of flushes out, especially these differences and assumptions between selection models and funnel plots with some sort of applied recommendations for that. And I do actually think that for this reason, there's probably way too much emphasis on funnel plots alone. In that paper, my student found that, essentially funnel plots remain almost the only method that's used in at least medical meta-analyses. I do think psychology has been a bit more sort of forward thinking with methods, but selection models themselves have assumptions. They typically require pretty large sample sizes to work well, and they have distributional assumptions, which if violated might cause the results to not work. So again, it's sort of, I think ideally we would be fitting multiple methods with an awareness of their differences and assumptions and not just blindly fitting multiple methods and comparing them, but fitting multiple methods and then thinking critically about, am I in a context where this method's assumptions makes sense, for example? I think it can also be wise to try to include some methods that are either more statistically conservative or that use fewer assumptions, often at the expense of precision. So an example of that is the sensitivity analysis. I mentioned where what you're losing is the ability to estimate how severe publication bias is. What you're gaining is a reduction in assumptions. And so doing a method like that, in addition to something like a selection model or maybe in some context a funnel plot would help sort of characterize what you can say under fewer assumptions, not no assumptions, but fewer. And I know one of our panelists also will be talking about some cool methods that do sort of Bayesian model averaging over multiple types of models. This is sort of a similar idea of being more like epistemically conservative about what mechanism we're willing to assume for publication bias. And in practice, multiple methods may well disagree. If you want some concrete examples, the paper by Maximilian Meyer is one to check out. I also have a preprint about p-hacking that kind of discusses sort of how badly things could go when we have p-hacking in addition to publication bias and how that can produce differences among methods that can be explained in terms of differences and assumptions. Okay, thank you, Maya. Same question goes to you. What would be best practices in a world where we have so many different tools and so many different specific contexts? First of all, this is a very good question but also very difficult to answer because I think, yeah, what Maya said, it really depends on the characteristics of the meta-analysis. So what I would do, and that's basically what she also suggests is first check which methods you think perform well for the characteristics of your meta-analysis and then apply these methods. And also very important, of course, is to fully report all the results of these methods and not be selective in the reporting of these results. And what you might also do is to just write a preregistration where you already say, okay, I'm going to apply these publication bias methods and then you have already, yeah, at least carefully thought about what methods you are going to apply and then if you're also really going to report these methods, I would say that this is also an important and good thing to do. Okay, I'm not a statistician so I have no idea how to answer this question but what I do want to ask is, are there sort of decision aids for meta-analysis to help them sort of, so is there sort of like, if you could feel out if you answer a couple of questions about the characteristics of your meta-analysis that it would at least funnel down the list of possible methods one could use and ones that would be considered appropriate for their meta-analysis that someone has maybe developed or are developing because I think, yeah, back in 2013, when I know there was a paper that published, sorry, a systematic review of these type of methods found that there was 100 of them. I'm sure there's probably double that now. Or maybe not double, but there's already a lot but if there's a way to help sort of, I guess funnel us in the right direction that would be helpful or maybe the answer is don't use any of them. I'm not sure, I'm sure that's not the answer but I'm curious to know if people have any advice on that. Maybe I can answer the first question. Rather there's some applications you can use to determine different methods. There's paper by Coutret all in cycle, advances in methods in psychological science that has some application like this where you can specify what's the expected effect size, what's the expected heterogeneity, what's the degree of publication by us, what's the number of studies and whether you are interested in bias, written in square error, et cetera, a slightly improved version of this a bit more simulation is by Tonkin Reed published in RSM in 2021, 2022. And they also updated this application to do that. The problem that I see with this kind of is that you need to put in the expected heterogeneity, the expected effect size and the expected degree of publication bias. And I don't really think you are able to do that before you analyze the data. And when you analyze the data with some of the methods, then all of these estimates are going to be biased based on characteristics of those methods. So you run some catch 22 problem when you are making your specification on your best estimates. And maybe you can do a few rounds but you have no idea where you will end up. So I don't really think this is, I think it's a good attempt but I'm not sure how feasible it is to actually perform this to get the actually best properties of those methods. And of course you can say I'm biased because I have my own methods in play that I try to develop. They try to deal with this based on the Bayesian model leveraging as Maya mentioned before. So I personally think this is probably like one of the better ways how to deal with this. But that's of course my opinion. So you can disagree with that. We have any other recommendations for us practitioners who feel a little bit lost in the woods? Yeah, really difficult to say. Matt, I was going to say I am a statistician and I have no idea how to answer that question. So it's just really tricky. It's very difficult to answer. So if we're going with our own pet methods I personally do like selection models. I think they're really nice. They have some sort of nice statistical properties. I think they fit quite well with what we are often worried about. And it's one of the methods that is also considered of course in the model averaging that Funtichek mentioned. So I think that's one of the methods that is part of this whole package, right? So the only difficulty with selection models is that there are also again many of them. There isn't just the selection model. People often consider a relatively simple one. There's sort of three parameter selection model where you just make a distinction between the likelihood of selection for studies where the P value is below 0.05 or above. But you can have many other types of selection models with more smoother curves or many more steps in this step function. And so then the difficulty again becomes well, which one do you choose? Or which one do you apply? Or do you apply all of them? And what do you do with this multitude of different results? Well, we actually specify a few of them in the package itself because you're right that there's multiplets different weight functions you can do. And actually like in practice, what we found that what usually plays some role is also direction of the effects that effects in the supportive directions are also more likely to be published. So basically if you have one set of P values then on P value of 0.5, that's additional step. And I think the current implementation has six different weight functions that differentiate between significant, marginally significant, the correct direction or not and different combinations of those assumptions. But yeah, I also like selection models. I like the statistical model behind them. And one thing that Maya mentioned was that usually it's a lot of studies to fit them. I don't really think it's that true if you go to the, if use Bayesian statistics because you can do some restrictions on the parameters which I think are quite sensible and that help with convergence a lot. For example, that if you have statistically significant, statistically non-significant studies, there's can be ordinal restrictions of the publication properties. It's quite unlikely that non-significant studies are gonna be much likely to be published than statistically significant studies. And these few restrictions that you can implement on the parameter space makes those models much easier to converge as well as some very small regularizations that you can input with the parameter distributions on the parameters. So there are quite big gains you can get from Bayesian statistics, especially in those cases when there is very little data by regularizing some of the parameters or some of the models which helps with the convergence and makes the behavior much more stable. And I think we also show this in the simulation studies that we run that the root mean square error of the methods that we show is usually one of the smallest. I mean, this can be interpreted as a playful selection models, quite broadly speaking. I just want to ask, so we also talked, of course, about small study methods, funnel plot-based methods would be the takeaway that these are done too often that we shouldn't use them as regularly as they're still quite often done at least in the literature that I know or that I have some kind of dim expertise in or do we really not know? It again depends on the characteristics of the meta analysis. So for example, I've also seen situations where this pet piece method performs quite well. So therefore I would definitely not say, do not use any of these methods because this pet piece method and also I think also the agus regression test if you have a large number of studies in your meta analysis then also this test then for small study effects is actually also I think a good way to go because generally has only if you have less than 10 studies then it is especially problematic with respect to statistical power. But if you have a large number of studies then this should be okay. Of course, depending on the true amount of publication by small study effects. So this is all hard to say but I would definitely not say that do not use these kind of methods. And these methods generalize quite straightforwardly to more complicated models. So while conceptually you can generalize the selection model to a multilevel multivariate meta analysis in practice that is extremely difficult actually how to implement this but these sort of yeah, pet to PZ regression type techniques you have to think about what kind of selection are you trying to pick up here? Are you thinking about missing studies or missing effect sizes within studies? So you have to start making sort of distinctions between that but they do generalize quite easily to this to these more complex models and that is one of their great strengths. Yeah, that's definitely an issue for selection models and not only how to implement it but also how to estimate them if they are implemented because for example, we tried implementing them and we have a implementation for multilevel selection models but we are unable to run it in reasonable human time to estimate any studies with more than five estimates per cluster. So that's another limitations in these cases. But yes, I think pet piece is quite good in many cases and we also included actually in the Bayesian model averaging ensemble because those methods generalized very well and they have usually small bias although very high variance of the estimators and we also run some simulations with only some Bayesian restrictions on the pet piece regressions. For example, restricting the coefficients to be only positive on the regression of standard errors which also seems to help a lot with the variance of the estimators. So yeah, there's also some improvements that can be made just in applying those methods even if you don't wanna go straight to the full Bayesian model averaging and combining all of those methods together. Wonderful, thank you so much. I'm afraid we're almost out of time. Thank you so much for the wonderful discussion. During the discussion I was reminded of this saying about missing data. The best way to deal with missing data is to have none. And maybe that's also true for publication bias. It is a difficult topic but I think maybe it was a, at least I found it very illuminating. And yeah, so again, that's it for this session. We hope you enjoyed it as much as we did. Thanks to all of our panelists. Again, we've also posted a link to the supplement with a few more references on some of the work and see you at the next session. Thank you so much. Thank you.


What's going on guys, it's your boy DPJ here today with another Destiny video and in today's video I want to pick up on something I spotted from the Taken King Alun's gameplay trailer dropped yesterday by Bungie. If you missed the trailer I will link it in that video description. Now I would have uploaded this last night but sub boxes were broken so I decided to leave it. I also have a video coming later talking about all interesting parts from this trailer so stay tuned for that. Now this at the moment is all speculation and until the Taken King comes out we won't know for sure. Now at 33 seconds into this latest trailer we see many epic things from the Warlock leg armor, that salt rifle the Warlock's holding, the hunter class item and helmet. They are all amazing points but they ain't the point of this video. What is is the weapon the Hunter is firing. Now I spotted this straight away and I believe this weapon looks identical to the Thunderlord except it seems to be oozing void energy and shooting void shots. Now we all know the Thunderlord is an Ark weapon but we also know that it's one of the year one exotic weapons that's getting updated. Now from one of the first Taken King reveal trailers we also saw another weapon identical to the Thunderlord but it was a solar weapon and I will show you guys that on screen now. Now I find it highly unlikely that Fungi will have two holding weapons based on the Thunderlord which offer different burn types. What I think we are seeing here is an updated Thunderlord. Now we know changes will be made to this upgraded year one exotic weapons and I think the change through the Thunderlord will give it or give us the ability to change its damage out for element type. I truly believe this to be the case and if I'm right the Thunderlord I'm sure will accompany everyone in their daily destiny business. I'm certain it will become the new go-to weapon just like the Galahorn was for that year one but who knows maybe I am completely wrong and Fungi have something else in store for us with this Thunderlord and its update. Maybe when we upgrade it we have to choose a damage type to upgrade to. Only time will tell. But guys tell me what you think down below in that comment section did you spot this from the trailer yesterday let me know. Thanks for stopping by as always and peace out until next time


yourself. This is the fruits of lots and lots of labor of folks in it. It's the kind of exciting. Hi folks, so hopefully you enjoyed those two presentations that we had pre-recorded. Happy to take maybe a quick question or so or we can take a look in the chat but just wanted to let you know that in the Unconference Room there will be a table for an hour in Floor 3. So if you're interested in looking at that you can go to the Unconference Building 6 Floor 3. We'll have a glam and cultural heritage chat room there. So if there are any questions I'm happy to field any or you can put them in the chat. So again Unconference Building 6 Floor 3 Table 1a. Hi folks, so hopefully you're seeing me after the presentation. We don't have much time before the next presentation but if folks want to keep talking we will have a table at the Unconference for the next hour. So that's Floor 3 in Building 6. So if you go to Building 6 Floor 3 for the next hour I encourage you to stay here and watch the rest of the presentations but if you want to go to Floor 3 for the next hour from 1600 to 1700 UTC I'll try to be there with anyone else who wants to talk about glam or other issues related to glam and cultural heritage. So if there are any questions in the chat or anything else we had one in the Etherpad. So feel free to go to Etherpad. The notes are all there and the slides are all there for the presentations and Kelly Doyle has put in some answers to the questions there. All right so we'll see you in Unconference Building 6 Table Floor 3 if you want to talk more. All right take care.


We have to start today the annual report. Learning outcome, after going through annual report of a listed company, remember listed company, students will be able to understand different contents of it. Besides financial statements, lot of other information for these stakeholders is available. So here you will see it's not only those four statements, but a lot of material attached to it you will learn. The organization chart will enable them to know the directors and the key members of the team. The people involved at the top level, there is an organization chart available. Notes to the accounts will help you to details of accounting policy adopted by the company. Number of accounting policies company use to follow. So in notes to the accounts you will get all those details of policies. Sometimes 20 pages of it. The annual report objective of entities is mandatory. The annual report is basically a mandatory thing. I mean, every company must produce at the end of the year the annual report. Previously they were preparing the what you call half yearly reports, interim reports, but this is mandatory that you have to, there is no choice about it, to publish at the end of the financial year. The shareholders want to meet the members board of directors in the annual general meeting because this report is discussed, is reviewed in the annual general meeting going from all other sites where it comes to the annual general meeting. And there the shareholders will see this report, learn this report about it that and then they want to see what the directors, chairman, other shareholders feel about the company while going through the reports because it is required that this report is sent to every member of the company, every shareholders. So it is expected from them that they should come prepared and if they have some objection somewhere any question to ask they can ask in this in the annual general meeting. The stakeholders is customers, suppliers, bankers, regulators, other decision makers like financial position and performance of the entity. Whosoever want to deal with the company, this report will be enough to take a decision. So all these stakeholders are watching what you are producing at the end of the year, the whole year performance, your assets, liabilities, everything is there. Essential of an annual report, the four financial statements that is the income statement and the statement of financial position, cash flow statement, change in equity statements, these are four major statements. Then attached to it, the notes to the accounts, a lot of schedules, etc. Pattern of shareholding, that is also very important thing that you got number of shareholders to the patron. How many institutional shareholders you have, how many the proprietors, the directors, shareholders, individuals are very few. Let me tell you individually, hardly 10 to 20 percent. The remaining 80 percent either is institutional investors or the directors themselves. Then there is a chairman message, a chairman report that is also very, very interesting thing because every shareholder want to see what our chairman is looking into. Because they want to listen from the chairman, how he feel about the company, what is his opinion about the company, the report we are reading. So that is also very important. Then the director's report, that is also there. And then 5 to 10 years financials. Consolidate, simple form, 5 to 10 years, your accounts, they also present in this report. There is only one page, but in this column in reform, here 1, 2, 3, like this 5 to 10 years reports are shown in the accounts. Key factors, that is key ratios, key numbers come in it. That is why it is known that what is the progression of the company. Its graph is going up or down. Then future objective statement. This is also what you want to do in the coming days. Basically the chairman has to tell this in his note that where we want to see the company in the coming days. Environmental statement. You are not polluting the work for the environment. Sustainability statement and value-added statement, employment statement, foreign consistent, these are various statements. And they are important, let me tell you. For example, this value-added statement in Pakistan, I started in 1990. That first time I produced this statement in the final account. So people appreciated it. And now many companies in Pakistan also, they are producing this statement. So the statement, I will go through this statement later. So then foreign currency statement means if you are dealing in foreign currency in the sense that you are importing, exporting. So you have to produce a statement.  So similarly employment statement, what you are doing for the employees, that is very, very essential nowadays. Then the stakeholder rely on external audit reports. There is internal audit as well. But mainly people want to see an independent auditor who coming outside the company and what he reports about these accounts. And they have an impression that the audit reports everything and they make decision accordingly. Because audit reports in the sense that it gives a true and fair view. Meaning that there is nothing hidden in this. The real thing is shown in this. So that is the important thing. Now financial statements, whatever these four statements are, in which financial position is also there, statement of other companies and cash flow statement notes. These are all comparative points. That is, you have to show both last year and this year. So that you immediately know that you are going through the last year. Whether it is your progression or not. Profit is increasing, sales are increasing or not.  In other reports, as I just said, value to statement, chairman, message director, I will discuss them a little bit. Financial is of five years audit report for the members. Compliance with code of corporate governor. Directors have to give this report that we have complied with the rules of corporate governor. And we have to tell them point wise. There are about two and a half pages in which there are 10, 20, 25, 25 points. We will tell about every point that we have done this, this, this, this. Then something will be written in it. Then pattern of share holdings, category of the shareholders, as I said earlier. In this we have total number of shares and in that how many percent are of directors, how many percent of institutions, how many percent of bankers, how many bankers, how many insurance companies and then how many individuals. This has to be shown as well. In this we give one more thing in the report, capacity. What was the capacity of your company? How many tons were to be made or how many, let's say, if you want to make something, then they have to give the capacity. They call it installed capacity. How much production could this plant give you? And how much did you achieve? This has to be told as well. What was the capacity of the plant and how much did you achieve? If you have achieved less, then you will have to tell the reason as well. In this whole report you will be available and I will show you the report. Thank you very much.


Hollywood Radio Theater In Thunder on the Hill Ladies and gentlemen, your producer, Mr. Irving Cummings Greetings from Hollywood ladies and gentlemen Tonight we welcome back from her recent trip abroad One of those rare combinations of Hollywood A versatile actress, a beautiful woman, a glamorous personality Claudette Colbert We will present Ms. Colbert in her original role In an exciting mystery drama Thunder on the Hill Produced by Universal International Pictures And also from that studio We have invited one of their newest, brightest stars Barbara Rush Now, Thunder on the Hill Darring Claudette Colbert at Sister Mary And Barbara Rush at Valerie Cobb Now the rain had beat down since the play We launched in the little valley The dread warning spread across the southern countryside Blood For the villagers, there was only one refuge The convent and hospital of our Lady of Reeds As Mother Superior, I issued orders To open our doors to all who asked for shelter Now, Stella, may I ask what you're doing here? There have been casualties, Sister Mary We'll leave this recreation room with an annex to the hospital I don't remember giving you permission to leave the ward You didn't But when everything's in a turmoil, I forget formalities There is no turmoil here unless we create it now Bring the car here, Willie Yes, Miss, Sister Mary Phillips asked my permission to leave the ward Well, Dr. Jeffries, I didn't know I planned to use this room for a children's dormitory You see, I walked them away from the contagion ward And I was preparing this for Dr. Jeffries' wife Well, we'll find some other beds Your wife, Doctor, she's no better? No, I'm afraid not Mr. Harmer is bringing her up from the village She'll be here any moment I'll have a bed for that in my office Please don't be offended It's just that it's my duty to see that things are done in the right way Oh, of course The right ways, Sister Mary, are your way Please return to the ward, Nurse Don't send anybody here with the will of her own Maybe if I were a thief being huffed with like Willie He would approve of me You're done on me, Yard I'll teach you calling me names Willie, Willie, no! No! Leave when you're safe Get down, Willie, put it down at once Oh, Willie My hands be clean, Sister They have no sin on them Go to the kitchen Sister Josephine needs help Yes, Sister Oh, we're all a little jumpy tonight I'm sure Nurse Lyft didn't mean what she said I meant every word and more What's happened here? What is it? I'm fed up, Reverend Mother Sister Mary, it would do her so good to be wrong once in a while If you weren't so positive about everything Perhaps your sister would still be alive Oh, she doesn't mean it, Sister Doctor, Mr. Harmer and your wife have just arrived Oh, thank you Excuse me Mr. Mary, Willie's proved once more that he can't be trusted He's a good workman, but as soon as conditions are back to normal He must be sent away Yes, Reverend Mother Ah, you're preparing this room for the children Yes, they'll be safer here, and they won't disturb the patient To be sure, how well you plan everything Oh, no Nurse Phillips is furious with me In fits of anger, Sister, we all say things we don't really mean But how she found out about your sister, I really... Phillips must have resented me for a long time We can't expect to be loved by everyone We've got to accept the fact that we're less than angels And to make the best of ourselves Reverend Mother, have I made the best of myself? Of course you have And you came to us later in life than most of the others Sometimes I think it was too late It's never too late to serve guard on the aiding Your record is a glowing example of it From novice to matron of our hospital in eight short years Oh, if I can only be sure that in changing my life I have succeeded in changing myself You mustn't hold yourself responsible for what happened to your sister I should have left her alone If she wanted that man, I had no right to interfere But he was all together and I'm a good person for her It was your duty to protect her I couldn't have taken the thought that it was I who... Your side is such a horrible sin Oh, Mother, no matter how I've tried I find no comfort, no pain It will come to you, Sister, in its own way, in its own time Now then, how many do we have here? So far, 72 adults and 25 children In addition to all our regulars Sister, you're wanted in the Great Room Is it urgent? A man and two women have arrived They want to borrow some petrol or a car Thank you, I'll see them in a moment These people, do you know them? No, they're another But I heard some of the villagers talking They said to bring her to a convent is blaspheming her A young woman The other woman and the man, I'm not certain But they appear to be in charge of it As if she were from the police I see Well, Sister Mary will take care of them Take off your coat, Miss It's so worried about my health, Sister Spirit I only want you to be comfortable We'll be back on the road soon Good evening, I brought you a tray It's just soup, but it's hot and nourishing Oh, I was told there were three of you Sergeant Melling will be back in a moment Here's your soup Sit down, won't you? Thanks, but I don't want him soup Please, it will warn you I'm not cold Then let me have your coat Perhaps you don't care about getting ill, but we do Why should you? Oh, for one thing, our wards are overcrowded Oh, I want to get out of here What's keeping him so long? Why are you in such a hurry? Oh, but I'm not They are Well, isn't there anything I can do? Perhaps if you'd explain Explain what? You know who I am You're only being kind because it's your duty Well, that's true And if I can't be of any help It only means I've failed in my work A very clever system Appeal to the thinnest conscience The trick that never failed Well, with me it does, I have no conscience Well, and I'll have no tricks left Except to ask you once more Take off your coat All right, I'll take it off I don't want you to be a failure on my account You'd make me even a greater success If you took your soup Now, will you stop? Really, me Is there a time when I... Yes? Well, well, they've all been evacuated, I'm sure But if there's anything we... Hello? Hello? Hello? Sister, I was just talking to the chief engineer The dyke, I was cut off Will you ring through, please? What? Oh, oh, thank you Oh, get ready, Pierce They're giving us petrol We'll be in a nudge in an hour I'm afraid you can't Oh, I'm not, ma'am What is it, Sister Mary? The dyke has been abandoned, Reverend Mother They expect it to collapse at any moment Oh, then if I may use your phone I'll have nudge senders to launch I'm afraid you can't do that either The wires are down There's no way to communicate with the outside But we must be in a nudge before tomorrow That's impossible, sergeant Even if the rain stops It will be at least two or three days before... Two or three days? I couldn't stand it Sister Mary, you'd better put the entire staff on emergency duty They're bringing in more casualties from... Valerie Good evening, Dr. Daphne How do you get here? Valerie Valerie Cohn That's right, sister Valerie Cohn is the murderer Valerie Cohn, she was on her way to nudge Prison to be hanged in the morning But now they'll just have to wait Oh, can't you see it in the newspapers? Postpone because of rain Just like a cricket net All right, miss Andy, now take her to my office Put her on the couch Don't leave her alone, sis Not for a second Among those now under our roof Was Dr. Jeffrey's wife, Isabelle She'd been ailing for some weeks A nervous disturbance The doctor had set up makeshift quarters For her in his office As for their meals They would eat with Mr. Harmer The pharmacist in his dispenser Mr. Daphne, Isabelle Breakfast is getting cold Where's Mr. Harmer? Oh, he'll be back in a moment We're taking some drugs to the ward Have you seen her again, Edward? Valerie? No Not since last night She's innocent I swear she is Really, your voice and the head And state be sensible Good morning Good morning, Harmer Well, it's good to see you up and about Mrs. Daphne Thank you Won't you sit down and join us? Oh, I had breakfast an hour ago But I shall refuse another cup of tea Edward, please let me see her It's impossible She's under arrest Oh, sister Mary Don't get up, doctor Good morning, Mrs. Daphne I've brought the ward charts But they can wait till you've had breakfast Sister, please Miss Kahn, where is she? Where are they keeping her? She was moved to my quarters this morning We decided it would be better If she were completely separated from the others Yes, the villagers are so set against her It's a pity But after all, she did kill her own brother Wicked, though he was Did you know Jason Kahn, Mr. Harmer? Only slightly Dr. Jeffleys is the one who really knew him What actually was the matter with him?  She gave her brother three taps A fatal dose Would have killed the whore I'll wait until she wasn't glad to see you here, doctor Mr. Harmer's redness Why not? I was a key witness to the crown You did everything you could to save her You brought up all the extenuating circumstances If only Willie hadn't ever had Willie? Our Willie? He used to help out of the Kahn's house Pandemon, just as he is here What did he over here? Well, Jason was in agonizing pain When I got there, I found Valerie terribly distraught She had no idea what you were saying He kept repeating I wish he were dead Oh, Edward I'm sorry dear Yes, we were all shocked by it It was very tragic of her You mean the court took Willie's evidence That's testing money? No, not Willie's, mine I had to admit, he was telling the truth Oh, oh Well, the charge I'll leave them here with you Oh, oh yes Thank you, doctor Oh, two sister, come in I, uh, I haven't had a chance yet to thank you For allowing us to use your rooms here There's no need to thank me, sergeant I'll only be a moment There's some papers in my desk Who's playing? Miss Kahn's, ma'am Since you have a piano Well, I hope you don't mind Oh, no, of course not She plays beautifully Tell me, has she been a difficult prisoner? Oh, not more than most, ma'am Of course, these last days are the worst After the home secretary turned down the appeal May I see her? Well, these rooms are yours, sister I have no right to stop you Thank you Please don't stop What do you want? I wanted to see if there's anything I can do To make you comfortable It's cold here, isn't it? I'll have Willie bring some logs for the fireplace How is Willie? Oh, I don't know Unpredictable as always And just to strike him He didn't look at me once during the trial Just a bite in his face But I'm not angry with him He only repeated what he heard Miss Kahn, as that music you were playing a moment ago I knew so little about music Was it Debussy?  Jason Kahn Your brother? Yes The one I killed You don't like it, do you?  Jason's music has great merit I will be remembered That's the only thing I still believe in Miss Kahn, is there nothing I can do to help you?   Why do you think I'm not guilty? Tell me Perhaps because I know how it feels to be guilty Well, suppose you do believe in my innocence What does that mean to me? Can you do something about it? I can pray for you More words Sister, if you really want me to believe you Help me to live through these last few days Oh, I will Bring me the human being who really cares about me His name is Sidney Kenham Is he here?   You have that faraway look in your eyes Is it still that unfortunate girl? There isn't last night From the first moment I saw her Something passed between us Sister Josephine, I know She's no more guilty of murder than... Than you are But you can't be such a... I am, I am certain It's as if I... It's as if I were driven from within to do something This great flood It must have been God's will to surround our house with water To let no one in and no one out God's will it is If I could only go to Norwich Norwich? I'd do anything if I could get there tonight Well, that's the first foolish thing I've heard you say in eight years Sister, I need your help My help? I want to know everything about that trial But how would I know? Those old newspapers you're always saving There's no way to throw out newspapers Why? I use them to line the shelves Oh, I know That's why they must be here somewhere Please, won't you find every newspaper you can from last January When the trial is on? Sister Mary, is this why? I must know what more about it, please Come then, we'll find the papers sometime Here's another one, Sister Listen, listen to this Prosecution If no one else was in the house Who then but you could have given your brother the three tablets? Miss Khan, I told you a hundred times I took one tablet from the fire I did not substitute the aspirin for the two missing tablets I gave him one tablet, just one tablet And here, down here Mr. Khan has kept the money Now, if we can find the other Who's there? It's me, Willy Come in You're doing this well I hear you talked it to Mary So I come to fetch her You'd be wishing to go to Norwich And so you will Got it Tonight? Willy found a way Nobody could but Willy Be your ready sister Yes, Willy I'm ready Come We go to Norwich now Make a friend and you make an ally There's a thought for you to keep in mind As many another American has Take the famous all-negro basketball team The Harlem Globetrotters As unofficial ambassadors In one year, they played ball Before more than a million people on four continents In Rio de Janeiro, they entertained crowds Of from 30,000 to 50,000 During one summer, they toured Europe and Africa Chalking up another 600,000 fans In 1952, they celebrated Their 25th anniversary as a team By circling the globe Yes, sir, the team Organized by Abe Saperstein Really gets around And their exhibitions have been more Than just a demonstration of American basketball They've been a lot more The team is a living example Of American fair play and sportsmanship In and out of uniform Abe Saperstein now carries a letter Which reads in part The Harlem Globetrotters have proved themselves Ambassadors of goodwill On any future tours, please call on us For any help we can give And the letter is signed by The United States State Department In being ambassadors of fair play The Harlem Globetrotters proved That by helping others, you help your country And now our producer, Mr. Cummings Act two of Thunder on the Hill starring Claudette Colbert as sister Mary And Barbara Rush as Valerie Kahn Willie had found a rowboat This was the means by which he takes Mr. Mary to knowledge It has stopped raining now The flood waters were heavy with mist from the park But Willie never so much at four as they are We'll be there soon, sister The dykes be right under us now Crushed flat by the flood Willie, do you know a man named Sidney Kingham? I, sister Do you know where he lives? Maybe Norwich this night Willie finds him Willie, did Mr. Kingham come often To visit Miss Kahn? Not often Mr. Jason, he hated it He hated it? He was afraid Mr. Kingham Be taken Miss Valerie away He was a mean one, all right Orting others be his pleasure Mr. Jason was a very fit man He hurt me Through a jog at me he did And caught my head I'd be bleeding like a pig Why did he do that? He said my hands be thickened and steelen He'd be looking for some letter But Willie didn't take the letter Not then But you did take it, late man I, after he caught me with the jog I, I, I took it Oh Willie He punishes Willie for picking Well, Willie be picking What did you do with that letter? I, I'd be hiding it in the room Hiding it? Why? Do you know what it says? What it says? No? I just smell it every night Before sleeping Willie, when we get back to the convent I want you to show me that letter I, I, sister Smells proper sweet it too Ah, lilac Like lilac combs the springtime We, we be near the land, sister Oh, terrible father I can't see a thing Stay close to Willie and be careful Soon we be in Norwich, proper They were back at the convent shortly after dawn With them was Sidney Kingham Mrs. Pierce rushed at once for sergeant's notice Who's this, sir, you've been to Norwich? Yes, sergeant Well, of course you know to find the authorities there That were here No, no, I didn't have it in mind Well, that's unfortunate However, I'll have Willie or whatever his name is Take us back immediately I'm afraid that's impossible, sergeant As soon as we return to the convent We'll be there I'm afraid that's impossible, sergeant As soon as we return, Willie's smashed the boat to pieces Oh, why didn't you stop him? He couldn't Oh, this is Mr. Kingham, sergeant Kingham?  Mrs. Smithsons had her baby He said you promised to be with her Yes, yes, I did promise What about her and the baby? She's all right But the baby's barely breathing It's hopeless But Dr. Jeffries is still trying I'll see him at once Just a minute Most of the villagers are here And they know all about you How you had to go and bring that That evil woman's love about Her pleasure meant more to you Than poor Mrs. Smithsons and her baby If I were you, I'd keep away from the villagers Thank you for warning me, Nasser I had everything, sister Mary Yes, the infant's alive But how much longer I can't stay Would you like me to take over, doctor? Well, you must be twice as tired as I am No, I'm not tired I won't undo what you can If you respond at all, try the oxygen game I wish you'd spoken with me before you went to knowledge I'm afraid you couldn't have changed my mind You'll break your heart if you get involved In this calm affair Well, call me if there's any sudden change Oh, doctor, how is your wife this morning?   Mrs. Jeffries? I just said lilac They're my favorite flower They may be smelling sweet on the letter If sister Mary lets me keep it What are you talking about? You break it as usual Tell sister Mary The letter Mr. Jason be looking for Jason come? Aye, don't you know that Mr. Jason's dead? Aye, he be proper dead Petl, we'll talk later Give the letter to me and I'll take it to her I don't have him I'll be hiding it in my room Mr. Harmer, I must go and lie down I don't feel well I'm so sorry, Mrs. Jeffries My husband, he sent for my husband He's still in the ward, Mrs. Jeffries I'll find him I'll go and lie down Please tell him to hurry We're all too occupied that morning for me to press the mattress for sister Mary's journey to Norwich But it was something I could not ignore Finally, I was able to summon them all to my office Sergeant Merring, Dr. Jeffries, Mr. Kingham and sister Mary Now, first of all, madam there's a matter of this report I've prepared a simple declaration of facts concerning the deliberate destruction of the robo Have you read this report, sister Mary? No, Reverend Mother, but I'm certain it's accurate I regret having to do this but there's been an obstruction of justice Oh, please don't worry, sister Mary I'm sure they'll forget all about it after valley Well, I mean... What do you mean to say is after she's been hanged? I'm sorry, Kingham Regardless of official charges, sister I wish you'd use better judgment But Reverend Mother, my heart went out to that poor girl Has any real harm been done in letting her see this man? The fact remains, she's a convicted murderess I don't suppose I should say it But Jason Kahn deserves what he got Why shouldn't you say it? You see, the torment and abuse vary constantly Oh, stop being so certain of her guilt Both of you Sister Mary, look at the facts, sister I prescribed tablets to let him sleep Harmer made up a three-week supply the day my wife and I left for a holiday Twenty-one tablets Before I left, I cautioned Valerie to give Jason only one each night No more, no matter how severe the pain And four days later, Jason was found dead of an overdose Yes, I know After four days, how many tablets should there have been? Seventeen And there were seventeen But two of the remaining tablets were aspirin Even so, that doesn't prove that Valerie puts them there Jason couldn't reach the fire He was paralyzed And even if he could Why should he want to replace the tablets with aspirin? It doesn't make sense First off, if Jason Kahn was paralyzed and Valerie Kahn was alone in the house with him And I told you what she said to me the time Will he overheard it?  Yes, I had reason to feel that she didn't When Jason had his second stroke, the one which left him paralyzed If she hadn't sent for me in time, I... I think he would have died So she could have let him die, but didn't Yes, she could have, couldn't she? She saved him from dying a natural death and then turned around and poisoned him I don't understand that kind of logic It's conceivable that one can come to the end of one's patience, you know? No, not Valerie Only the day before Jason died, I begged her to send her to a nursing home so that she could be free to marry me But you wouldn't hear of it You couldn't say she'd come to the end of her patience, could you? I'd let no sense in putting her on trial again Nothing we can say can change a jury's verdict Oh, please, keep away Lonely break your heart Well, if you don't mind, I'd like to be alone with Sister Mary Thank you for coming, gentlemen Yes, thank you, Reverend Mother Would you mind closing the door? Yes, ma'am Dr. Jeffery told me that Mrs. M's baby's a little better We won't know much for a few hours yet If the child survives, he says we'll have you to thank We have God to thank How much better it would be, Sister, if you can find all your efforts strictly to your duty Oh, she's innocent, Reverend Mother Never in all my life have I been filled with with such complete convictions Then you must renounce that conviction You'll have no further contact with Miss Karn You'll not speak to her again But, Reverend Mother This girl has prepared herself for death It would be cruel to raise false hopes in her heart That will be all It was evening once again As I walked to Bethbers, I saw Valerie and Mr. Kingham in the garden Watching closely was Mrs. Kier Valerie, please, you're only torturing us But we know what's going to happen to me Why not talk about it?   No, and told me But if you didn't do it, there's only one other person who could have Who else had a reason to kill Jason? But I didn't do it And I didn't do it? Well, nobody did it Maybe it didn't happen Maybe it's all a joke Valerie Jason's at home waiting for his dinner We decorate this, we get him I don't want to go in Oh, it's beautiful The evening, the fresh air I didn't do it, Sidney I'm innocent and I'm young and I'm in love with you I want to live I want to be your wife I don't want you to touch any other woman but me I don't want to die Darling, my darling Mrs. Karn, Mrs. Karn Mr. Mary Something important has just come up This letter What is it? It was written to her brother to Jason Karn How did you get it? From Willie Your brother accused him of stealing it He punished Willie Then later when Willie signed the letter He decided to keep it It was his childish way of satisfying his resentment Do you recognize the writing, Mrs. Karn? No It was obviously written by a woman Mr. King, please read it Here is Jason I'm mortally afraid of getting in touch with you And yet I must We are suspected I haven't been questioned but I feel it There is nothing either of us can do Except to go on as if we'd never met If you're asked, you must deny everything Take care of yourself and please, please get well I'll never change, I couldn't if I wanted to But, but no signature Have you any idea who it could be? No The envelope postmarked Norwich But he knew women in London but But Norwich Why didn't Willie produce this letter of the trial? He was afraid And besides he never read it He kept it only because it smelled sweet Are you sure your brother never spoke of anyone? No, no telephone calls, messages Mr. Mary You are definitely warned not to come in contact with the prisoner I know, Sergeant, I'm sorry I have absolutely no right to continue to interfere Here's, take Ms. Carnes to her room From now on, no more personal favors and no visitors Take her away Give up, Sister It's too late Come along, Miss There's nothing more you can do As for you, Sister Yes, Sergeant I've reported you again to the Mother Superior See you at once Five There's a thought for you to keep in mind As many another American has In 1945, Lyle Hayden was sent to Iran By a privately financed organization To help the farmers with their agricultural problems At first they were listless and disinterested But Hayden started a small demonstration farm To show them what could be done He began to drill for the water He was sure lay beneath the villages And when he hit it His second hand pump began pumping 15,000 gallons an hour Now the Iranians welcomed his help With their aid he purified the water Removed the threat of malaria from the irrigation ditches Started a successful chicken breeding program He opened a free school To teach the children reading and writing And it was so successful That the Iranian Minister of Education Asked him to organize his teaching methods In other villages Hayden offered a teaching job To any young villager who could learn To read and write The successful ones came from his night school classes As the months and years went by Hayden continued educational and agricultural programs Throughout the country And today, what prosperity The peasant farmers of Iran enjoy Can be attributed to the tireless work Of Lyle Hayden Who combined the best qualities Of missionary and businessman To win the thanks of a grateful people Once again, an unselfish American Proves that by helping others You help your country We pause now for station identification The curtain rises on Act 3 Of Thunder on the Hill More including Claudette Colbert as sister Mary And Barbara Rush as Valerie Kahn Yes, I've sent again for sister Mary And for good reason This latest interference was inexcusable I cannot defend my actions, Reverend Mother Other than to explain It's fine, sister You openly disobey my orders I felt I had to see Miss Kahn Because of the matter This letter suggests only that Jason Kahn Had a secret romance But doesn't it suggest That a third person's involved, Reverend Mother A person who may have decided to take revenge That's merely a guess You've done nothing but harm This unfortunate young woman Raising false hopes Bringing her fiancé here Rekindling her passion and desire to live All that I've done, I've done Because I felt compelled to help her Compelled, yes But not so much to help her As to prove once more that you're right Oh, Reverend Mother Surely you don't think that I Stubborn fanatic manner In which you cling to your opinion I can't even call it an opinion It's just a wild notion An idea It's not just an idea Suddenly this letter You were convinced of her innocence long before You saw that letter Admit that much I do I was convinced of it the first time I saw her And every moment I've spent with her Since has only strengthened my faith Don't call it faith, sister Call if I its proper name A premonition, a wild notion How can you be so stubborn?  If I could only believe that God was with you Yes? Reverend Mother, it's Willie In his room in the bell tower Unconscious and a terrible wound on his head Who could have done such a thing? You've called out to Jeffrey? Yes, and the poor lad's belongings All strewn about If it's someone who tried to rob him Rob Willie? That doesn't make sense Except for the letter What does Willie to do with this letter? It goes back to when he worked for Jason Cards He found it, he kept it Oh, don't you see, Reverend Mother? Someone here in our house Must have found out about it And tried to steal it Nonsense It probably was one of the villagers They're all outraged with him For helping you to bring Mr. Kingham here For destroying the boat But if there's one child I forbid you to go on with this No more of this playing, Scotland Yard You have a hospital full of patients They're your only concern, not this letter There's the fireplace, sister Mary I want you to throw this letter Into the fire No, I can't I beg you for the sake of your soul I can't, I can't Then I shall burn it Oh You can forget about the letter now And try to forgive me Willie was badly hurt, but he'd recover He had no idea who his assailant was Dr. Jeffries left him and went to his quarters His wife was lying down She was very distressed Did you get it, Edward? Get what, Isabelle? My letter, the letter I told you about But you didn't tell me about the letter, dear? I did, I did The letter I wrote, Jason You've been sleeping, Isabelle You had another one of your dreams I'd hope you get over them Edward, this wasn't a dream How do you know?  It wasn't a dream Ask them, ask them Let them know how ill you are I've told you before, Isabelle I can help you only if you trust me I don't think you'd like me to admit you to a nursing home Edward, no No don't It was a dream A bad dream We'll have supper with Mr. Harmer You'll feel better, dear I made it especially for you Jeopard's pie and Yorkshire pudding Especially for me, sister Josephine You make it sound like my slasper Now, now, child, you mustn't even think about it Haven't you heard the news? The telephone lines are being repaired The flood water is running off They'll probably come for me tomorrow Please Eat your supper before it gets cold There's something the chaplain reads to the last I heard it once from the cell next to mine I am the resurrection and the light Fails alone And he that believeth in me Though he were dead, yet he shall live And he that believeth in me shall never die If you don't mind, sister The audits are no visitors Yes, Mrs. Kess I'm leaving I must follow dear lord for I'm lost Grat me, please And forgive my sins and help me Who is it? Kingdom, I must speak to you, sister Mary I'm sorry, Mr. Kingdom But it's terribly important I need your help The telephone's been fixed Melling's already called the Norwich police They're sending a launch Then it's all over I won't let them kill her I'll get her out of here somehow But that's impossible Now if you help her Look, if you could arrange for Mrs. Pierce To leave Valerie's room for just a few minutes I can't do that And you mustn't You'll be caught, you know you will Then I'll do it without your help You know nothing about it But I do Oh, this makes it clear how wrong I've been I've done nothing but harm Nothing but evil will come of it You've lost your courage, sister You know she's innocent What about the letter, the attack on Willie?  Oh, yes, here Did you know that Jason Carnes Once wasn't arrested in London For assaulting a man he claimed infocusing? It has no bearing on his death, Manador And did you know that Mr. Harmer the chemist Has two key rings? Sister, please What difference does it make? Well, I found the testimony very interesting It's right here, sister They asked Mr. Harmer Why do you keep two separate key rings? And Mr. Harmer said In case one gets lost They're identical keys I remember every word he said Identical keys and identical order And identical key rings Ah, there was more to it than that Listen to this, sister Do you recognize all the keys, Mr. Harmer? Indeed I do Well then, name them The keys are as follows House, car, garage, barn, shop Poison cabinet, dorm Wait a minute Wait, there's something wrong here Something wrong? Look at this newspaper The counsel for the defense asked Mr. Harmer If he'll also name the keys on the second ring Yes, I know Harmer said it was ridiculous That the keys in each ring were identical But the lawyer insisted, didn't he? And here's what Mr. Harmer says House, car, garage, barn, poison cabinet, shop Store room Mr. Harmer is a meticulous man But don't you see? In naming the keys on the first ring He said house, car, garage, barn, shop Poison cabinet, store room And on the second... On the second, he said shop After poison cabinet Well now, what did I tell you? Never burn the newspaper Sister, sister, come with me Please, we've got to see Mr. Harmer I said, sir, I'd never make mistakes That's one of the few virtues I can boast of In all modesty, of course But you've just read what the newspaper says Your own testimony, Mr. Harmer I don't argue that, sister Mary I merely suggested the mistake was not mine Of course, class, perhaps, for the newspaper reporter Then you're certain the order of the keys Was the same on the screen? Well, now that's very easily confirmed This is the key ring that I carry with me And over here on the shelf is the set of duplicates Jamie, I... Oh, I hope I'm not disturbing anything Not at all, Dr. Jesses Dr. How's Willie? Oh, he'll be all right, I think I was on my out to see him now I just remembered I... Hadn't given my wife a sedative Anything I can do for you, Mr. Harmer? Oh, no, no, Dr. Just a little mess about my key ring Oh, but neither Here are my two key rings They were both entombed by the court I got them back I never thought to look I suppose we do so now Each ring has 17 Correct? And each key... It... I do I do Of course, Mrs. Edicure Edicure? Well, that's not what I mean It's me You are right They're not in the proper order at all Someone took the keys to the poison cabinet They put it back on the ring Is it wrong, please? But who could have done it? Who? I hope no, Mr. Kidman I think it's perhaps us Oh, Willie Who else, Mr. Harmer? Well, any of my professionals I can tell for that matter The doctors, the visiting nurses of the district They all have the wrong of this distinction Fine Sergeant Melling, Mr. Harmer You must tell them at once Of course Of course, Ed, once Mrs. Edicure? How can I help you to keep resisting me? Edward, no No, I've had my sedative I don't want another But this is only half a grain, darling Then you'll be short of youth No, Edward, don't I don't want another Edward Mr. Mary had not gone with Mr. King and Mr. Harmer in touch with Sergeant Melling He remained in the dispensary Now in her hand was a tiny box Dr. Jeffries had dropped it in the waste basket when he left the dispensary a few moments ago The box had contained a powerful sedative Please, Ed, with no more It's Sister Mary, Mrs. Jeffrey You've got to get out of bed You've got to keep moving You'll be all right if you keep away Edward, there it is Jeffrey, you knew Jason Kahn I'm not allowed to say it Oh, you must tell me You can save an innocent life Now lean on me and just keep walking Where are you taking me?  Jason Kahn was a hopeless crippled doon to die No No, he was going to get well Isabelle was going to go away with him What is only one thing to do We must tell Sergeant Melling the truth That's all right There's no way to escape I'm tired How are you going, doctor? He's downstairs No, I saw him just a moment ago He's gone up to the bell tower Bell tower?  I've known it for a long time My love for Isabelle is a thickness An incurable one And you would have let Valerie die I placed the two aspirins in the files For a diverse suspicion for myself It happened to fall on her As you can see, I'd stop at nothing to keep Isabelle Why couldn't you have let things alone, sister? Why did you have to interfere? Where's Sergeant Melling? Where?  Why don't you pray? Maybe your higher authority will hear you Haven't I? I beg you, haven't I? The rail, sister Let go of that rope Let go of that rope It was Willie who first heard the bell and saved her Willie whose room was near the tower And who, in spite of his wound, overpowered the unfortunate doctor He fell, your ever mother Fell to the stone below I had no thought to hurt him Only to help Sister Mary God forgives you, Willie You have the gratitude of us all I'd be better by morning I've been lighting the fires like always Come five o'clock Sister Mary Brother mother, I disobeyed you again Yes, thank God you did You've shown great ingenuity and courage I'm sorry it was made it so difficult for you Thank you I don't deserve your praise It's Sister Josephine who really saved this town Me She insisted on my reading the newspapers After I'd given her up hope Sure, I only insisted Because I happen to see Miss Collins Unafraid, praying for almighty God Now I know her, mother We were chosen by him Yes, I think you found the answer, Sister To be right is a heavenly gift That's wasted unless it's wrapped in meekness and grace Theographies Theographies In a moment, our stars will return At ten minutes after two, on the afternoon of August 5th, 1949 The country of Ecuador suffered one of the most disastrous earthquakes of the present century The dead numbered from six to eight thousand Twenty thousand or more were injured and a hundred thousand were made homeless Fifty-three towns were destroyed or badly damaged But from United States Army installations in Panama Came immediate aid Military planes shuttle back and forth with emergency supplies Food, medicine, clothing And especially tents for the homeless Besides material things Doctors, nurses and engineers volunteered from far and near With their own hands in the work of saving lives The warm-heartedness of these angels in uniform Is remembered in Ecuador as a bright light in its darkest hour Such acts as these, by you and your friends today Are shaping our world of tomorrow And now, Mr. Cummings, with our stars And here they are coming forward for a well-deserved curtain call Claudette Colbert and Barbara Reich How about your show next week, Irving? Next week we are again fortunate in having an outstanding screen personality An actress who has proved herself expert in both comedy and drama Lovely, Ginger Rodgers And for next Monday night, we thought you might like to hear Miss Rodgers starring in a delightful comedy role That of a housewife suddenly possessed of all the money she needs Because she discovers it grows on trees I'm sure you'll agree that the theme of this gay universal international picture Will appeal to everyone Well, it certainly will, Mr. Cummings Good night, good night Good night, and hurray back Third in our cast tonight were Norma Vardon as Mother Superior Leo Britt as Dr. Jeffries Paul Treese as Willie Mary Flynn as Isabel Herb Butterfield as Mr. Hammer Othello Neschmiss as Mr. Josephine Nano Hurley as Sydney Joseph Kearns as Melling Gene Wood as Nurse Phillips Lois Corbett as Pierce And Yvonne Petey as Mr. Agatha The Hollywood Radio Theatre is produced by Mr. Irving Cummings Our orchestra directed by Rudy Shrager This is Ken Carpenter inviting you to be with us again next week For another worldwide presentation of the Hollywood Radio Theatre Brought to you through the facilities of the United States Armed Forces Radio and Television Service


This was one of the things that I paid, well, I went $80,000 and dead. Guess it wasn't priceless. Yeah! Yeah! Yeah! Yeah! Yeah!