Hello everyone, my name is Christina Young and I've been on the pan-cancer analysis of whole genome project for the last two years or so. So this is a large-scale project that is a use case, can consider to be a use case for all the cloud computing basics that you've learned over the last day and a half. And so this is a project where we utilize multiple cloud resources, not just one, we used actually 14 cloud resources to get this project done. And I'm going to show you some of the obstacles, some of the challenges we have encountered, some of the lessons we have learned, because the lessons learned are very valuable as we're only going to get bigger and bigger with our data analysis. There are also a lot of resources that came out of this project, whether you are a researcher, a bountenetician, there's hopefully something that you will find useful that we consider to be legacy data set or legacy analysis pipelines. So at the end of this, you will run one of the workflow, BWMM is an alignment workflow. It typically takes days to run, so that's why you would just run on a very small test data set just to get a flavor of it. So what is the pan-cancer analysis of whole genomes? This project started out two and a half years ago with the goal to collect 2,000 whole genome for cancer patients. So we want a tumor whole genome and also normal whole genome. So this is, you can consider an extension of what TCHG has done for pan-cancer exomes. So we made a call out to the ICGC members. We have very overwhelming response. We end up collecting over 2,800 donors with whole genomes. And this is different from the exomes because now you have data to look at non-coding regions, structural variations, and any pathogen insertions. But of course, we still are interested in any driver mutations, whether they're coding or non-coding, and also driving driver pathways. So this project is organized with a steering committee of five members from different institutes, from Peter Campbell from Sanger, Gettigetz from Broad, Jan Colbell from Amble, Lincoln Stein, who's the director of bioinformatics and biocomputing here, and Josh Stewart over at UC Santa Cruz. So at the beginning, we also asked researchers to submit abstracts on what they would like to do with this data set. So we received 130 abstracts and based on these abstracts, 16 working groups were created using the different scientific themes. And then there's also a technical working group, which I'm involved in. We are responsible for aligning all the genomes because one goal we want to make is a uniform alignment and also uniform bearing calling. This will allow us to eliminate any variations that is because of different processing pipelines. So these are the 16 working groups. You see obviously we have mutation calling at the top that's very important for this project. We also have groups specialized in looking at regulatory regions, transcriptomes. So transcriptome is another data set we do collect in this project. And we also have methylation data at the smallest scale, but it's still important to look at. And of course we look at driver pathways, mutational signatures, and now we also can look at germline cancer genome across these patients. So all in all is a very interesting set of questions, scientific questions to probe. And now these working groups are working hard to try to make some discovery out of this data set. So out of these 28, 100 donors, they are really collected from multiple projects. So as you can see there are a large number of projects coming from the United States. These are actually from the TCGA projects that have whole genomes. And then we have other projects coming from Europe and from Asia as well. So based on this map we got to make our planning. How are we going to collect the data? Because there's so much data we cannot actually have just one center centralizing all the data. We need to distribute the data and this map actually helps us plan because obviously you need a server or data center in Europe, in Asia, and also in North America. This bar chart just shows you the primary tumor sites that this project have data on. Each little color is actually a different project. So for example in pancreas is actually we have four different projects contributing the pancreatic endocrine tumor from the Italian group, from the Australian group, and we also have pancreas adenocostenoma from the Australian group and also from Canada. So it's actually a mesh of projects contributing to the different primary sites. So the initial roadmap as I said was to collect 2,000 donors. That alone would give us 600 terabytes of unlined BAM. So that's quite a lot of data to handle. And we asked the data owner to submit this data by giving us lane BAM levels. And it's very important right from the beginning that we establish the metadata that has to go with the data. Otherwise it's just BMS to track. So they found the metadata, they upload the BAMs, and they upload to a server called Genos. It's a gene torrent server. And the good thing about Genos, although they had a lot of technical problems like hung uploads. I know someone is laughing there because they have a bad experience. Your uploads will hang, it will abort, you have to contact the administrator to help you out. But it does have one advantage. It does validate your metadata. If it is not correct or matching, it will not accept your submission. It also makes sure your file size, it does check zones. Your file sizes, file names are everything as expected. So once it gets in, it's very useful to have all that metadata associated with the BAMs. So we started off doing alignment back in August 2014. It took us about four months to align the 2000 donors. And then afterwards, we actually have a second train for people to submit more data because they did not make it into the first train, but they wanted to be part of it. And that's why we got another additional 800 donors. So after the alignment phase, we have three core varying calling pipelines. One from Sanger, another one from DeCabs Enombo, and a third one from Broad. Of course, at the time, there was question, why these three pipelines get to be the core? And why do we need three? Why can't we just do with one? It seemed like the decision was political at the time. But when we looked at the validation, so on the right, this is our path to validate. Out of the 2000 donors, we picked out 63 donors. They were picked because they had sufficient DNA for validation. And also because of the material transfer agreement that a sample from one country can go to another country for sequencing. So with these 63 donors, we did the deep targeted sequencing. It actually took nine months to do because it took time to strategically pick the variants for validation. You cannot validate every single one of them. So they didn't need to be picked out. And then a panel had to be designed. And it simply took four months to order a panel before we can actually sequence in the lab and do the analysis and got the data back. Now, thankfully, after all that validation, we realized the three core pipelines do perform much better than the other callers that participated in the validation phase. So we did have the three core pipelines correctly. And even better is that when we merge the calls from the three pipelines, we do have better accuracy than any of the pipeline alone. So that's why we actually merged the calls from the three pipelines to get a consensus set that gives us very high accuracy. So as I said, in the alignment phase, we had already 600 terabyte of raw data. It also gives us 600 terabyte of aligned data. So we had to host this data in multiple sites. And also questions we had at the time also, well, if we aligned it at one time at one center, aligned it a second time at another center, we would get the same results. And we did check and we do. So in order to track this data, we use elastic search indexing. So this is compiled every night so that we know where the data is, what state it is, it is in, whether it has been aligned or not aligned. Before we had elastic search, we had situations where some sample was aligned multiple times. And so you don't want that. That's just a waste of your compute. So to do this, we had seven data centers over at UCSC that at the NCI cluster at the time, it was hosting CG Hub. So all the TCGA data could only be hosted there. We also had a compute center at the University of Chicago that protected data cloud. And then in Europe, we have three data centers, EBI, Barcelona, and DKZ. George has a question. At those stacks, we have basically our tendency and we have accounts and we can go in there, start VMs, and do any work that we need to do. Other places like the University of Tokyo and Barcelona, Barcelona is actually an HPC environment. In those environments, we don't even have accounts. So we only could give the software to the team over there and they run it for us. They are our cloud shepherds, as we call them. They're all Illumina data, actually. Yes, so it just so happened everyone was using Illumina, so we didn't have any issues with different kind of data. So to do the alignment, we had, at the time, we basically asked the data owners to upload the data to the local regions. So there's no overlapping of the data and each center was just responsible for aligning the data at their own center. And this requires about, this requires an eight core machine, 16 giggram, about five days per sample. We had nearly 6,000 samples to align, so that's why it took that much resources and that much time. So as I said, it was done within four months time. The second train comes in with 800 donors. That's okay. We knew how to do it all over. That was easy. But then when it comes to varying calling, it was actually a lot harder because, first of all, these pipelines were not just ready to go. They were running very well. They've been running very well in the local institute at Senya and Brode. For example, you know, Brode has something called fire hose. This is sort of their August data to run productions.  So whenever when you try to move pieces of code like that, it bugs are created. We're all humans. So things are not smooth at all. It takes a long time to put that code to a different environment. They all happen at different timings. And also these pipelines are very specific to their own sequencing centers. So for example, Sanger had a typical way of naming their samples. Their tumors and normals are always named differently. But then when they run it on another sent to who happens to use the same names for the tumor and their normal in the header, they run into issues. So this was a challenge for every pipeline here because they're facing something that they've never seen before. Now Brode also had pipelines that are completely new. They're not even published. So it was even more challenging to try to run these in the production environment when they may have to pull back and change the parameters. But the key thing here is that more compute is required. So here's actually a list of their algorithms if you're about imitation and are curious about what was done. In each of these pipeline, we call SNVs, single nucleotide variants, indels, structural variants, copy number, germline. And so they all each have their own algorithm. But all these, they start off with this downloading the data from Genos and eventually they have to upload the data back to Genos. So this is what George said before.  Now these are core hours that were needed per donor. So 800 for Sanger, 800 for DeGavza, and 2300 for Brode. These are average numbers based on our runs on Amazon. And I'll tell you that after running so many environments, Amazon typically gives us the best performance. We don't know exactly why maybe they just have the latest CPUs that have more optimized IO network performance. So when you go to any other different environment, you would expect a slightly slower runtime. So we had a challenge also that back then that we didn't really know what these pipelines need in terms of cores or memory. So there was a bit of trial and error. And like George said, the lesson is start small. You pick different kind of samples, bake in small, and try it out so you get a flavor of what exactly to do next when you try to scale up. So when we scaled up, we actually went up to 14 compute resources. We're lucky in the sense that our members volunteered additional compute resources, such as iDash over at UCSD, Sanger Institute, and also even here at OICR, we added additional compute resources. So here at OICR, we also have an open stack. And very importantly, there was a change in policies. So for the longest time, we could not put TCGA data in the cloud. And then the policy changed. We're allowed to process TCGA data and also ICGC data in commercial clouds. That's a big change. So that's why we started using Microsoft Azure and also Amazon Web Services. That really allows us to scale up, even though it costs a bit, but that allows to really scale up and get resources when we need them. So where are we today? We have actually finished all our analysis. But as you can see, we had that burst of BW alignment. And then the three core pipelines actually start at very different times. And so that's why we had this gradual flow of completion over two years time. And we did some calculation. If we were to do this all over again, and if we managed to say start all three pipelines at the same time, right after the alignment, we can possibly get this project done within four months, provided that we can get all the compute needs that we can. So we'll say at Amazon. So that's a very valuable lesson because it would be much faster if you want to do this all over again. Now, just to complete the story, core analysis is done, but it doesn't mean that the project is done for the data to be good. So in order to get good quality data, we actually get the calls out there to all the researchers, ask them to take a look, okay, where are the problems? What are some problematic donors? You don't expect every donor and ever sample to be perfect. So with about three months of looking at the data, they found that we had to exclude 6% of the donors. Sometimes it's the obvious reason that we don't have any clinical information. Other times it's because we've discovered that tumor itself was contaminated with foreign DNA. Sometimes the normal that gets contaminated with the same donor's tumor DNA. And then there are cases where we see CDNA or mouse contamination. And also this exercise requires looking at a lot of plots visually. And then based on visualization of structural variants, then some artifacts could be discovered. So 6% being blacklisted, that's pretty normal, I think for a project of this scale. And then we had another 2.6% of donors that have low level contamination. We did manage to rescue them, we think, by filtering the contaminated calls, but it's never complete filtering. So we just flag them as ones that people have to pay attention when they do the analysis. So ultimately we end up with 2,583 donors. And some of them have multiple tumors. And then half of them have RNA-seqs data to match with it. So this is a good resource for anyone interested in large-scale analysis. Now when I talked about filtering of artifacts, there are actually multiple kinds of artifacts that needs to be filtered out. So there's oxidation. There is PCR and strand bias. We also did non-robust mapping. That means we take the read and blot it against the reference. And if they map to a better another region with a higher score than the one that was aligned to, then that read is out. And then we also built a panel of normals using this cohort in order to get rid of some of the false positives. And any SNVs that overlap with germ lines or the thousand genome SNPs, those are filtered out. And in some small cases we did have calls on chromosome Y, even in female donors. So those were filtered out as well. And other groups have looked at it and tried to look at annotations. So for example, we have a mutation of signature group. They think certain things are artifacts. And also there's the observation that they're an enrichment of SNVs near indels. So that needs to be flagged in case someone finds something interesting in those regions. They need to be aware that they could be artifacts. And then we provide other estimates and ratings to help guide people in their analysis. So now finishing the story is that using the consensus strategy, we managed to get a very high precision and sensitivity for our SNV calls. And then for indels, as you know, indels are always hard to call. If you overlap any two independent indel caller, the overlap is only 50%. So in this case, we did manage to get 90% precision, but of course, sensitivity slightly lower at 60%. So any questions about the project at this point? Okay. Then I'll talk about the lessons we have learned from this project. These are more, I would say, technical implementations that we should do differently the next time. So during this project, the way that it's done, we have 14 environments, but all we have is actually something that is centralized metadata that we can look at to see what donor, whether the donor has gone through alignment or which varying calling pipeline has been done. And certain, if the pipeline has not been run, then okay. The project manager will tell a specific compute center or a cloud shepherd to run a variant pipeline on a specific sample. So this is where the project manager has to communicate to the cloud shepherd. We use git to do that actually, just put certain files or data in the git so that the cloud shepherd can go there and know which one, which donors to run through each pipeline. But the monitoring is a bit scattered. So in order to know what's going on, project manager actually has to ask the cloud shepherd, hey, I haven't seen any completion of donors over the last couple of days from your compute site. What's going on? So it was more of a human intervention in this case. And so what I'll show you in a couple of slides is a better case that we can automate these things and make it a lot easier, especially when you have multiple clouds to manage. Now, there is also the debate, well, we have academic clouds, we have Amazon, why don't we just stick with Amazon? It's a lot easier. Or maybe let's just go with academic clouds because it doesn't cost us anything. Well, I think we need both. The reason is sure with academic clouds, you have like Collaboratory, for example, it's a low upfront startup cost. So that's a good environment to start small and test and learn from it before you scale up. But also keep in mind, each environment, it's going to require at least a half full-time employee as a cloud shepherd to monitor that site. So the more sites you have, the more FTU you're going to need and salary costs as well, as well, not just compute. The other reason, the other cases where we cannot use academic is because one workflow such as the Broad actually requires a lot of resources. It required 32 cores and 256 giggram. And very few institutes can give us a large number of VMs with that kind of resources. That's when we actually have to go to Amazon and Microsoft Azure. Okay, so well, what about we run on commercial clouds? That's great because you can start up really, you know, a couple hundred of VMs with only half an FT monitoring these jobs. So you can get a really burst of productivity there. However, there are some tricky things here. Certain data cannot go on the cloud. So specifically, the German cohorts have told us that the data cannot be processed on any commercial clouds. They can only be processed in academic clouds. So that gave us a challenge as well. And also, so I think it's both, because commercial clouds are all owned by you Americans right now. So it's a bit of both. But if we run everything on commercial clouds, some of the donors will cost us a lot. They have run for over two months and even three months at a time. And you just wait and you don't know when it's going to finish. We're a little nervous about our costs at the time. But now we have learned we can actually use BAM statistics to predict which ones are going to be long running. And then we can save those for academic clouds and not pay for the long running time. So as I said, metadata is key. Our projects so far have only focused on whole genomes and RNA-seq. But more and more data are going to come online with BISELF-I-SEQ. And it's definitely important to establish the metadata right up front when you start a large project like this. Now during our projects, as you can imagine, we experience a number of outages at the data centers. Or like you said, sometimes you try to download from EBI and that EBI is not available. And then your instance is just sitting there and waiting for that download to happen. So A, I think we need to do replicate data to multiple data centers to have redundancy. And then next we need to have a downloader that is smart enough that if it tries one data source, it doesn't work, go to a second data source, go to a third data source. And if all the data sources are not available, shut it down, shut that instance down, and then notify the cloud sharper. I'm having trouble downloading from all these data centers is maybe we can't run anything for a day or two that has happened as well. Now also we have crashes at some sites, unexpected, maybe power outage. But for whatever reason, we need some way to handle these crashes more gracefully, because some cloud shoppers have actually spent a good day just to clean up after a crash. So the queue can be cleaned up and then can go to the next job. So the queuing system needs to be simpler, I hope, for cleanup like that. And also if we can have a queue that goes to multiple clouds, then if one queue, one cloud doesn't work, then schedule it to another cloud. That would be very nice to have. And so as I said, we could be smarter than we did during this project. So George has already showed you an example based on the size of your BAM file, choose your instance correctly right from the beginning. So in this project, because of the way the queuing system was, we always use the same instance size for every sample. And then so it might fail because it didn't have enough memory, enough just space. That's something we could avoid by choosing the right VM right from the beginning. So some kind of calculator would really help. And then we can also predict our long runners based on BAM stats. So that's important. Now as I said, cost of compute is typically a little lower than your FTE. So we try not to get a cloud shepherd to get involved with failed jobs. And so it would be nice if the queuing system is a little smarter. Yes, it's start off with a good VM already. But if the job fails, maybe progressively try another VM that is larger with resources, and try it again, maybe try three times, and then eventually it does come back, then the cloud shepherd can get hands on and look at what the problem is. And like George said, we need monitoring. So very good monitoring system to tell you of any problems. So you don't have to look at every single VM to find out if they're running or not. Because sometimes you have 100 running, and it's nice to have a report that a VM has been idle for a couple of hours already, report back alerts you so you know what it is doing because it's easy. We have had VMs sitting idle for days just because no one caught them. Last thing that may not be as obvious is actually to validate our results before submitting the data back to the server. A lot of times we had good data done, but then we had cases, well actually maybe a chromosome was missing. Those little things that we have all experienced but forgot to validate before uploading the data back. So this is what I like to see as a project manager. I like to just be able to have one system, an Augustrator, that allows me to queue my jobs to multiple clouds that gives me monitoring across multiple clouds, give me alerts, because at some point you can think of this actually as maybe even a grad student who is doing a very large-scale project. They got the DACO and DBGAP approval. They got a budget from a grant. Just Augustrate the work wherever is possible. And then of course at the end, sometimes they get a bill back, but then now this is algorithm pushing rather than pulling data into their local machines. They're just pushing the compute algorithms into the cloud that already has the data local to the system. So all this data moving should be avoided and should make it easy for one person to compute across multiple clouds. So why are these lessons valuable? There are a couple projects probably coming up that will be using large-scale analysis like this. So there's already a pan-prostate initiative that's going to involve whole genomes, exomes, and many other data types. The data is going to be in US, Canada, UK, Germany, Australia. So again, this is very distributed. Actually, I forgot to tell you, for the Australian group when they participated in the pan-cancer project, the bandwidth was absolutely terrible. They couldn't even upload the data to any servers. So they end up shipping us hard drives. We received 40 pounds of hard drives and have to upload the data for them. Now, the good news is that there's now an AWS Center in Australia. So hopefully that will solve the problem in the future. Another large-scale project is ICGC Met. This is an extension of ICGC that will include clinical trials and a lot more clinical information. The scale is going to be almost 100 times bigger than pan-cancer. And the plan is to collect the raw data, do uniform alignment, do uniform variant calling. There will be multiple data processing centers, typically based on region again, and we'll be doing exactly what it has been doing for pan-can. So this time, the project is going to last 10 years. So you're not going to have a project manager who'll be willing to sit there and assign workload on a daily basis. So all the things that I've just said needs to be really automated and there needs to be some intelligent system to get the job done. Any questions about lessons? Eight years? Getting more aggressive. All right. Sorry. We are just putting in a grant. So ICGC Met was started in 2018. So we have some time to put in the infrastructure and getting the project ready. Okay. So after all this hard work from a lot of people, what resources are available publicly to people? Either now or very soon in the future. So of course, data. People were going to care a lot about this cancer data. Methods. I mean, we put a lot of effort into perfecting these methods, gains validation data. So we'll share those two. And they're best practices that we have learned from this, we've established in this project. And those needs to be shared as well. So first to talk about data. Where can you find the data, right? So if you're not familiar already, this is the ICGC data portal is at dcc.icgc.org. And this is a portal where you can find the data you're interested in. So in this case, so you see that on the left, there's a faceted search interface. So you can pick the project you're interested in. So for example, these are all liver projects. And so yeah, so liver got picked. And you can, it's just like shopping. You narrow down your data set that you're interested in. It shows here what your query you have been using so far. And then from here, if you click one of these file ID, then you go to a page that will tell you a lot of details about this file. So in this case, this is a BAM file. And then it is actually available in multiple repositories that is listed here. So you see Peacock, I think this is Heidelberg, this is Heidelberg, Barcelona, London, and also the data is in Collaboratory as well. So this gives you a choice of where to download the data, which we'll get into next. But if you click on this BAM stats, you will see something like this. So this is actually a pretty cool tool that Vincent Faraday's team has used to give you real-time statistics of the BAM. So it's actually streaming the BAM. If you go to it, it will tell you the coverage. You can even zoom in on different chromosomes to get a sense of the coverage in those regions. It will give you all kinds of reads information as well. So it's kind of cool. There's another page, also a similar page for VCF that's being developed that you can visualize your base changes or your variants along the chromosome. So this is being developed right now. Going back to the file page, yeah. No, you don't actually. This is just on the website. And the way that they have the security setup, it will not leak any specific germline information. You can try it. It went through a lot of security checks before this. So this one will be individual, but they mask, if it is a germline that is not the same as the reference, it gets masked. So that's why you won't see any germline information that is identifiable. If you brick it, let us know. I always like breaking things. Going back to the file page, there's another thing that you can click on, which is the metadata page. This just tells you when this was generated, the kind of specimen type. But this is sort of an example of the metadata that we keep track of. Now this is all open data. So you can look at it without logging in or having any kind of account. But there is data that requires authentication. So if you already have a DACO application approved, then you can log into the site and you will see something like a token manager. So this is where you can set your token in order to download from Collaboratory, because a lot of data is already in Collaboratory, or to download from AWS. So yesterday when George asked you to try out the download, we didn't put any token in, because you don't have DACO authentication. And also we asked you to download data that was open access. That's why you did not need the token. So coming back to the page after you have the search results, you can download the data very easily by clicking a button called Manifest. So everything that you have searched and selected, you're interested in, it would just come in a manifest and you'll get in. So this is kind of what you will see at the beginning. It will show you, okay, you might start it. So this is GIF. So it actually is a little video that repeats itself. But when you first start with the manifest, it has actually over 1.5 petabyte of data if you download all of it, because they're actually residing in multiple repositories. Same files in multiple repositories because we have data redundancy. But it has a very smart feature that if you click this box called Remove Duplicate Files, then it will decide what to, it will remove the duplicate files for you based on the priority of repos you choose. So then that's why you can now move these repos up or down. So if there's one that you're specifically interested in, say Collaboratory, you want to use that as your primary repo, move it to the top. So most of the files will be from Collaboratory. The other files that Collaboratory doesn't have, it will go to the next repo to download. So this is a very smart tool and very useful and you don't have to worry about removing duplicates and all. The other thing that you don't have to worry about with this tool is where your data is coming from. So there's another new tool created by this, this Vincent's team is, they call the ICGC GET. As a universal tool, you could be downloading from Amazon, from Collaboratory, from any of the Genos servers, even from GDC, that is the new genomic data comments hosting on all the NIH data. All of it, you can just put it in this ICGC GET command, click tells it to download and then specify the manifest that you have just created previously. And this tool will just do everything for you. Otherwise, you will have to install different tools like GT download, also the ICGC storage client and other tools that are specific to the servers. Yeah, so the ICGC GET will have a configuration file where you put your token, your key as well, in order to handle all of that. So this is a, so ICGC GET is a universal tool, but the other tool that you tried yesterday was ICGC storage client. Now this tool is slightly, is sort of slightly different because it's specific to Collaboratory and AWS. And it also has additional features that ICGC GET doesn't have. So downloading using a manifest, I already told you already, there's something that's interesting called BAM slicing. So instead of download the data and then use SAM tools to view a region, you can actually use this tool to just get a small region back and look at either through SAM tools or actually I haven't tried IGV, but it would be nice to be able to see small regions in IGV as well. So you don't have to get a big file to your computer. Another nice feature is actually called manifest, a fuse. So if some of you work with fuse file system, you can actually mount foreign drive onto your local computer and it would just look like your own local drive. So you can do LS, you can do DU, all these commands. So this allows you to do that again. By giving it a manifest, you can mount all those files that you're interested in onto your own local drive and now you can explore it. You can look at the files and see whether those are indeed what you're interested in. So that's very powerful. Everything that I've just told you, you can definitely look at it in more details at the user guy at docs.icgc.org. It's very easy to follow some of the commands to get download the data. Yes, Francis? Yes, so as part of this project, we created mini bams that extract the reads flanking the variants. So it could be SMVs, indels and structural variants. Only those relevant reads are pulled out and the mini bams will be available through the portal eventually. We have an upload. Yeah, so for the entire project, if you look at the raw bams, the aligned bams, they're 800 terabytes in total. And then the mini bams comes down to only four terabytes. So it's half a percent of the original. So just the last couple slides. I wanted to talk about the workflows that we have. So when we started the project, we didn't use dockers. Back in 2014, it wasn't quite as hot. And then gradually we shift to dockers. And also we're now making all our workflows available as dockers. So we want to make sure things are reproducible. And any other researchers can use the same algorithms. And they're specifically registered at dockstore.org. So you ran the commands yesterday is a lot easier to just have a couple of commands and look at the and describe the workflow in common workflow language. So we're going to run one of them at the lab. So so far, the BWMM is available through dockstore. And also the Sanger workflow, DKFZ, Enombo workflow, those are all at dockstore now. We are going to get our broad pipeline in there as well. It's a little bit more complicated because there are a lot more components. And then the other one that we are working on is actually all the filtering methods that we've used. It's easy to ignore the filtering methods. But as someone said, that's actually the secret sauce to get really good data. And a lot of it actually came from the broad team. And for the longest time, they kept it a secret. And now they're sharing with the community. So it's a very good thing. So lastly, there are some best practices that have been established by this project. So for example, even just looking at your data and to determine whether it's good or bad is an art. So we will put together a Rhodes gallery of sequencing artifacts. We want to show some plots. They could be rainfall plots, IGV, anything from that help us identify problematic samples. We want to illustrate those examples with the community. And like I said, the code for filtering out the artifacts will also be made available. Is that politically correct? Okay. So the lecture part is actually over. But yeah, so we'll come back and do the lab after lunch. Any questions? So definitely, we definitely have library effects. So one that we famously call the Sanger effect. The Sanger knew that their library preparation has certain mutation context that are in high demand. So they filter those out and other sense. So we actually have to do those kind of filtering afterwards. And as we go, we see other effects that are specific to centers. But we haven't got the really systematic analysis by sequencing centers. It's definitely on the list for us to do. Yes. Oh, so during library preparation is very possible for oxidation to occur. And then there would be cannot remember the context of the mutation, but it will be very dominant. So if you look at a Lego plot is very dominant. So what can remove that artifact is something called oxo G algorithm is developed by the broad team. It's publicly available. So we apply that to remove the artifacts. But since then the broad team also wrote a protocol experimental protocol to add certain reagents into the library prep to avoid that kind of oxidation. Yeah. Yes. It's pretty much anonymized in the sense that clinical data we asked for specific terms that are not identifiable. So we're not going to we didn't ask for birthdays. For example, we will ask for age at the diagnosis. And we don't ask for postal code. So that's how I that's we piggyback on what I see GC has established. So that really helped. So the metadata is indexed into elastic search. So it's pretty much in a JSON format. It's possible. And you can do reports and queries out of it. Yeah. Yes. So when the okay, so first off is that there was a change in policy by TCGA by DB gap and also I see GC saying that okay, we allow our members to put the data in the cloud. But then for the ICGC portion, we actually had to go back to each data owner and asked if they will allow us to put it in Amazon. It was actually a very, very lengthy process because we made a request. They come back with a lot of questions. Okay, what are the implications? How save is AWS? Do they abide by these rules? Sometimes they ask for ISO certification. The lawyers have questions. They had to go through their REB again to make sure their patient do consent, this kind of environment for their data. It was a very lengthy process. And it really took people like Lincoln Stein and Bath and Noppers to and Tom Hudson to really negotiate with them. So eventually, most of them came through. There were still other projects that didn't come through China. I believe a breast cancer project that span across multiple countries in Europe. So it was a little harder to get consent. And then the German. So those are the ones that never did not consent to putting the data in the cloud. And also one thing to point out is that OICR is considered the custodian of the ICGC data. So we have a responsibility. And so we do have cyber insurance. And in order to get that cyber insurance, we actually had a threat risk assessment of our system before getting that insurance. So that involves getting a third party coming in to test the system, trying to break the system, and they do penetration testing as they call it. And eventually we pass with flying colors. All right. Thanks, everyone.


the 38th lecture in the course design and engineering of computer systems. In this lecture we are going to study more about the topic of atomicity. So what is atomicity? We have discussed this a couple of times before in the course. So atomicity is a property where if there is a sequence of operations then sometimes in computer systems you require that all of these operations they do not appear as different different steps but they appear as one atomic unit only then the system will work correctly that is you either complete all these operations or you do not do any operation at all. But for this atomic unit you cannot execute half of this atomic unit and leave the other half right no partial execution should happen. So such set of operations are called transactions and all the operations in a transaction must execute or none of them should execute. If you execute a transaction partially then you will have all sorts of errors and bugs in your system, your system will not have correctness okay. So that is atomicity the ability to execute all the operations within a transactions or if there is a failure do not execute anything at all okay. So that is called all or nothing atomicity. So what are some examples? You know we have seen this before when we were studying file systems you know for a file system system call if you are implementing there could be many changes being made to many disk blocks and all of these should happen together but you cannot do half of them and leave the other half say if you know some failure happens in the middle you cannot just do half and half. For example you know the sequence of operations are if you are writing some data to a file okay. So there is a new block with the file data then there is your inode in which you are storing a pointer to this new block. You know you are storing the block number in your inode and there is a bitmap where you are marking this block as occupied all of these changes should happen together. If you do some and you do not do some you will run into issues. For example if you add this block number to the inode but you do not fill in this data in the block then what will happen when you are reading this file you read this block you will find garbage in the file. If you you know allocate this block to this file but you do not mark the bitmap then what will happen the bitmap will think that this block is free it will allocate it to some other file and your data will get overwritten right all sorts of bad things can happen. So if a failure happens you skip all these changes do not do any of these changes at all but if you do them do all of them together okay that is atomicity. There are other examples also that we have seen you know if you are doing a checkout from an e-commerce website the billing and shipping should happen atomically you cannot say that I have charged your credit card but there was a failure I did not ship the product to you that is not acceptable. Similarly a common example is you know transferring money from you if you are transferring money from one account to the other you deduct the money here you add the money here you cannot deduct the money here and forget to add the money here that is not acceptable. So why will atomicity be violated of course you know people are not out there to you know violate atomicity on purpose but due to bugs or you know most common reason is failures you are in the middle of a sequence of steps and a failure has happened and therefore some were executed and some steps could not be executed and whatever you executed were maybe lost some changes were lost due to memory contents being flushed out things like that right due to failures is when atomicity is usually violated in computer systems and in such cases when failures happen of course we cannot avoid failures but when any failures or faults happen what we want to do is either you complete the operation or you do not do anything at all and that is also okay you say a failure happened this account transfer did not take place this checkout did not take place is okay but you cannot say failure happened I did half of it I did not do the other half of it. So in this lecture we will see what are the techniques to ensure atomicity of transactions this is also a part of fault tolerance okay. So we have studied this principle of modularity right applications are built as modular components and with modularity atomicity is very very important if you do not have atomicity then it is very hard to build modular applications why is that so suppose you have module A that is sending some request to module B right module B has some API it has a set of requests that it is exposing you can ask B to do all of these different things and A is making a request to be asking it to do a particular operation. Now when processing this request suppose B has to do multiple steps to handle this request then if B completes all of these steps then B will send back a positive reply to A saying your operation is done now we could not complete any of these steps then B will you know send a failure or no response will be returned to A that is okay then A will retry the entire request okay B does all the steps we are good if B does not do any of the steps we are good but what if we perform some of the operations I did these two steps but I didn't do the other two steps in such cases what is the response that B will return back to A it cannot return a positive response it cannot return a negative response and how will A retry there is no API request to say do half of the previous request know that does not exist. So therefore if you want correctly clean modular applications then you need atomicity if a component is exposing an API and saying I will do this job for you then you better do the job for you or do not do it at all in which case I will retry that is okay if you do not do anything I will retry if you do it well and good but if you do half of it then the other component is kind of left in a lurch doesn't know how to retry okay. So therefore when you design APIs of modules always keep this in mind you know give your APIs in such a way that you can ensure atomicity don't have an API where you are doing like 20 different things and therefore sometimes you do half of it sometimes you will not do half of it then your return values will all be messed up. So therefore when you design APIs make each service fine grained enough that you can execute it atomically. So there is also another definition of atomicity which we used previously in the course which is what is called before or after atomicity that is if you have a sequence of operations these operations should appear like one unit they should either occur completely before or completely after other operations but they should not be overlap between these operations. So sometimes you want transactions to be fully before or fully after other transactions but not you know you don't want this kind of an overlapping scenario. We have seen an example of this you know when threads access shared data for example if they access a shared counter then you have these three steps where you know you load a counter then you increment the register value then you store it back into memory and we have seen that you know if these sequence of operations overlap in a certain beard way then you will have race conditions and your counter is not updated correctly we have seen all of this when we were discussing threads. So in such cases what we said was we need atomicity either you know one thread fully finishes then the other thread starts or after this thread finishes the first thread starts but you cannot have this overlap that is also considered as atomicity but to differentiate between the two types of atomicity that is all or nothing atomicity this is before or after atomicity a transaction appears like one unit it lie there fully before another transaction or fully after another transaction and we have seen this we have studied before how do you do this if you remember you use locks. So each thread will acquire a lock do its set of operations release the lock therefore this kind of overlap cannot happen because there is a lock statement here and there is an unlock statement here therefore this thread cannot start executing in the middle. So locking and of course we have studied locking deadlocks how do you take care when you have multiple logs all of this we have studied before. So this is another definition of atomicity which is called before or after atomicity but in this lecture we will mostly focus on all or nothing atomicity. Now how do you do all or nothing atomicity the most common idea in computer systems is what is called right ahead logging or simply called logging which is all the steps that you want to do in your transaction you write them you log them onto some persistent storage and then you execute them this will guarantee atomicity how is that possible let us see in more detail. So whenever you are doing right ahead logging you know you have a transaction you have a series of actions that you want to perform atomically that is called a transaction. So you do not just directly go start executing these operations one by one you know then you would not have atomicity what if you do half of it and then you crash. So instead you will execute them in the following manner you will first start a new transaction assign a unique transaction ID then you will you know identify some log. Log is nothing but some storage space in some persistent place like a disk or a data store database something where it will stay safe even if failures occur power failures occur. So for example you cannot store your log in main memory because that can get wiped out after failure. So you will store it in some persistent storage. So you will first assign a unique ID for this transaction you will start logging and what you will do is in the pre-commit phase so this logging this right ahead logging algorithm has multiple parts. So the first part is what is called the pre-commit phase in the pre-commit phase what you will do is you will record all the operations that are part of this transaction that have to be performed as part of this transaction. You will not perform them yet the original copy of the data you will not touch but you will only note down I will do this I will do this I will do this I will do this in the pre-commit phase.  After this then you will start installing your transactions then you will start replaying or installing these transactions executing these operations one by one using your log. Now you start modifying the original application data and once all of this is done of course then your transaction is completed all the sequence of steps have been done then you will clean up your state and you will clear this log entry that is the end of your transaction. So how is this helping us what so now let us see how is this helping us to recover from failures.  Then you know that the list set of steps is not complete therefore you will not do anything to this transaction you will abort this transaction and your original data has not yet been touched in the pre-commit phase. So therefore it is as if nothing happened no change has been made you are good. What happens if there is a failure after the commit point you have still not completed all of these operations you only done some of this but there is a failure but the good thing now is you have the list of operations in your log when you restart you see this log entry you know that you know it did not finish because if it would have finished you would have cleared the log entry therefore you know that it did not complete it failed somewhere in between. So what will you do you will start replaying the log you will start executing all of these operations one by one and you will get to the end of the list and then you will clear the log entry and therefore if a failure happens after the commit point then all the operations in your transaction are complete if it happens before the commit point nothing is complete. So either way in no matter where the failure occurs you have all or nothing semantics in your application this is the key idea of write ahead logging. Now there are a couple of ways in which you can do this write ahead logging what we have described so far is what is called redo logging that is you do not touch the original data initially you will only log all the changes and then you will say commit and then you will proceed to install all of these changes. So during the recovery phase it is called roll forward recovery. During the recovery if a failure happens in between a transaction then during the recovery you are installing the transactions you are making your changes to the original data but pre-commit no changes are made to the original data and any recovery is done by redoing the operations. An alternate way is what is called undo logging that is in the pre-commit phase you will log the old value of data and you will directly make changes to the original data. So your original copy is edited but only after the old value is preserved. So now you have all the old values stored in your transaction and then you will write a commit entry. Why is this done? Because if you have changed made only some changes and you crashed in between you have not finished your transaction then when you restart you can see that you know some changes I have made but I have not made all the changes I do not see a commit point therefore the transaction is not complete. Using these old values you can undo the changes during recovery that is called roll back recovery you can roll back if your transaction has 5 steps you only did 3 steps you did not finish the last 2. So what you do you will roll back you will undo these first 3 steps also so that you have your all or nothing semantics. And how are you undoing you are able to undo because you have the old values stored in your log you are not just directly editing you are first saving the old value then editing the original data. So you can see that there is a slight difference in semantics between redo logging and undo logging but both of them achieve the same purpose in the end and you can choose which technique you want depending on your application for example undoing operations may not be easy in some applications you know you cannot say you cannot add money into a bank account and later on say oh sorry I am undoing that it may not look good. So depending on your application you can either do redo based right ahead logging or undo based right ahead logging. So let us see some examples you know so this crash consistent file system this is what we have seen before when we studied file systems. If you are using file systems in a situation where you need highly consistent entries on your disk then you will have to use some kind of logging so that even if crashes happen in between a system call your file system remains in a consistent state it will never be the case that you know you have added a pointer to a data block to an I know you did not write data into the data block so your file will have garbage such scenarios will never occur if your file system is crash consistent. So how will you do that whenever your file system updates multiple disk blocks what you will do is in the pre-commit phase you know here are all your disk blocks that are you know the I know data block bitmap all of these have to be updated as part of a system call you will first make the changes on a separate copy and you will write all of these change disk blocks to your log. You will not change the original disk blocks you will not change the original I know but you will make a copy of the I know then you will edit it here you will edit the data block here edit the bitmap here all of these changes you will make in the log then you will write a commit entry and now you will start installing you will you know copy this disk block onto its original location you will merge this copy with the original location and so on you will start installing the transaction. So that if a failure happens in the pre-commit phase then you have not touched the original values you are good if a failure happens in the post commit phase then you already have all the changes ready with you you simply go and replay them from the log in this way your system call will be completed. So right ahead logging is used to ensure that your file system is consistent in spite of a crash. Now let us look at a more complicated example suppose you have again going back to our standard example of an e-commerce system suppose you have a server that is managing all the checkout requests you know user has added some items to a shopping cart and the user says checkout then you have to you know build the user's credit card for all of these purchases and then you have to ship the order. So let us consider some such checkout server and there is an API this checkout server has a API where you give it a shopping cart it will check out that order and it will ship the order will do the billing and shipping for you. So you have your front end or client whoever is making a request to this checkout server and this checkout server will do the billing and maybe to do billing it will contact the bank or somebody else some other server it will contact to do the billing and then it will contact somebody else to do the shipping it will finish all of these operations and it will send a confirmation back to the user. This is your checkout server and now suppose let us take this example and let us see now this checkout server has to do two things atomically it has to do the billing and shipping atomically we have seen that you cannot it cannot be this case that this guy has billed your credit card but then a failure happened and it forgot you know do the shipping to you that should not happen. So how will you do this checkout atomically? So again you can use the same idea of write a head logging you know instead of directly executing the operations you know the billing and shipping what this checkout server will do is it will create some entries you know it will create a order information where it will note down all the details of the order these are the items this is the billing information this is the shipping information all this order details will be created and they will be stored somewhere in some database okay in some persistent storage somewhere this order information will be stored okay it will all be logged all the details. At this point the transaction is committed and this can be a confirmation can be sent to the user and now you start performing all the information is there you start executing them one by one okay billing shipping everything you start executing. So that even if in all of these steps somewhere in between you fail then when you restart you can resume okay you can you know all the information is there with you and you can resume okay if the server fails before the commit point of course you know the order is not even recorded in the system there is no confirmation sent to the user everything is lost and the user will retry that is okay you know the user will retry with a new order and the second time hopefully it will happen and if the server fails after the commit point you know the billing is done then all the information is there at the server either a new replica you know new passive replica became active or this server only restarted whoever is continuing this operation will have all the information in the log can resume execution of the operation and the user won't notice a failure right in this way you can use write ahead logging to first note down everything you have to do and then do it so that in between somewhere if you stop you still have a record and you can do it again properly. So here but there is a small complication you know if you are thinking through this carefully you will notice a small complication. So consider this scenario okay the server recorded all the details of the order and it starts executing the order you know it has a sequence of steps it has to do it it has to do first billing it contacted the bank it charged the credit card at this point it was just about to do the shipping it didn't do the shipping yet and at this point the server crashed and then the new server replica came up or maybe the old server restarted whatever and then it look at its log or this order is there in my log it will start executing the order again it will start replaying the order again and then it will start once again it will say billing and it will once again bill your credit card it will talk to the bank charge you again is this acceptable of course not it is not acceptable okay but you might be wondering what went wrong we did the same thing with the file system and it worked and now why is it failing so here is a complication okay so there is this property of some operation being idempotent what is idempotent operation is if you repeat the operation multiple times also no harm is done whether you execute the operation once or you execute the operation 10 times the effect will be the same such operations are called idempotent operations for example writing the value of 1 into a variable a this is an idempotent operation if I do it once a will have the value of 1 if I do it 10 times a will still have the value of 1 on the other hand incrementing the value of a this is not an idempotent operation if I do it once the value will be incremented by 1 if I do it 10 times the value will be incremented by 10 right so this is not an idempotent operation so replacing old values of the disk block with new modified disk block where we have seen in the file system you know you log all the changes and you install these changes one by one into the original disk blocks that was an idempotent operation why because you are taking the entire disk block overwriting the old disk block but charging a user's credit card that is not an idempotent operation you charge 10 times and 10 times the amount will be charged okay so some APIs in your system are not naturally idempotent okay and therefore for such APIs this replaying the operations after a restart such mechanisms is redo logging all of this won't work very well okay them so how do you fix this problem the best way is your modules implementation must somehow try to ensure that any API that you expose is idempotent for example if your banking server is exposing some API saying you know charge a credit card it is exposing some API using which you can send a request to charge a credit card then the banking server should ideally implement this API in an idempotent manner for example what it can do is it can maintain a database of all the transaction IDs that have been built recently and you know you created an order you build the back you told the banking server hey build this order charge this credit card and now after 10 seconds you come back and say build the same order then the banking server has to realize if it keeps a list of all the transactions it will realize this particular order ID of this particular user it has been built just now therefore I will not repeat it again and it will simply send you a confirmation saying done okay in this way the banking servers implementation can make this you know credit card payment and idempotent operation in which case if you come back to this scenario the server did the billing in the middle of billing it crashed then when a new replica comes up it will once again request billing but the bank will not charge twice why because this API of to the bank is idempotent so the bank will say if the previous billing did not happen then it will charge if the previous billing happened it will simply send back a confirmation saying your order has been built it would not charge it again twice so if you have idempotent APIs if you are a module and you are implementing all your requests in an idempotent manner then it is very easy to implement operations atomically it is very easy to implement atomicity of transactions so especially in a multi-tier application you know you have many different modules you know you have your client that is or a front end that is talking to this checkout server and which is talking to some banking server for payment some other shipping module for shipping all of this you have a complicated multi-tier system right at every level you should ensure idempotent operations okay so how do you do this for example your client has made a request to check out a shopping cart if the checkout server finished all the operations it sent a confirmation back but somehow this confirmation was lost for whatever reason now the client will once again come back and say check out this order then this checkout API should be idempotent this server should remember that this particular shopping cart I have just checked it out I do not know why this guy is asking me again so it will not execute the order again it will simply send back a confirmation saying done but of course if the previous request failed for whatever reason and that is why the user is retrying then the checkout server will execute the request correctly okay similarly at the bank also you know if the checkout server asked the bank to charge a credit card if the and again it makes the same request again if the previous request succeeded the bank will simply send a response back it won't re-execute it but if the previous request failed then the bank will re-execute okay so every API in the system has to be idempotent then it makes it easy to design your systems and how do you do this one way is you give some unique identifier you know when the user checks out that order that transaction has a unique identifier so that everywhere in the system you are tracking that request using this unique identifier so that when that request comes the checkout server will look oh this identifier I have already processed it I will not redo it again or I have not processed it I will execute it right this is concept is similar to TCP sequence numbers you know TCP if there is a packet loss and you know the same packet is retransmitted and somehow the receiver gets two copies of the data it will not it is not like in your file that two copies of the data you know some line in your text file or email will be repeated twice you know that will never happen why because the receiver is using the sequence numbers this is sequence number 500 another packet starting at sequence number 500 it will filter out the duplicates and it will only use one copy in the final data it is giving to the application in this way this concept of always filtering about duplicates retrying idempotent API is these are all important if you are building a large system and this really simplifies the design and implementation of your system and this idea is used in many places for example if you have RPC frameworks you know some client is making some RPC to a server and some RPC frameworks they guarantee exactly once RPC execution that is no matter what failures happen we will execute it exactly once how is this guaranteed again by you know you track this RPC request and if for some network failures you repeat the request then the server will remember ok this RPC request came just now I did it so it will not redo it again it will simply send a confirmation back saying I already done it here is the final answer in this way several places in a complex system the use of idempotent APIs is very useful so that you can do this right ahead logging you can replay your log but still get correct results in the end. So that is all we have for this lecture in this lecture we have discussed what is atomicity you know all or nothing atomicity before or after atomicity then we have seen how you do all or nothing atomicity using right ahead logging and we have also you know gone into a little bit more detail and we have seen that in real life systems when in large multi-tier systems if you are doing right ahead logging and you are replaying your operations then it is very important that all your APIs are idempotent as much as possible so that even when you replay a request it is executed exactly once within the module. So I would like you to think a little bit more about these concepts of you know atomicity and idempotent operations in your real life and other systems which operations are idempotent and how is this idempotent semantics guaranteed what are the mechanisms used to make things atomic and to make things idempotent. So think about these things a little more not just from computer systems but you know in other real life interaction system. So that is all I have for this lecture thank you very much and let us continue our discussion in the next lecture. Thanks.


I'm the Comic Weekly Man, the Jolly Comic Weekly Man. And I'm here to read the funnies to you happy boys and honnies. Yes, boys and girls, it's Comic Weekly Time.  Right out of the pages of Puck the Comic Weekly, straight into your living room, your friend, the Comic Weekly Man, the Jolly Comic Weekly Man. Little Miss Honey, how are you today? I'm just fine, thank you. Good. Tell me, do you know the story of Robin Hood? Oh, sorry. Well, we're going to begin reading the story of Robin Hood next week in the Comic Weekly. Oh, I can hardly wait. Now tell me, do you remember who Alan O'Dayle was? Oh, yes, he was Robin Hood's friend. Well, I've got a surprise for you. You're going to meet Alan O'Dayle today. I am? You are. Walt Disney has just finished making a movie of Robin Hood. And there'd be no Robin Hood without Alan O'Dayle. So I want you to meet Elton Hayes, England's best known singer of ballads, who plays Alan O'Dayle in the moving picture of Robin Hood. Elton, this is my friend, Little Miss Honey. Well, I'm hopefully happy to meet you, Little Miss Honey. Oh, and I'm in a picture of Robin Hood. Well, certainly. And I had a wonderful time making it. Oh, tell me about it, please. Well, you see, King Richard the Lionheart leaves England to lead a crusade to the holy land. While he's gone, his treacherous brother, Prince John, conspires to seize the throne and has the wicked sheriff of Nottingham to help him. Well, now, Robin Hood, who is faithful to the good King Richard, rallies around him in Sherwood Forest, a band of fearless men who defy the bad sheriff and Prince John, and help return Richard to power again as king. And the story of their adventures is an exciting and dashing one. Oh, that sounds wonderful. I like Robin Hood because he fought against the bad men. And you helped him, didn't you? Yes, as Alan O'Dayle in the picture I did. Please tell me more about Alan O'Dayle. Well, if you remember, Alan O'Dayle was a singer who also played a lute and always entertained Robin Hood and his merry band by singing delightful old English ballads to them. Yes, and whenever Robin Hood was tired and sad, Alan O'Dayle would cheer him up by singing to him. He would indeed. How well you remember. Oh, she remembers all right. You know, Elton, I'm sure she'd like to hear something about the songs that you sing in the movie. Well, there's one that's called Riddle the Diddle the Day, which I know you'll love. Oh, would you tell me about it, please? Well, I'll do better than that. I brought you a recording I made of it. Oh, thank you. Could we play it right now? Well, yes, if you wish. Oh, goodie. And here for everyone to hear is Riddle the Diddle the Day sung by Alan O'Dayle in Walt Disney's movie Robin Hood. Good and true. Well, we fight to gain the right. And so I pray, may you, oh, I'll sing a song. A roll of key-songers are rolled along my way with a hater, a re-die, and a derried item, and a riddle in Riddle the Day. Oh, that's a lovely song. Yes, I like to do it. Well, and I tell you what, you can keep this record. Oh, really? Thank you. Not at all. And I know you'll have much pleasure in reading Robin Hood in Puck the Comic Weekly. I'm sure we will, because the artwork will be in those wonderful Walt Disney drawings. And that should make everyone happy. Well, Elton Hayes, you've certainly made us happy by coming to tell us about Robin Hood. Yes, and thank you very much for the record. You're welcome. And I'll be listening when you tell Robin Hood's story next week on the Comic Weekly. Well, thank you. Good bye, Alan, at least. Puck the Comic Weekly? Very well, I will in just a moment. But before I do, let's listen to this nice man. Now, here we go with Puck the Comic Weekly. And on the first page, Big Ben Bold. Magic words for the music, please. Very well, my lady. Paint and punch and dodge and twist. It's a knockout blow from Big Ben's fist. Ben sets out on a tour of the country with Spider, his manager, to box the local champion of different towns. After several one-night stands, they arrive in Beaver Rapids, where Ben is to fight the local favorite. Ben climbs into the ring, looks over the crowd, assembled, and sees businessmen, cattlemen, car-punchers, and some Indians. And then, a man in a 10-gallon hat wearing cowboy boots makes an announcement to the crowd. Hey, nobody's collected off of him yet. Last picture top row, a tall, husky Indian walks toward the ring. Hey, you show him, Johnny! First picture bottom row, the Indian climbs into the ring and begins to remove his head rest. Spider, Ben's manager, says, now, Chief, listen to me. You know a fighter with Peter Tomahawk, catch him? The Indian replies, cut the corn, little man. I learned enough about box fighting as intercollegiate heavyweight champ to make your boy wish he stood in bed. You catch him? Spider stares in surprise at the educated Indian as he goes to his corner. Then he goes to Ben's corner and says to Ben, Hey, Ben, Mr. Hanema was no Simon pure kid, so don't take no foolish chances. As the fighters go to the center of the ring to get their instruction from the referee, another Indian appears beside Spider and says, You see, Johnny, tall, pine, wind, fair fight, Mr. Spider replies, ah, Peter's sitting bull, I don't need no directions. Suddenly Spider feels a knife pressing against his back.  Johnny, better window. He wants a fair fight when he sticks a knife in Spider's back. There's nothing fair about that. No, and Ben is in a strange town surrounded by all those rough strange men. You think the other people will see that Ben is treated fairly? Well, that's something we'll find out next week. Now? Oh, now to page three, because I know Prince Valiant will be there. Right, over we go. And you are right as usual. Here is Prince Valiant on page three. And last week, remember, he was trying to get all the evidence that that Sigurd Holm was a tyrant that was cruel to his people. So he disguised himself as one of Sigurd's men and slipped into the castle after dark. And then he didn't. Yes, he has the evidence. So let's read and find out what happens next with Prince Valiant in the days of King Arthur. Eckert, Brickett, Gray Malkin and Quince. Music romantic for a fair, fair prince. Daylight comes.  He sits in the dark sleeping stall while slaves clean away the debris of last night's big party. Finally, Val parts the hangings and beckons one of the servants over. He must now risk treachery. The servant comes over and Val whispers to him, I am your prince. I've come to free you from slavery. Will you help me? Oh, yes, sir. Even though it means my life if I am caught. Quickly, Val changes clothes with the servant. And then last picture top row, dressed in the servant's rags, he sweeps up an armful of straw and shielding his face with it makes his way out of the dining hall. He hides in the stable during the day and after dark that night he takes all the rope he can find and second picture bottom row makes his way to the platform overhanging a deep abyss. Then quickly he fastens his rope to the logs of the platform and then last picture slowly he descends. Then he comes to the end of his rope. Below, the water roars madden. If he drops, he will fall into the rushing waters yet he cannot return to the castle and there in the dark, he hangs. Did he drop down to the... It looks like a long fall but if he goes back to the castle, he'd be caught. Well, I wonder what he'll do. Well, next week I'm sure we'll find out. Now, would you like to see what's happening to Hopalong Cassidy? Oh, yes, please. Very well, go to the very last page of the first section. All right. Did Hopi capture Seldon, that man who was trying to run Hopi and his friends Buck and Rose Peters out of the country? And Seldon told Hopi that the man who was behind all of the trouble was Judge Padgett of the town of Rimfire. And now could he inquire? We certainly can. Here we go with Hopalong Cassidy. Six guns blazing as he thunders along. Give us music for Hopalong. Hopi, lucky in California, is saddling up to Leeds. And Hopi is saying to Buck Peters, well, thanks to Seldon's confession, justice is finally caught up with Judge Padgett. The rock and W belongs to you and Rose. Free and clear now, Buck.  Rose Peters tells Hopi as far as she's concerned, troubles are just beginning because it'll take a lot of fixing to put this ranch in working order. California says, well, don't look at me. I got to be getting back to the bar 20 with Hopi and the boys. Suddenly Lucky says, hey, look, Proudly a horseman headed this way. Buck exclaims, oh, no, what? Last picture top row of the riders ran up in front of Buck. One of them says, you own this bread? Well, yeah. And I got the deed to prove it. First picture bottom row of the man says, well, now and up. We're taking you into town, Peters. Buck reaches for his gun. Hopi says, no, no, no, no. Stow the gun, Buck. We'll go along with you till this matter's cleared up. Short time later, third picture, bottom row. Buck is brought into the sheriff's office and shoved into a chair. He stares at the circle of men grimly facing him. Then he says, what's this all about? One of the men says sternly, or empire owes you something besides a treatment you received at his newest citizen. And since the town needs a sheriff, we just held a special election. And you won. Ha, ha, ha. Buck sees the faces of everybody in the room breaking through with rins, and he exclaims last picture. Why, you mean I'm sheriff? Where's Hopi? I gotta tell Hopi. Oh, he already knows about it, Buck. Cassidy's agenda gave us the idea. You won't catch him now. He said to tell you he's on his way back to the bar 20. Ha, ha, ha. Oh, that was a good joke on him. Ha, ha, ha, ha. Buck's going to be the sheriff, and I know everything will be honest and fair. Yes, that was quite a joke those men play, Don Buck. Yes, he has these. I think it's wonderful that he had a man like Hopi. Yes, Hopi's troubles are over, and so are Buck's. Now it's time to pick up the first page of the second section of Puck the Comic Weekly. And I know, Sheriff. And I'll read Dagwit and Blondie in just a moment, but first here's that nice man again with something interesting to say. Now here we go again with Puck the Comic Weekly, and on the first page of the second section, Dagwit and Blondie. Magic words for the music, please. Very well, my lady. Ramefou, Ramefum, Zim, Zim, Zambi. Can you give me music for Dagwit and Blondie? Today, Herb Woodley is home alone while his wife Tootsie's away visiting. Herb says to himself, oh, it's lonesome with Tootsie away. I'll phone Dagwood and see if he wants to play trackers. And a moment later, Dagwit, who's outside Herb's window, hears Herb calling Blondie on the phone. Oh, hello, Blondie. May I speak to Dagwit? And Blondie, last picture top row on the phone in her house says, I think Dagwood is out in the yard. Hold the phone, Herb, and I'll get him. Oh, thank you. First picture, second row. Blondie dashes out of the house and says to the children. We've got to find Daddy. He's wondering the phone. At this moment, Dagwit, who had been out on the lawn and heard Herb ask for him, comes into Herb's house and says, oh, I heard you calling me Herb. Well, what did you want? Oh, Dagwood, hold the phone for me. Well, your Blondie's out looking for you. I've got to find her and tell her you're here. Herb dashes all about, but can't find Blondie anywhere. Last picture, second row. Dagwood sits in Herb's house. The phone in his hand soundlessly. First picture, third row, Herb sees Dagwood's daughter Cookie, and he yells, oh, Cookie, where's your mama? Well, she's running all over like mad, trying to find Daddy. At this moment, Blondie is in the house at the phone again. At the other end of the line, Dagwood wakes up, hearing the voice, and the phone says brightly, Herb is not here. This is Dagwood Bumstead. I will give him your message when he returns. First picture, bottom row, Herb dashes into the Bumstead house. Oh, Herb's house. Yeah, I know it. I know it. Give me the phone. I want to talk to him. Blondie hands Herb the phone. Hello, Dagwood. This is Herb. Dagwood on the phone at Herb's house says, yeah, what did you want me for, Herb? Suddenly, a black look comes over Herb's face. Oh, gosh, now I forgot why I phoned you. Blondie looks very surprised as Herb walks out of the house. He meets Dagwood coming out of his house. And as they pass each other, Herb says, last picture, if I think of it, I'll call you back. OK, Herb. And Blondie goes. Is that really was funny? Herb not being able to remember what he called Dagwood about in the first place. I was too, uh, uh-oh. I was so busy laughing, I forgot what it was myself. So did I. But I haven't forgotten Roy Rogers. He's right below Dagwood and Blondie. Oh, and I was pretending to be Teddy Knox, and that's the son of the owner of the Knox Ranch. But Roy and Corny discovered that this was somebody in disguise. So Roy quickly overcame the boy, and they put him in a covered wagon. But the boy. And then Roy rode off to the ranch to investigate, leaving the boy with Corny Maxie. I wonder what he'll find out. Well, let's read and see. Here we go with Roy Rogers, king of the cowboy. Hi-yip-yo, now here we go with Roy and Trigger. Hi-yip-yo.  But the trick doesn't fool Corny. Third picture top row, he turns around and says, Teddy Knox's notes inside that locket, and he said that he's held captive at the Box K Ranch.   Suddenly swings the broom. Knox corny out. Now I got to do something to stop Rogers before he finds the real Teddy Knox. Meanwhile, first picture bottom row, Roy arrives at the Box K Ranch. He sees a man in the porch. He rains in and says, howdy. Like to see Mr. Knox about a personal matter. I'm Roy Rogers. The man answers. Lawyer cost as at your service. I regret to say Mr. Knox died last month. His aunt Teddy now owns the Box K. But he isn't here. May I help? Roy takes out the gold locket and shows it to the man. Well, this gold locket was tied to a Box K. steer. Any idea how it got there? The lawyer looks at the locket. Then he sees the boy riding up. This mounts at one of the buildings a short distance away. The lawyer stalls for a time saying, um, locket, huh? Well, he sees the boy with a shovel begin to sneak up behind Roy. And he says, very slowly, last picture, uh, frankly, the locket once belonged to Teddy's dead mother, Rogers. But, uh, this information do no good. Now, Roy says, huh? Well, why not? The boy who's behind Roy licks the shovel to knock Roy out and says to himself, well, you will know in a second, cowboy. This Roy would turn around and see what's going on behind him. So do I, because these men are up to something crooked. Do you think that boy will knock Roy out? Well, that's something we'll have to wait until next week to discover. Now let's go over the page, because I know you're anxious to read Flash Gordon. Yes, I am, because Flash is on the planet Rhea, and he's been fighting the Giants. And remember that Sami had conquered the king of the Giants and then challenged Flash to a hand-to-hand combat, promising that the winners should rule the planet. And last, he moved toward Flash angrily, and I'm anxious to see whether they'll keep their word. Let's read now and find out. Here we go with Flash Gordon. Regga regga dune dune, Sasuke Matash. Let's have music for heroic Flash. Sami lies on the ground, Flash kneeling over him, the victor. The Giant Reans begin to close in on Flash. But quickly, Kara, Sami's sister, bars the way. She tells him that Sami had deposed the king and now has bested Sami by combat. And by their real law, Flash should be king.  I have no desire to rule your planet. If I am king, I hereby abdicate. And then rising, he shouts, Long live! And for a moment to re-instand in sun's surprise, first makes her bottom roll. With a fickle enthusiasm of a mob, they take up Flash's cry. Lifting Kara to their shoulders, they bear her to the palace in triumph. As her first official act following the coronation, Kara summons Flash. She tells him that she is queen through his kindness, that he should name his reward. Flash smiles and says, all I ask is peace between Rhea and Earth. Not only does Kara grant Flash's wish, but she orders his rocket ship loaded with Rhea's most advanced scientific marvels. After brief farewell ceremonies, Zarkov takes the controls. And last picture, the spaceship blasts off for the perilous trip back to Earth. But it's a dangerous trip back to Earth. You think you'll get back safe? Well, that's something we'll find out next week. Now let's go to the very last page, because I know you'd like to read Dick's adventures. Oh, you know I will. As they founded a town and named it Seattle after a friendly Indian chief. And last week, with the Indians, we're next in. Well, let's read now and find out. Say the magic words with me. Rikki, Rack, Rack, Rack, Rack. That's some music for adventurous Dick. Dick and old Seattle listen carefully. Sure enough, off in the distance comes the sound of drums. Last picture, top row, old cheap Seattle cries. Look, the hot fires of vengeance, the black smoke of death. My hot tempered young warriors are coming to fight their last battle. But they are doomed. The pale faces cannot be beat, even though they die. Quickly, Seattle tells Dick to alert the settlers. And he hurries to meet the warriors, his own people, to try and stop them and turn them back. They refuse to listen to Seattle, but rush past it. Big picture, second row, to burn and massacre the pale faces of the village. Sadly, the old chief, first picture, bottom row, awaits their return. Late that night, the Indians file past them. Defeated and broken in spirit. The pale faces have won. Next morning, as the white men are at their work, one of Dick's companions looks out to sea and sees two ships approaching. Dick exclaims, well, now we can roll up our sleeves and get back to work. You see, those ships are coming in. They all want lumber. There's plenty of work for everybody. Sure, the Indians too. Where are they going to grow? That'll be because it's much better for people to get along together. Yes, and as the man says to Dick, there's work for everybody, the Indians too. Will they have peace now? Well, I hope so, but you can't be too sure. We'll find out more about that next week. Now look below Dick's adventures as Rusty Riley. Oh, I've been so much on board that ship, we're all right, because you remember that mean man, Blacky Kirk, won the sink this ship so the horses would be drowned so that he could collect the money on the insurance. Yes, and Mr. Murray, you're right. Yes, and Mr. Miles had sold the horses and they were being delivered by ship to Florida by Rusty and some others. And when a storm came up, the captain of the ship ordered all hands to take to the lifeboat. And it looked like everybody on the ship got to shore all right, but I'm worried about the horse. Well, let's find out what happens next. Here we go with Rusty Riley. Gallop and run till the road is dusty. Give us music for his horse and Rusty. On the island, Rusty says to clam the old sailor they've been friendly with. He clammed by the time the tide's all out. I believe we can practically go walk out to the ship and maybe get the horses off. Yeah, you might be right, Rusty, but that tub don't look like it's damaged none. She's only around. A couple of good sea tugs will pull her off as soon as the skipper notifies the coast guard. Tech says, well, that might be true, Clem, if the skipper notifies them. Well, I don't know. So far, I can't figure why, but I don't think the captain and Kirk want help, because if they did, why did they put the ship's radio out of commission? Pete suggests that they ought to search the island and see what it's like. Last picture, top row, Tech says, yeah, you're right, Pete. Clem and I'll take the shoreline. You young steers explore the interior. You're better pit it for scrambling up them rocks. Find this plenty of grass and fresh water on the island. Good place for the horses. And then, second picture, bottom row. Pete and Rusty discover a man lying on the beach. Collie, Pete, this is ship's radio man, the one they call Sparks. Is he dead? I don't think so, Rusty. No, no, he's in bad shape, but he's alive. [?]. Tech's is worried about the horses, and he and Clem decide to get to work to get them off the ship while the tide is low. But meanwhile, last picture, in the ship's launch, many miles down the coast, the captain of the ship and Blackie have been cruising around, hoping that the ship will be pounded to pieces and be sunk. The captain says, hey, we're nearly out of gas, Blackie, haven't got too much drinking water. Blackie replies, OK, crump, start looking for a place to put in. By this time, that tub must be smashed to scrap by none of those rocks. So do I. I don't think there's anything worse than being cruel to animals that can't protect themselves. Neither do you get the horses off the ship before the tide comes in again. Do you think they will? Well, we'll find out next week. Now, that's all the time I have. But before I go, don't forget, next week is the week we'll begin reading Robin Hood, Walt Disney's Robin Hood. Now here's that nice fellow with some more interesting information. Hey, and all you boys and girls, I've got to go now. All right. OK, that's a date.  Be sure to meet me with our little friend Miss Honey next week when I read Pop the Comic Weekly. For I'm the comic weekly man, the jolly comic weekly man. I'll be back to read the funnies to you happy boys and honeys. Don't forget, boys and girls, see you all next week. Your friend the comic weekly man, the jolly comic weekly man.


Everybody forced here ripping a box of 2022 NT collegiate football for James F. Here we go, James. Let's see a nasty bullpatch auto All the sketch of fear factor was great all of his sketches were great It's just a five o'clock free crack giveaway They did the intervention on them. They're like, no, he'll be here any second three two one. It's five o'clock.  Oh, yeah Another one of those cards James cook this time 43 of 99 college silhouette signatures Oh when he's in the bugs So good. Oh, we got another college silhouette signatures David Bell 8 of 99 When he's walking on the coals Your factor likes a cigarette with his toe. Oh My god a Jersey card. I'm delirious, dude CD lamb number two out of ten Day five. I'm delirious patch card CD lamb. It's like the paint from the one of the numbers on the Jersey Dual patch. Let's go Knowles cam acres Dalvin cook 20 of 25 Team dual materials RPA 249 Damian Pierce has huge auto here Will sloppy a nice Damian Pierce though and RPA 3 of 25 Garrett Wilson Nice RPA's on the back for you James. Thank you again Three color Wilson there we will get these out to you