The only memory is that guy, and it's about to happen. Oh, here we go. Yikes. All right, so we begin with introductions. Please. We'll start on your end there. My name is Christopher Ado. I work for IBM. I'm super lucky I get to work from home almost all the time. I live in Portland. I'm PTL of the Community App Catalog Project, so I spend a lot of time with that, and Def Core, and Ref Stack, and some other things. And then every chance I get, I try to help or learn a little bit more about OpenStack Infra. So I was really glad that I had the chance to do this with these guys. So my name is Paul Boulanger. I talk really loud. And I work for Red Hat. And like Chris, I'm lucky enough to work from home in Ottawa, Ontario, Canada. Recently became a InfraCore member for the infrastructure project. And like Chris says, it's just an awesome project to work with and help OpenStack developers make OpenStack better. Thanks. My name is Elizabeth K. Joseph. I work for Hewlett Packard Enterprise. Just like the two of them, I work from home. I'm based out of San Francisco, California. I'm also working upstream on the OpenStack infrastructure project as a core member. And I'm happy to be here today to talk about some of the internals of how we make everything work with the OpenStack development workflow. So we're going to get started. I probably don't need to explain what OpenStack is to this audience. So I'll just move on to quickly talk about some of the projects that OpenStack encompasses. And again, you probably know. In the OpenStack infrastructure team, it's our job to support all of these projects that come into OpenStack infrastructure. So these are some of the early projects. We've since moved into Big Tent. So we're supporting a lot more projects. That link there gives you a link to the official or the listing of projects that have project teams. The infrastructure team is also responsible for supporting other parts of the project, not just the code teams, but also projects like the documentation team in OpenStack, the quality assurance teams, release management, translations, and internationalization. Pretty much anything that has to do with the components that ship with OpenStack or the surrounding supporting documents in the OpenStack project are things that we support. You may be familiar with the OpenStack release model. It releases every six months, much like Ubuntu and Fedora, sort of what they're based off of. The development is all done on the master branch of the OpenStack projects. Sometimes projects will have feature branches, but we won't really talk about that very much. And then after the stable release, there are other stable release branches that are maintained that we have to pay attention to in our infrastructure. Contributors, again, from OpenStack, all over a bunch of different companies. This is a screenshot from Stacklytics of the contributors during the Metaka cycle. We work with companies, organizations, non-profits, government, local and national governments, all kinds of different types of contributors. So we need to support a unified workflow for them inside of the OpenStack infrastructure. We also have to be able to scale the infrastructure because the number of contributors to OpenStack has been growing very fast. In order to do this, we provide a consistent tooling for all the OpenStack projects. So that means we don't want people using different code review systems. We don't want people using different vision control, different tooling. We all sort of tell everyone to use Garret, which is our code review system. We want them all to use Git. And we want all of these changes to go into the same place and be centralized. So we don't want any sort of projects, but specific weirdness being pulled up based on any of these, any different changes in our tooling.  We'll dive into some pieces of this as we go on. But the developer infrastructure for OpenStack pretty much has all these components. So you interface with a code review system using Git review, which is submit your changes and then you review through the web interface. We'll dig really deep into the test and build automation here. But we also do things like host a bunch of repository mirrors for various things that we need to test against. We have massive log servers to host both static logs and searchable logs. And then we run all the other things that developers need. We run the bots on chat. We run the mailing lists, the paste bin for the OpenStack project, all the ether pads, things that developers use for collaboration. As a team, we also do all of our systems administration in Git. So the OpenStack infrastructure team itself uses the code review system. So we use Puppet and we put the Puppet config files into Git. We do code review. We do testing against those. And so all of our changes are also online and open source just like the rest of the OpenStack project. Just like anything else, anyone can sign up for our code review account. Anyone can look at our changes. Anyone can propose changes to our team projects. And then there's a link right here to the system config repository, which is sort of the central repository of the infrastructure team that will define all of the servers that we've launched in the project, as well as a bunch of configuration things. The development environment for OpenStack itself, as you might be aware, people are using Python. So we're supporting the testing of Python software in our development environment. Developers get in touch with each other in IRC. There's an OpenStack dev channel. There's also several meeting channels and then projects tend to have their own channels. Developers use DevStack to test their changes against oftentimes. So we're making sure that that's, along with the QA team, making sure that works and is working inside of our infrastructure to test changes. As we'll talk about the code that people submit to the OpenStack project is all tested when the code is submitted. So it's not, after it's merged, it's not any other time. It's when people submit changes and it's approved that the code is tested. And we make sure all code is tested before it hits the repository because, again, we have this diverse group of organizations and companies and governments and other people using it. So we want anyone to be able to pull down the master branch of any OpenStack project and be able to make sure, be assured that it's been tested. We believe this ensures code quality. It protects developers from each other. No one can intentionally break the master branch, really, because they have to have passed code testing. And the same process is pretty much for everyone. So there's no one person in the project who can bypass testing and land their code. Everything in this is automated. I can scroll through this slide a little bit. The tests come into our check queue when the software's uploaded, the patch is uploaded. After it's been approved by reviewers, it goes into the gate queue, which is the one in the middle here. And then after it's passed its test has been merged, there's a post queue for any tasks that need to be done. So in the case of documentation, that may mean updating the documentation on the website. And all of this is automated. As soon as you upload it, it automatically happens. And the infrastructure team's job is to make sure this automation doesn't break and add more as the time goes on. And then to sort of set the stage for the rest of the presentation here, this is our CI workflow, the continuous integration workflow, from I guess from sort of my perspective as someone who's works on the project. And well, this will be important as we discuss the components. So you have your little computer down there. That's where you write your patch. You submit it up to Garrett code review. From the code review, it gets sent over to Zool, which manages, it sort of manages making sure that all patches are appropriately tested against each other. So you don't have one patch coming in and then another one that conflicts with it, causing trouble in the master branch when it lands. Zool makes sure all the dependencies are properly tested against each other. Zool passes things off to something called gearman, which will then pass it off to one of the Jenkins masters we have. Jenkins tests all the code by passing it off to a fleet of servers in what we call node pool, which is a bunch of donated instances from various open stack providers. Once it passes or fails those tests, it all gets sent back down the pipeline and you end up back in Garrett. That's when a comment will be left in Garrett, explaining what happened with the test, past failed and it'll leave a little check mark or a minus one depending on whether past or failed. At that point, human reviewers go into Garrett and they do their code reviews. Once they approve it, it gets sent back up this whole pipeline for testing just one last time, because the patch could have been sitting there for two days or two months. So we wanna make sure it's still passing all the tests that much time later. Once it passed, it comes back down, gets merged to get repository and then you have your code landed. So just keep this in mind as we go through different components so you can sort of think about where you are. I'm gonna pass this on to Chris now to sort of talk about this from a developer perspective. Thank you very much. That was very informative. Question, how many people can I get a show of hands have landed a patch in OpenStack in the room? Okay, not too many and that's cool. So this is probably the exact audience we were expecting. So this process flow you will have seen if you have looked at the developer workflow document which is where we point everyone who wants to start contributing to OpenStack so that they have an overview of what the process is from the developer side of things where you probably don't need to know as much about what's happening on the back end. Like Elizabeth said, everyone starts their patch by cloning from master. Then you'll see the box for your local environment. You've got now a clone of whatever project you're planning to submit a patch against.  So running, using talks, assuming those pass and you hit get commit and it sends it up to Garrett and the review server where it will go through the first round, the first pipeline that Elizabeth was talking about of the gate checks. And if it fails, you'll see I think in the next slide I will show you what it looks like but you'll get the immediate feedback that something's wrong or semi immediate depending on how backlogged the gate is. And you have an opportunity then to make some other changes and amend your original commit and keep working on it until it passes and it's ready for review by other people. And then once it gets approved, it goes through the last level of testing through the, what's the last gate before it gets merged? Check and then gate. Okay, so it goes through check and then gate and then it gets merged back into master. So Garrett was, like Elizabeth was saying, this is one of the common tools that we require everyone to use for the sake of sanity and in order to allow OpenSec to continue taking as many patches and contributors as it does. It was originally developed by Google for Android. It's a standalone patch review system. Has lots of different integration points and you can categorize the reviews that are going in there and everything can be accessed via REST API. Most, I'm not actually sure what the numbers are but I think most people use Garrett through the web UI but there's also a project or a program called GERDI. Is that written by Mr. Blair? Written by James Blair, who's in the audience today which can also use all this on the command line in the terminal but we won't talk about that right now. So this is the, what a review, what a patch looks like. How many people have reviewed patches? Sure, hands a few, okay. So this is what you see. You've got the commit message and some extra information like closes bug and we'll talk about how that is integrated with Launchpad so Garrett can also update blueprints and bug track, the bugs in Launchpad shows you who authored it and then on the right hand side you'll see the responses from the different CIs for this particular one and this one has two patches and you'll see all the way down detailed responses and code review comments from different people and also the responses from the different CI systems. This is also, if you, sorry, back here, if we clicked on the API PY, we can actually see the code and see a side by side diff of the code itself and oops, sorry. You can add line by line comments, so if you were reviewing the code and you wanna point out either a mistake or a way something could be done better, this is where you'll add comments to it as a reviewer and obviously see those comments if it's your code. This is integrated with Launchpad. Like I was saying in your commit message you can say fixes, closes bug, you can say implement blueprint, you can say partially closes bug, so in the commit message if you specify that your patch relates to a specific bug on Launchpad, the tool will be integrated, so you'll end up seeing the link to the related patch and then eventually if the patch actually closes the bug and the patch gets merged, Launchpad will be updated and show that that bug has been closed because the code was released. If you are just looking through the review as you can see what is to the right, it doesn't show the last column. Maybe you can't, this is terrible, no? No. You can imagine to the right of this, oh wait, yeah, perfect, thank you. So there's extra columns at the end. Those three columns indicate that a patch has been verified by the automated by Jenkins. We'll have a check mark if it's been indicating that it's been reviewed positively and then the last column is the if it's been approved for workflow. So the patch two under the one that's highlighted shows a patch that has been approved, reviewed and approved for workflow but that one hasn't actually passed the gate test yet. So the first column is the one you'll see the plus one or you can see with the change sale options to use dashes that's got a minus one. One important tool that everyone uses is Git review. This was written by, was it by someone in OpenStack? Maybe, I think it was, yeah. So this is another OpenStack tool and it allows you to very easily review changes still through the Garrett workflow. Can use Git review to review other people's changes too so you can go into a project and if someone has, like for instance you wanted to see, which was this one, yeah. So this one has changed number 308264. If you did Git review dash D in that number, it will check out that branch in that particular patch of that person so you can very quickly and easily duplicate or get their code to try to replicate and test whatever they're working on. I get review is really easy to use. There's not too much more to say about it. And then obviously all the usage details are in the reviewer workflow, workbook thing. So the two types of tests that we rely on are unit tests and integration tests. The unit tests are what I was talking about earlier like using talks that you can run in a virtual environment and you probably should be running them before you submit your patch for review. Make sure everything passes rather than make the gate give you that feedback. But then the integration tests get much more complicated and this is where it's touching sometimes third party CI or lots of other really complicated bits and testing your patch in one project against all of the other interrelated projects that it touches. And so the intention here is to test the effect of merging that change and in a minute, I think maybe one or two more slides and Paul is gonna talk in much more detail about that. I'm trying to talk kinda quick too so we'll have time for some questions and answers too. And I think I'll hand it over to you because the Git Prep and Zool Cloner stuff. Sure, if you're much better at it than I am. Thank you. So ultimately this is the tooling that we use in Jenkins to get the branch of Git into the CI pipeline. So initially we had basically a batch script called Git Prep which did some clever things with Git to grab the target branch from Garrett and make sure it was cleanly checked out onto the remote node and then actually merged said branch against master. And ultimately if any of those steps failed then the patch was minus one and it would be the responsibility of the developer to rebase and so on and so forth. The newer approach is using Zool Cloner and we'll kinda get into that a little bit further down the road but ultimately Zool which is the heart of everything has a little command line client that still leverages Git but a lot of this functionality of testing a merge before it gets pushed to the CI node is handled by Zool locally. And ultimately it's just a faster way to scale out Zool because ultimately we found that we were running into bottlenecks in the gate for this process. So I think we're up to about seven or eight servers of Zool that just deals with merging code because there's so much code that's going through the pipeline at a given time. Okay, so to kind of expand on our integration testing in the beginning we had DevStack and DevStack gate. Ultimately that is a project in OpenStack that is primarily driven by Bash to deploy OpenStack, the OpenStack projects onto a fresh node and DevStack gate is the job that is responsible for that and like I said, it node pool will boot a pristine image, a fresh server. DevStack runs to install the OpenStack components that are needed and then we run integration tests atop of that such as Tempest to validate that the Nova is working appropriately. I wanted to add in here Puppet, so another team Puppet OpenStack or OpenStack Puppet Integration project works almost exactly like DevStack except it's leveraging the OpenStack Puppet project Puppet modules to do all of this deploying. So it just adds another layer of real world testing in the pipeline and because Puppet is, the Puppet modules are quite well maintained and that's what operators tend to use. It just adds, like I said, an additional level of real world testing. Okay, so Zool, I mean, we could do a whole presentation just on Zool but unfortunately we only have about 19 minutes to get through it. Zool, again, another project that was created by James Blair, very, if you get a chance to talk to him, he's in the back, very smart man. So the purpose of Zool gets its name from Ghostbusters, basically it's the gatekeeper and you can't get through the gate until your code is properly tested and it interfaces with Garrett and Jenkins and ultimately it allows for very flexible pipelines through your actual CI workflow. We're gonna kinda, we being OpenStack kind of really have a three phase pipeline where we talk about a check which is when a patch is uploaded. It runs a first series of patches. Once it's approved it then goes into the gate pipeline where it may do the same testing or maybe a thin down version of testing and then ultimately we have a post pipeline where once the code is actually committed like Elizabeth was saying, maybe you need to generate tar balls to be uploaded somewhere because you wanna check out a tar ball of the static or the contents of the Git repo. Bottlenecking, we're not gonna really dive into this because a lot of the changes, sorry, a lot of bottlenecks in Zool are being dealt with. I mean, Zool in itself is a project in its own. We're working on a version 2.5 and even a version three which like I said, it's a fantastic way of how we're enhancing Zool to deal with all these projects. And I don't think Zool would exist today because of all the CI problems that we've had in OpenStack and a lot of it is just becoming with, we couldn't use Garrett to merge all the code because we were pinning the CPU in Garrett because we had so many projects. So Zool just externalizes that and makes it fantastic. Okay, so this is ultimately a simulation of Zool and hopefully it'll work. I just have to click, I think. Okay, so in this scenario, so we have three basically patches that have been uploaded to Garrett and Zool receives these patches. So basically Zool then initiates the process to build these patches on the remote CI by telling Jenkins to launch them on remote nodes that are managed by node pool, across multi-clouds. So in this example, patch one, two worked great, patch three and ultimately four failed. So one of the cool things that Zool will do is it recognizes that patch three is a problem. Zool will take that out of the path. It will then re-attempt to run patch four and ultimately patch four will then run again and if patch four is successful, then now the pipeline is patch one, two and four and three has been rejected. So patch one becomes green, gets merged into Nova, patch two becomes green, gets merged into Nova, patch three has failed, is returned back to the developer to tell them to fix their patch, maybe they got a formatting issue, patch four has been successful and is landed into Nova. So you can see this process really allows developers to almost work in silos and not be dependent on other developers' problems. Zool takes a lot of that headache away and allows our patches in our pipeline to move forward at a rapid pace. So this is kind of an example of Zool pipeline. This is our gate and I won't get into all the sub triggers and so on, but ultimately what this means is in our pipelines, the gate pipeline is considered the highest. So when test resources come online from node pool, this pipeline gets the access to those nodes first and ultimately what it does is it will listen for a trigger from the Garrett event system and ultimately if it's successful, it ultimately, sorry, if it's successful, it says a plus two, which tells Garrett to go ahead and merge the code. If it fails for whatever reason, it tells Garrett it's a minus two and the code is not merged. And in a case of a failure, code, once the developer re-uploads their change, it doesn't go back into the gate, it has to start the process again. So it hits the check pipeline, goes through that, if it's successful, then gets promoted into the gate pipeline and then if it pass, it is allowed to land. So this is kind of an example of how Zool configuration looks like. This is Nova and ultimately it's showing four pipelines. So in the check pipeline, which is when somebody uploads a patch, we do a pep eight, which is basically a formatting and linting, we make sure that the code is formatted across a standard use case. This one is showing that we're testing against Python 2.6 and Python 2.7, and then we're also testing against Tempest DevStack. And you can see basically the check gate and the check in the gate are the exact same tests in this example. Most projects do not have the same amount of tests. Usually if they're a non-voting, meaning you're running kind of like some experimental testing that it's not stable yet, it would be considered non-voting. It would still run in the check queue to give the developer feedback, but you would not run a non-voting job in the gate simply because it wastes resources from infrastructure's point of view. And then from the post pipeline, this is what happens when the code has actually been merged.  We update some documentation which goes to docs.opensack.org and we do some translation updates. And finally, when a release happens, which is the tagging process in Git, we want to run a tar ball and we want to update documentation.  So templates, yeah, ultimately, there's tons of templates and they're very interchangeable among projects. Everything is managed through Git. So like Elizabeth and Chris were saying, the infrastructure team heavily relies on this process to manage all of our tooling. And a lot of time is spent going through patches that projects are submitting into infrastructure to change job definitions and so on. So from infrastructure's point of view, we don't tell projects what to run. Well, we don't tell them how they run. We expose the tooling that allows them to run and they have a lot of flexibility of how said jobs work. Basically, we have another tool called Jenkins Job Builder which takes YAML file and publishes it and we take YAML file which gets pushed into each local Jenkins master. So again, there's no manual touching of the Jenkins interface to configure all these jobs. It's all automated in an automated pipeline. So again, here's kind of an example. I'm gonna try and go through this a little bit quickly. But ultimately, this is a template for our PEP 8 which is our linting process. You can see, if anybody's familiar with Jenkins, these are kind of what Jenkins expects. So in Jenkins, you have a builder process. The first one is just ultimately a very simple no op that tells us which template was being used more for debugging purposes. Zool Git prep upper constraints is more related to PEP to limit which versions of libraries a job is installed. And then we do some install disher packages which is another project of ours called Bindep which allows projects to tell what OS libraries to be installed outside of Python. So say you need Firefox for example, for whatever reason. And then we revoke sudo which means that the job doesn't necessarily need root credentials. And then we run PEP 8 which is basically a talks macro. If you type talks, PEP 8, it'll then run. When it's finished, publisher basically takes all the console logs, takes those logs out of Jenkins, pushes them up to logs.openstack.org and allows developers to review the changes. And ultimately, we run this on a Ubuntu trusty node. So from a template, you can have a group of templates and you can see in this, there's a handful. So this is all of our projects that subscribe to our Python jobs templates would run these series of tests, PEP 8, Python 2.7, 3.4, PIPI, docs, requirements, so on and so forth. And this is how you would apply a template. So you can see this is for Nova. A lot of things happen on this patch. We're not gonna go into it, but ultimately, in here somewhere, you'll see, I think it's in here, Python, well, maybe it's not Python jobs. Oh, there it is, sorry, it's Python DB jobs. This is an old snapshot. But ultimately, you can see in the gate, the Nova project runs a series of jobs and it's one of the most heavily, one of the most projects that uses the most resources. We've got less than 10 minutes, so I'll kind of skip over this. This is the process that we use for archiving our logs off of Jenkins slaves. So because of NodePool, all of our Jenkins slaves are short-lived instances, usually about an hour. And NodePool will generate an image from scratch, runs all of our testing, and then it deletes it and destroys it. So we need a process to archive that information off. Basically, we have a very simple on the left-hand side, SCP, where we take static text files and move them off somewhere. In the middle, we have a more complex process where we take, this is the process for log stash. So you can see we have some Gearman client server. We move the logs off to some log workers, which then convert it and then upload it into Elasticsearch. And then finally, you can see on the right-hand side is just the process for SSHing into the servers.  Okay, so scaling hardware means, so basically we have, I think we're up to about six or seven cloud providers now, which we run all of our jobs across a donated hardware. And we're very thankful for that. And basically, it just allows us to deploy a lot of changes across a lot of clouds. And it's great because it allows us to scale up and down, it will, and more specifically, it allows people's codes to run against different clouds. And all of the clouds may not work the same way. Everything, however, if that's not for your needs, say you have specific hard drives that you wanna test a backend against or something along those lines, you would set up a third party system there's this process that allows you to ultimately set up a mirror image or a mirror setup of the infrastructure CI and allow Zool and Garrett to talk together. So when a change from a patch hits open stack CI, it may get pushed down to IBM CI as well so that IBM can run specific maybe customer scenarios. And ultimately, as long as IBM is pushing those logs up into the public and the open source developers can review them, I think that's basically the minimum criteria. So anybody can have their own CI in the gate. You just need to make sure that the project that you're gating against allows kind of some two-way communication. Very simple, we have multi-master Jenkins servers. I believe we're up to eight now. So we have eight master servers of Jenkins that we just use to trigger jobs and they are all exactly the same provisioned through Git and YAML files and so on. And the reason for that is because the amount of jobs that we were pushing overloaded a single master. So we added a second and then we overloaded that one and now we're up to eight. We use a gear man plugin, which I believe was written by Clark, which is the backend that uses between node pool and Jenkins. And you can see our masters are called Jenkins01.openstack.org, Jenkins02 and so on. And right now we're basically at a capacity of 750 test nodes that can be launched at any given time. So yeah, it's a lot of nodes. And I believe they have four cores, eight, no, what is it, 32 gigs of RAM? I mean, it's not, it's some big hardware. So you can see in a given timeframe, these are kind of showing a graph of how many nodes that we're launching. We're peeking out at one point in time at about, you know, 615 nodes. And all of that, like I said, is controlled by node pool. It's a very elastic concept, automates all the creation of the Jenkins slaves and basically takes care of registering a slave node to a master and yeah, manages multiple open stack clouds. And here you can see kind of more details, I believe, let me see if I can scroll down, but yellow means we're building a node, green means it's available to be consumed, blue means it's in use, and magenta or purple means it's actually being deleted. So you can see it's, in this case, we're up to 825 nodes at one point in time being used on a Monday morning. And I think we're at the end of our slide. Yay, so we have exactly, I think, five minutes for questions. If you could line, yeah, there's a microphone over there if you could line up to ask questions, because there must be questions, there have to be. Success, guys, I think we answered every question anyone could have. Not me, we're good. All right, so the question was, you talked about log stash and logs from your build. So how does, how do you link the log output on the build server or the testing to the code change that was submitted? How do you associate those? So there's, so we mentioned there's two ways. There's the thing we upload to logs.openstack.org or into Swift. That is, how do we link that exactly? So it used to be just like you could, you could figure it out on logs.openstack.org by appending the patch number to a certain path. With Swift, it's slightly different. I think, I don't know how we're, so we do send a link back. And so the developer just clicks on the link. I'm not sure exactly the piece that figures out what that link is. I think it's a hash from Zool or something like that. Yeah, I think it's still just a thing you can figure out. I bet Jeremy knows. Yeah, I mean, there's fantastic people in infrastructure that know all that information, but we didn't really dive into that because it's such a complex process. But ultimately Zool knows about the change and it's at the heart of it, it's the core, and we propagate that information through to both Jenkins and Log Stash and so on and so forth. And it goes back into Garrett, so it's tied to the patch. So you can see, you can click directly in the comments and get straight to that log. Yeah, okay, got it. All right. Any other questions? Okay. I have a question about, how do you monitor in the open stack infrastructure?  So there's two ways. Well, first we don't. We don't run dog use or anything. We probably should. We always need more people coming on our team and helping us, and now that you know how it all works, you can help contribute. We do use cacti to over time track what we're doing server usage-wise. So not everyone on the team can log into the server, but they can all look at cacti.openstack.org and see like, hey, this ran out of disk. Someone should fix that or they can sort of debug and see how RAM usage and CPU and network and all that tracked in the public cacti thing we have. So that's really pretty much what we have for monitoring. That actually speaks to one thing that I kind of regret we didn't mention. The open stack info project is entirely volunteer-based. And so if you think about this project, the fact that there isn't a dedicated, paid-for team of people whose only job is to maintain this infrastructure, it's just a bunch of volunteers that are doing awesome work. It's kind of volunteer, like I'm paid by HP and my job is actually. But HP has volunteered me to do that. Yeah, and to add what Elizabeth and Chris said, we also have stats D in the back-end running, so we don't have an active process to let us know there's a problem. Nine times that a tenant's a developer that tells us that there's a problem. So we need to do a better job of having some sort of automated notification, but at the scale we're talking about, nobody wants to get email bombs when something's broken. Yeah, exactly right. Somebody, people just hop in IRC and say it's broken. Okay, you have time for another question? Yeah, I think we have another minute. How about security? I mean, yeah, who can access to their server, how you keep that password or the public key across all the server? So we have a core group of people, I think there's eight or nine of us now, we who have SSH keys that are deployed to all the servers. So those are trusted members of the community. It takes a year or two to get up to speed on the team, of you being around and being part of the team and reviewing patches and being pretty much working full time on it to be entrusted with that core privilege and root access to all the machines. Beyond that we allow a couple extra people here and there who understand our workflow if they are subject matter in a certain spot. So maybe our asterisk server may have a couple other admins on it who know specifically about asterisk. And then as far as passwords and things go, of course that is the part that is not open source about our projects. We keep them in a Git repository on one of our protected nodes that we have access to. And then we share the details that way in a Git repository or a GPG encrypted file, depending on what the password is and what it's for. And I mean to just add to that, there's very little that you can't do without root access. So usually when you're trying to troubleshoot a problem or something is down and you need to look at sensitive logs, that's really only, I find the only need for root access. Everything is all there and it requires some creative thinking sometimes. But if it's failing in the gate, you can take our tools, download it locally and you should be able to reproduce the same problem locally. If you take the time and effort to download it all, launch it in a local Jenkins or local VM and so on and so forth. Okay, okay, thanks a lot, asset. Hey, I think we're done. Wait, wait, there's one more, please. Hi, I want you to start contributing, right? In which part of the OpenStack project do you guys recommend me to start? Oh, that's such a loaded question. So. Infrastructure. I mean, what I usually recommend people start with is the docs project because it's the easiest to test locally. It kind of, it has the fewest moving pieces. So really big difference, right? Between testing your patch to Nova versus testing a doc update. But doing doc updates gets you really familiar with the Garrett workflow and used to kind of using that process in a pretty easy to consume way. Yeah, I mean, to add to that, I think if there's something you personally, if there's something in infrastructure that you find lacking and you want to drive that, that's also a great way to get involved with the OpenStack, Infra Team. But if you want some low-hanging fruit, I mean, by all means just ask on IRC. We also have a spec process which kind of explains what we have in the pipeline that's coming in the following months and years. But yeah, I mean, there's a lot of stuff that just needs help with. And it's just a matter of just saying, I think, hey, what can I help with? And it's that simple. Yeah, and the specs are up at specs.openstack.org and that'll give you a nice overview of all the specifications that different projects are looking at. If you are particularly interested in something, you might look at one of the specifications that's already up there. You're awesome. Thank you. Thank you. Thanks, everybody. Thanks. Oh, wait, we have one more, please. Just a really practical question about DevStack. Is it when you bring DevStack up once you get it running and then if you shut down the server, then you can't just start the server again. You have to unstack and restack, right? Is there a way to just bring the services back up again without having to run the stack again? I just wanted to clarify, that's the reality, right? Yep, I mean, there's, in theory, there are ways, but it's not like anyone in DevStack Project will tell you just flat out, no, we don't ever want to support that. Got it. Hey, can we just get my mic? It's mainly because it's not launching things as integrated services and like you would expect from system D or something. All right. Yeah, I just wanted to clarify. Thanks.


Good afternoon, and welcome to today's workshop, as you know, co-sponsored by the McLean Center, the Global Health Initiative, the Urban Health Initiative, and the Robert Wood Johnson Disparities Program. I learned this morning that Dr. Olapati will not be able to give today's talk and has asked Jim Fackenthal, a research associate who works with Dr. Olapati in her program, to work from her slide set, as I understand it, and to present the talk that Dr. Olapati intended to give. Jim has very quickly reviewed the slides in the last 10 or 15 minutes, but knows the work pretty well. And so it's a pleasure to welcome you at short notice. Thank you very much for coming. OK, well, thank you for having me. Well, thank you very much for coming. Frumi apologizes for not being able to make it. I will do the best I can to improvise over the slides that she had prepared, and we'll see how it goes. I just want to give a brief overview of what I think are the most important points we want to make. First, we want to talk about the role of genetics in cancer risk, especially breast cancer risk, the role that ethnicity and historical genetic background plays in breast cancer risk, and how that's going to play different roles in different parts of the globe. Want to talk a little bit about how we detect cancers early on and the ways we can bring that to bear on global health disparities. And finally, want to talk a little bit about personalized medicine, because in a lot of ways those two subjects are very closely connected, interested not just in one-size-fits-all prediction methods or one-size-fits-all treatment methods, but in matching prediction methods and treatment methods to individuals. And that match often has to bear in mind personal background, including genetic ethnic background. OK, so there are lots of risk factors associated with the onset of breast cancer, like any other cancer. Being female is obviously an important risk factor, although male breast cancer does happen at about 1,000th the rate of female breast cancer age. And we'll go a little bit more into that later on. Genetic risk factors are what I want to talk about mostly, but also family history, which is tied to genetic risk factors. And then there are a lot of different terms we use to describe unopposed estrogen exposure over the course of a lifetime, including early monarchy, late menopause, late first birth, nulliparity. Previous history of breast or ovarian cancer, even in cases where we don't have necessarily a clear genetic link between a risk factor found in the genome and breast cancer risk, having a strong family history is still a strong predictor. Obesity and, again, some other terms, many of which are probably different ways of looking at unopposed estrogen or hormone exposure risk. OK, well, in 1994, there was a big breakthrough in the identification of a Mendelian risk factor. That is a single gene that, when inherited in the mutant form, caused an extremely high likelihood of developing either breast or breast ovarian cancer family syndrome, very high likelihood of breast cancer risk. And it was called, imaginatively enough, the breast cancer gene, BRCA1. And it made quite a lot of news. And a few years later, BRCA2, a different gene on a different chromosome, but also associated with a very high risk of breast cancer, was identified. And here you see the structures of those two genes. These are simply numbering the exons. Each one of them has a very large exon 11. BRCA1 is a very large gene. BRCA2 is an even larger gene. They both encode immense proteins. These are the kinds of things I think about in the laboratory, what these messages do, how they're regulated, and the different kinds of proteins they encode. Both of them play strong roles in DNA repair. And we'll say a little bit more about that later. But if you look at the literature, especially of BRCA1, you'll see that the number of different proteins that this gene encodes participates in just about every pathway and just about every metabolic schema that you can imagine. These are highly functionally diverse proteins. It's also important to remember that BRCA1 and BRCA2 don't just exist to prevent breast cancer. I mean, we call them tumor suppressor genes, but they play numerous roles during the development of an individual. They're both critically important for embryogenesis, especially early neurogenesis. And they play critically important roles in transcriptional regulation, chromatin modification, DNA repair, the kinds of functions that you would expect to be absolutely critical to every cell in the body throughout development. It's curious that mutations in these genes result in high risks of breast and ovarian cancer and not just, and not cancers of general metabolism in all potential tissue types. So this illustrates a little bit of the way BRCA1 and BRCA2 were discovered. This was done by a linkage analysis as opposed to an association study. We'll talk more about the difference between them. But in a linkage analysis, what you do is you look for a family history of affected individuals and you try to find genetic markers on different chromosomes that seem to travel with the disease in the pedigree that you see here. And that's how BRCA1 was isolated to chromosome 17. BRCA2 was isolated a few years later. And because these were so easily identified with strong family history, especially family history among first-degree relatives, it has become standard practice now for patients who have a strong family history to have their BRCA1 and BRCA2 genes sequenced. That is, it's a good predictive measure of who is going to have a mutation in a family. So it is often the case that if you're from an Ashkenazi Jewish ancestry and you're seeking genetic counseling, a genetic counselor will see that you have a strong family history. They'll suggest that you have BRCA1 and 2 sequencing, and they might suggest, if you're of Ashkenazi background, that you only have those three mutations screened for. Saves time and money. From other populations, though, that have not undergone a recent bottleneck, there aren't going to be any very high likelihoods that the mutation will be just one of a number of a small number of alleles you have the entire gene sequenced. So this availability of the sequencing, availability of knowing whether your family is a carrier of a mutation in the BRCA1 or 2 gene, some people believe may be contributing to health disparities. That is, awareness of genetic testing, access to genetic testing, and the capacity to act upon that information. Some people believe is an important aspect of health disparities. If you look at the prevalence of BRCA1 and BRCA2 mutations in families that are associated with a strong family history, that is, or at high likelihood of risk, that is more than two affected individuals within the family. According to a paper by Rita Nanda from this institution, you can see that 27% of caucasians that were described by this, I'm gonna lose it, okay. 27% of caucasians carried a BRCA1 mutation and 14% carried a BRCA2 mutation, whereas the numbers were considerably smaller in African-American populations. Only 15% carried a BRCA1 mutation, only 13% carried a BRCA2 mutation. Well, BRCA1 and BRCA2 are probably it for high penetrance Mendelian mutations. That is, we're probably not going to discover a BRCA3 or a third mutation that when mutated or inherited in the mutated form is going to be associated with a very high risk of breast cancer. Chances are the rest of the genetic components to breast cancer risk are going to be in genes or factors that have individually much smaller individual contributions but collectively contribute to significant risk. And we describe these statistically by being variants in genes associated with very low odds ratios. That is, we've pretty much taken a relative risk of 10, that is a very high likelihood of being associated with the mutation. We've seen in BRCA1 and BRCA2, there are a couple other genes that are associated with high breast cancer risk in the context of other multi tissue syndromes like Lee-Frowman syndrome or Calden syndrome associated with high breast cancer risk. These are very high mutations and these genes are very high penetrants. But what we're looking at now is instead of very rare mutations associated with high risk, we're now looking more and more using genome-wide association studies at much more common variants, each of which is associated with a comparatively low risk and it's combinations of those low risk factors which you will not see in a family history like a Mendelian factor that is contributing to the rest of the genetic component of breast cancer risk. We've also come a long way in stratifying what we used to call breast cancer, the single disease. And some of this has come from expression studies using tumors from lots of different breast cancer patients done by our collaborator, Chuck Peru, now in North Carolina who moved there after he did this work with Brown in Southern California. And what he noticed was that breast cancer, if you look at the expression of lots of different characteristic genes that are either up-regulated or down-regulated that distinguish tumor from normal or one tumor from another tumor can be broken down by molecular signature into at least, probably more now, five different groups. There's the Claude and Lowe, which is sort of basal-like but it's associated with a reduction of a couple of other markets including the Claude and Marker. There's the basal-like, that is these are tumors that have a gene expression pattern that is very similar to what normal basal epithelial cells look like. There's the HER2-like, one of the strong predictors of outcome in tumors is amplification of the HER2 gene. And these are tumors that have a strong signature that's similar to HER2-amplified. These are normal-like and that's sort of a vague description when you look at the gene expression pattern of adjacent normal tissue and these tumors, you don't really see a big difference, so they're called normal-like tumors. And the luminals, A and B, that's two slightly different gene expression patterns that look very similar to the normal expression pattern of luminal epithelial cells. It is Claude and Lowe and basal-like that we typically call triple negative by immunohistochemistry. And by triple negative, what we mean is when you stain for antibodies against estrogen receptor, the progesterone receptor, or HER2, they're negative for all three. And that is a very aggressive behavior. If you look at mortalities associated with these different tumor subtypes, what you see is that the basals are typically the most lethal. What we call by molecular signature basal-like, what we call by more convenient IHC assays that you could get from a typical pathology assessment, triple negative, those aren't exactly the same thing, but they overlap a lot. Those are typically the most lethal and the most refractory to most kinds of treatments. Interesting thing is when you look at tumors that are associated with BRCA1 mutations, you see that their molecular signature is very basal-like. BRCA1 mutation associated tumors are also lacking in estrogen receptor, lacking in progesterone receptor, and lacking in HER2. So that suggests they may have a common etiology, but more clinically relevant is the idea that a lot of the targeted therapies that we're using now, therapies directed against the presence of an estrogen receptor, for example, the serums, for example, they're not gonna work. And herceptin, which is a monoclonal antibody, it's a humanized mouse monoclonal antibody directed against amplified the HER2 receptor, which is indicated when you see HER2 amplification. That's not gonna work, because it's not amplified. So the conventional targeted therapies we have for treating ER positive and HER2 positive tumors are not gonna work on these very aggressive, fast proceeding tumor types. So this raises the questions, are there population differences in the incidence of triple negative or triple threat breast cancer? Is that nature or nurture? And does it explain health disparities? Okay, breast cancer is a global problem. It's not always perceived as a global problem, though. There are some people in some places who believe, and not without reason, that cancer is a disease that older people get. In fact, the older you are, the more likely it is you're gonna get it, because the principal cause of cancer is bad luck. Genetics influences it a lot, environmental factors influence it a lot, but what it's influencing is the underlying bad luck. It's gonna happen. There are a lot of countries that are devoting a lot of resource to infectious diseases that are killing people at a very early age and are wondering why should we worry about cancer when our population isn't always getting old enough to get cancer? Infectious diseases are taking out a lot more people. And it is true that you're going to see a higher incidence rate in places where you have higher incomes. And that is something we're working to address, especially in some of the African nations, some of the emerging developing nations, where we're seeing increases in cancer incidence as a lot of the other types of diseases are on the wane. But there are other considerations, and a lot of it has to do with what's being offered and who's taking advantage of it. When you look at people who have cancer and see at what rates they die, between 81 and 83 and 1992, the rates of mortality for blacks and whites were similar. But then something happened. The rates of mortality went down for white population in Chicago, stayed roughly the same for black populations. Not entirely clear why that is. But one thing that we're gonna try to impress on you is that the proportion of these triple negative breast cancers, of these basal-like breast cancers is higher in black populations than it is in white populations. These are the cancers that are more refractory to treatment. It may also be influenced by access to treatment, but right now I'm just gonna focus on the genetic characteristics. Oops. Okay, that shows that the study was done in different cities with different outcomes. Why do we think about the African diaspora? There are two important points I wanna make. First, a lot of what we study in breast cancer genetics has to do with what the background genetic diversity is in different human populations. And the first thing you have to remember is half, if not more, of the genetic diversity that occurs in people occurred in Africa before anybody left Africa to populate another nation, okay? So half, if not more of the genetic diversity that exists on Earth existed before there was a diaspora from Africa to populate other continents. In fact, I've heard some African scholars say that there is more genetic diversity between East and West Sudan than there is in the rest of the world, and it's plausible. I don't know if it's true, but I like how it sounds, and it's plausible. The other important thing to remember is a slightly technical one. A lot of the genetic diversity we look at comes from samples that go into what we call the International HapMap Project, where they take samples from different populations, Han Chinese, Japanese, the Ceph, which is a Caucasian population, and a Yoruba population, which comes from Nigeria. And these are taken to be the representative examples of peoples from throughout the world, and the starting point for looking at genetic diversity, and sort of the landscape on which we project our risk factor hunting. Note, though, that slave trade did not just come from this one small spot in Nigeria, which gave rise to African American populations in North and South America, came from all over the West Coast of Africa and even a little bit into Ethiopia. So all that genetic diversity that was taking place during the first half of human evolution, that is the source of the African population during the diaspora, not just what's going on in the Yoruba as in the HapMap. So that's an important consideration to keep remembering. Okay, so many of those samples that came from the Yoruba population, that is an ethnicity within Nigeria, came from a collaboration between the University of Chicago and the University Hospital in Ibadan in Nigeria. And many of those samples that went to the HapMap were duplicates of the samples that came to the University of Chicago as part of an earlier collaboration. And part of what we wanted to look at was what's the genetic landscape difference between the populations that received most of the study in North America, the populations that gave rise to the mutation prediction models, the populations that were used for the clinical trials for various kinds of treatments. What is the genetic differences between those populations and a population that is ancestral to much of the North American African American population? So the first one of the things we wanted to do is look at the spectrum and frequency of BRCA1 and BRCA2 mutations. Remember I showed you an earlier study that showed that BRCA2 mutations were less frequent in African Americans than they were in Caucasian populations. Well, we looked at the frequency and spectrum of those mutations in this Nigerian population, which was primarily Yoruba, although there were some Igbo and some other ethnicities represented in here. And what we found was that a very high number, 7.1% carried BRCA1 mutations and 3.9% carried BRCA2 mutations. Why are those high numbers? Well, most of the other studies, they look at mutations in patients who have strong family history and are therefore very likely to carry a mutation. We didn't do that. We looked at all patients, regardless of family history, and found we had extremely high numbers. These would be considered high numbers, even if we had selected only for patients that had family history. So why is it that in this Nigerian population, we have BRCA1 and 2 mutation frequencies that are higher than any have been seen before, but in the African American population, those mutation frequencies are lower than have been seen before. Not entirely clear why that is. Part of it may have to do with admixture. Part of it may have to do, though, with the genetic diversity that gave rise to the African American population in North America. Not all North American African Americans are descended from the same population that gave us modern Nigerian Urabas. Now, this is kind of a cute diagram. It shows how the prediction models work. What we're interested in looking at is the models that allow us to predict whether somebody is a likely carrier of a mutation. And most of the information we use to try to make that determination is what is the strength of their family history.  And what you see is that if you have zero affected family members, only one in 20 patients will carry a BRCA1 or 2 mutation. If there's one affected member, that drops to one in 16, still not a very strong predictor, but it's one in 16. If there are two affected family members, one in eight, and if they're more than two, half. Okay, so even when you have very strong family history, half of the time it's not going to be because you carry a BRCA1 or a BRCA2 mutation. It could come from other reasons. And we showed you before that the BRCA1, the tumors associated with BRCA1 mutations have a unique molecular signature that makes them look very like basal-like tumors. This shows that phenotypically, they also have a distinctive pattern. Medullary and atypical, medullary, well, all tumor types show those on occasion, but here in BRCA1 associated tumors, they're more frequent. High mitotic rate, that just means that cells are dividing a little faster or more of the cells are actually engaged in dividing. Higher degree of aneuploidy, that's an association with a genomic instability, possibly associated with a defect in the BRCA1 gene itself, which is so important for DNA repair. High proliferation fraction, I don't know why that's different from high mitotic rate, but I guess it is. And as I mentioned before, ER negative, PR negative, and NOHR2 gene amplification, these are the indicators of triple negative tumor status. Frequent TP53 mutations will, of course, most tumors have frequent TP53 mutations, and this is what's very interesting, and I alluded to it earlier before. This is a pattern described for basal-like tumors. It's a pattern described for triple negative tumors, and it's the pattern more commonly seen in young African-American women than in other populations. So when I mentioned before that part of the reason for the differences in decline in mortality after these different kinds of drugs were available, part of the reason could be because the frequency of triple negative breast tumors is higher in young African-American women than it is in other populations. So this, again, is simply illustrating a study that we're participating in where we're taking tumors from different sample centers, not just tumors, but epidemiological information. We're collecting blood samples, preparing DNA, RNA, looking at some of the not just genetic risk factors, but also calibrating the genetic landscape that we have to work with in identifying risk factors. And this is simply an illustration of what a pathologist can do to help stratify different tumor types because it's very difficult and very expensive to try to get RNA that you need to do the molecular characterizations. So luminal A's are gonna be characterized by staining positive for ER and PR. Luminal B are gonna stain positive for ER, PR, and HER2. The HER2s, which are gonna be typically ER negative, will stain only for HER2. The basal likes will stain for cytokeratins five and six and for the epidermal growth factor receptor and the unclassifieds just don't stain with anything. So this is one of the assays we use for determining who has what kinds of tumor. And using these kinds of assays, we see that African-American women, premenopausal and postmenopausal, African overall, are gonna have a higher frequency of basal-like tumors than other populations. And remember, the basal-like tumors, the triple negative tumors are the ones most refractory to standard treatments and are also usually typically the fastest progressing. And we mentioned these before, Fumi inserted this slide. Again, there are different breast cancer risk factors. Again, we're talking mostly about the genetic risk factors, but we've mentioned estrogen receptor, ER a couple of times. Now you can see why all these different terms that really describe a lifetime exposure to estrogen like early monarchy, late menopause, first birth, null parity, exogenous hormone use. That's why these are considered risk factors. Now, in addition to genetic factors and environmental factors, we're considering a realm that sort of falls in between. That is epigenetic factors, risk factors. These are covalent changes to gene structure that can be inherited like a mutation, but can also be environmentally induced. So it's almost like a return to a Lamarckian model of evolution. Some of these factors that can induce some of the covalent modifications, by which we mean methylation of certain C groups in DNA, we'll get into that in a minute, are stresses which can come from social interactions or failed social actions, lacking interactions, and there's an active research program going on here to address all of those things. This is a map associated with that Chicago wide study and I really don't know what more Fumi would have said about that, but this is a collaboration we're doing in conjunction with Suzanne Kansen who's very interested in glucocorticoid receptors and other hormonally inducible responses to stress, one of which could be things like social isolation or a certain other neighborhood associated environmental influences. We also mentioned previously looking at personalized medicine that is trying to apply therapies that are especially well suited to somebody's personal and more specifically genetic and molecular issues. We said, for example, that you can't apply a hormone-based therapy to somebody who's got an ER negative tumor because there's no target for it, so that is, but on the other hand, then that's somebody you can spare from side effects when they wouldn't have benefited from the therapy. So that's an example of the kind of personalized medicine we're looking at. Here is another approach that we're taking and it draws from the observation that one of our recent graduate students in the lab was working with, she was looking at one of these epigenetic modifications I mentioned before. I said it was a methyl, it was a covalent modification of gene structure. It's actually the addition of a methyl group to a cytosine base in the context of a CPG island. That is, if near the promoter of a gene of interest, you have C, G, C, G, C, G, some of those C's will acquire a methyl group. When that methyl group is attached to the C, then certain modifications are going to, it'll attract methylated DNA binding proteins and those are going to attract other modifiers that modify the histones in the regions and promote an inactive chromatin configuration. So a methylated C is an indicator of transcriptional inactivation or a promoter of chromatin configuration that inhibits transcription. We expect less expression, therefore from a gene whose promoter is associated with a methyl C. And what she noticed was that there are CPG islands in the promoter region of BRCA1. And in addition to BRCA1 genes that are mutated, being associated with these basal-like phenotypes, sometimes when you see not mutated but promoter methylated BRCA1s, those tumors will also be associated with BRCA1-like phenotypes. It is as if the gene was inactivated by methylation rather than by being mutated. We also mentioned that BRCA1 is intimately involved in DNA repair and it's believed that part of the reason that cells start accumulating the kinds of mutations that ultimately can lead to a cancer is because of the failure in the DNA repair because of the failure in BRCA1 gene function. There are lots of genes though that are required for DNA repair. Another one of them is the PARP gene. And what this student noticed is that in tumors that had methylated BRCA1s, BRCA1 gene expression level was much lower as you predict in basal-like tumors that were unmethylated at the BRCA1 site, BRCA1 expression levels were much higher. If you look at those same tumors, again, where BRCA1 promoters are methylated, PARP1 expression is much higher but in unmethylated tumors it's a little bit lower. So from this and similar observations, the idea is that if a tumor is associated with a defect in DNA repair because it's lacking in BRCA1 function, either because it's mutated or because the BRCA1's inactivated by methylation, you might disrupt DNA repair functions even further if you inhibit PARP. That is, to become a tumor, you need to have a little bit of genomic instability but not too much. A little bit of genomic instability allows you to accumulate the kinds of mutations that allow tumor genesis to progress but too much just leads to metabolic meltdown. You don't have enough genome to support life of the cell. So if the tumor is already lacking in some DNA repair functions for lacking BRCA1 function, you can push it over the edge into lacking too much DNA repair and simply kill the cell if you give it a second hit, an inhibitor of the PARP1 gene. This illustrates part of the stem cell model. A lot of people are wondering why is it that BRCA1 associated tumors are so often ER negative, PR negative, HER2 negative, that is basal like? It came from an important, one model came from an important observation that showed that the BRCA1 gene function is necessary for tumors to differentiate into basal like epithelial, or for normal breast ductal and lobular tissues to differentiate into basal like epithelial cells and luminal epithelial cells. That in the absence of BRCA1 function, a progenitor cell, which would have differentiated into those more mature epithelial cell types will stay basal like. It will stain negative for ER, PR and HER2 new. The idea then is that the reason that BRCA1 inactivation leads to all these triple negative tumor types is it because lack of BRCA1 function traps cells into that progenitor luminal cell type. It doesn't allow cells to progress into the luminal cell fate. And so those cells never had an opportunity to come into existence, so they're not the ones that are able to give rise to the tumors. And this just illustrates what I was talking about a second ago. Imagine a two-hit model. If DNA repair is allowed to progress with normal base excision repair, which is a process governed by PARP1, and by homologous recombination, which is a process governed by BRCA1, you have full levels of DNA repair. If, however, you inactivate homologous recombination pathway because you've inactivated BRCA1, you still have base excision repair operating because PARP1 is still there. You still have a viable cell. Some defects in DNA repair, but there's enough DNA repair by alternate pathways to allow cell viability. Same thing, if you have a PARP deficiency because you've hit it with a drug, for example, that's a PARP inhibitor, you have some defects in DNA repair but the cells are still viable. And this is important because when you apply a therapy, you don't want it to poison all the cells in the body. You just want it to poison the cancer cells. Well, the rest of the cells in the body are not lacking in BRCA1 because they have an undergone complete loss of BRCA1 gene. They still have one viable copy of the BRCA1 gene left. So by this mechanism, a PARP inhibitor isn't going to be as toxic to other cells in the body, just the tumor cells that have lost all BRCA1 gene function there where they have lost all capacity to do homologous recombination because of having lost BRCA1 gene function and lost the capacity to do base excision repair because you've inhibited the PARP functions. Then that is a cell which isn't just defective for DNA repair, it's a failure in DNA repair and so the cell just collapses from inability to support itself with a viable genome. Now, I mentioned before that there are other targeted therapies. We mentioned estrogen receptor and I also mentioned HER2. That is, there is a portion of chromosome 17 that can become amplified. Many, many copies of this tiny portion of chromosome 17 that carries the HER2 gene. When that occurs, you have too much of the HER2 gene product which is a growth factor receptor and that results in tumors that's staying positive for HER2 or are detected HER2 by fluorescence and C2 hybridization and those patients and only those patients will receive the drug Herceptin which is target against the HER2 gene product. Herceptin is not without side effects. You don't wanna give it to everybody, only the people who would benefit from it. Okay, now when Fumi shows these slides, she points to points of density and says very clever doctor sounding things that I'm not gonna try to replicate here because I don't understand any of them. She says clever sounding doctor things here too. Ulcerated is one of them but what we see in a lot of African cases is a very late presentation of breast tumors. There are a couple different ways of thinking about late presentation and this confuses a lot of people so it's important point to make clear. Sometimes late presentation means a tumor has gone ignored for a very long time and is only presented in a clinic when it's something other than a worry when it's actually causing extreme pain or limited mobility. Late presentation can mean something else though when we're talking especially about triple negatives or basal like breast cancers. These are tumors that upon their earliest detection look like late stage tumors. Possibly because they proceed through early stages much, much faster. Possibly because they don't even bother with those early stages. They just jump right into what we call a late stage tumor. So that's an important distinction to make when you hear people talking about late stage presentation. It's not just a delay between detection and presenting in the clinic. It's also talks about the speed of different tumor development rates and they are very particular to different tumor subtypes. Okay, so this is actually too vague to describe so I'm not gonna try it. So this is an idea which isn't just something we're thinking about in the lab. It's something that is a big part of the attention in the budgets going on in the National Institutes of Health. Fostering innovation, research to identify the most promising technology, improved delivery of evidence based interventions, changed genomics for a developing country to genomics by developing countries to build local research capacity, develop ELSI expertise, leadership development programs to form mutually enhancing partnerships. And again, this describes a lot of what we're doing in our collaborative efforts with University Hospital in Ibadan in Nigeria and with other places where we're working in Senegal and South Africa, other places where we're collecting samples. Here's a picture of Fumi. Again, she apologizes for not being here. I know it's a disappointment. She's a whole lot prettier than I am. This is Clement Adibamo. He's our surgery contact at the University Hospital in Ibadan and this is some of his crew there. This is De Jong Ho. I don't know if he's here but he's with the Center of Health Studies here at the University of Chicago and let me stop there. Thank you for your attention and take questions. Yeah. Is there currently treatment for triple negative breast cancer? And if so, does, or if not, would an early detection by a mammogram help? Yeah, early detection always helps. There are plenty of, yeah. Okay. Is there currently a treatment for triple negative breast cancer? If not, would early detection by mammograms help? Yes, the treatment is not personalized treatment but there are standard chemotherapy treatments that are used and early detection always helps. I mean, it almost seems to go without saying but people went and studied it anyway that people with more aggressive tumor types are going to be more easily treated with early detection than if they present too late. More promising targeted therapy though that is in a lot of development that are these PARP inhibitors or triple negatives. Other questions? I want to go back to that Chicago slide you showed of the difference in survival after 92. And I'm trying to get the take home lesson that you and Dr. Olapati are offering. Are the differences that we see, is part of your claim that the differences that we're seeing in populations like this primarily accounted for by the genetic variation or that the genetic variation is one of the elements to which you would add as you suggested in your talk lack of access to care, lack of early screening, lack of adequate treatment but I couldn't quite get the bottom line. Yeah, what can't say primarily we can only say what the different contributors are and right now I don't think we have a good feel for what proportion each contributes. One of the contributors is that African Americans have a higher frequency of these treatment refractory tumors than other populations but we don't know if that's the predominant or if that's more than just a slight contributor to the difference. I see, so you're not making that claim. No. Only that it ought to be included among all the other things that may be contributing. You said that the methylated genes get passed down so how can that be if the Cs are methylated and then when you make some more DNA you pull some Cs out of the environment but those aren't gonna be methylated. Okay, so remember the reverse complement of CPG in the other direction is CPG and there are enzymes that recognize hemimethylated DNA which is what you get after one round of replication. That is this C is methylated but this newly added C on the other side is not. There are enzymes that recognize that hemimethylated DNA, recognize the methylated strand as being the old one because it's methylated. They decide that's the right one that anything that's not methylated must be new and if it's different must be a mistake and it methylates it. Can you talk a little bit about early onset breast cancer and sort of what is going on in that area currently what populations are being affected the most? Early onset breast cancer, yeah. Okay, yeah, good point. Sorry, I didn't mention that before. So one of the characteristics of BRCA1 and BRCA2 associated tumor risk is earlier onset breast cancer. That is an earlier age of getting breast cancer than you would see among sporadic populations. In fact, there are studies that show that if you have a BRCA1 or two mutation, your chances of getting breast cancer by the time you're 40 are pretty high but if you haven't gotten it by the time you're 45 it's a little lower and if you haven't gotten it by the time you're 50 it's lower still and eventually it gets down to the level of which you'd expect to have population-based sporadic breast cancer. One of the reasons we thought that early onset breast cancer which is what you would predict from having a strong genetic component might be part of the same etiology as what we see in African-American patients is because in addition to being high frequency triple negative, high frequency basal like tumors in African-American populations and African populations despite very, very different environmental influences have a higher frequency of early onset breast cancer than in other populations which is again, one of the reasons why we thought there would be such a strong genetic component to breast cancer risk in patients of African ancestry. But I want to thank Jim back on Paul very much for stepping in before me. We really appreciate it.