Good afternoon, everybody. My name is Bob Trug. I'm the director of the Center for Bioethics at Harvard Medical School. And I'd like to welcome you to our health policy and Bioethics Consortium, which is part of a series of events that is organized and led by Dr. Aaron Kesselheim, who will be speaking to you in just a moment. I just wanted to, first of all, thank you for being here. And also put in a little plug that the Center for Bioethics has something organized on almost every Friday from 12.30 to 2 in this time slot. And we rotate between a clinical ethics consortium, research ethics consortium, health policy and bioethics consortium, and an organizational ethics consortium. So if you block out that time slot in your calendar, you'll always have something to do and a free lunch on Fridays. If you're not on our mailing list and you'd like to be, please go to our website and let us know. And again, thank you for being here. And I hope to see you at ones in the future. So Aaron? Well, thanks, Bob. And thanks to the entire Harvard Medical School Center for Bioethics staff for helping put this together. And thanks to you all for coming out. It's really a pleasure to see so many people here inaugural meeting of this seminar series. So let's see. OK, so this is the first of the year. We're going to this is a set of them that we are organizing through my group, the Program on Regulation Therapeutics and Law. Portal is a research core within the Division of Pharmacropidemiology and Forework Economics at Brigham-Rumans Hospital. We have two full-time faculty. Amit is here, a number of us, additional affiliated faculty. We have a number of great postdocs who work with us as well and numerous students. And our focus is on the intersection of law, therapeutics, and public health, which is what drew us to try to organize this session. Our budget is generally funded by the funders you can see up there, the Law and Genital Foundation, Commonwealth Foundation, Harvard Program and Therapeutic Science. I want to particularly thank for their support as well. And we run courses in addition to this, which is a course for some of the students at the Center for Bioepics. Amit runs a course on public health law at the School of Public Health. I run a course in FDA law at Yale Law School. And you can follow us on Twitter, I hear, at portal underscore research. So I just want to make sure you guys mark your calendars for upcoming ones of these. I'll say this at the end, but in case people have to leave early. Our session next month is on international trade and access to medicines, discussion of public health concerns about high cost medicines and international trade agreements with Tom Boyke from the Council of Foreign Relations and Amy Kaczynski from Yale Law School. And then we're going to talk about the hope and hype of precision medicine in November with Sandra Galle, who's the Dean of the Boston University School of Public Health and Calla McCrae, who's the Chief Cardiovascular Medicine at the break-up. Okay, but today we're here to talk about the ethical involvement of patients in the FDA regulation of new products. And we are honored to have two expert discussants who have come in to share their experience and expertise with us. Diana Zuckerman is the President of the National Center for Health Research, a non-profit research and educational organization that conducts and analyzes research and uses the results to improve the policies and programs affecting the health of adults and children. She has testified about the safety and efficacy of medical and consumer products dozens of times before Congress, federal agencies, state legislative committees, and even the Canadian Parliament. She trained in epidemiology and public health at Yale Medical School and worked for a dozen years as a congressional staffer, as well as a senior policy advisor to First Lady Hillary Clinton. In her current position, she's also, in addition to her current positions, she's also a fellow at the University of Pennsylvania Center for Bioethics and was the first non-physician elected to the Women in Medicine International Hall of Fame. She previously chaired Maryland's Women's Health Promotion Council and currently serves on CMS's Medicare Evidence Development and Coverage Advisory Committee and the Board of Directors of Reagan-Udall Foundation and the Alliance for a Stronger FDA. We also have Walid Gulad. Dr. Gulad is an Associate Professor of Medicine at Health Policy at the University of Pittsburgh, where he leads the Center on Pharmaceutical Policy and Prescribing. His research focuses on physician prescribing practices and on policy issues affecting access and adherence to medications for patients. He was the recipient of Career Development Awards in the Department of Veterans Affairs and is currently funded by the VA, NIH, CDC, and State of Pennsylvania in multiple studies of pharmaceutical policy and prescription use. He's a former member of the FDA Advisory Committee on Non-Perscription Drugs and was important for this panel a member of the FDA Advisory Committee that reviewed Phlebancerin in 2015. Dr. Gulad is a board certified in internal medicine and completed a residency. In chief residency, we were actually residenced together in internal medicine at Brigham Limit Hospital at Harvard Medical School. So a little bit of background for today's conversation. So under the Food, Drug, and Cosmetic Act of 1938, clinical evidence for investigational new drugs must be collected before prescription drugs can be sold to the public. And the statutory requirements for the development of this information is that the information must provide substantial evidence of efficacy for the intended use of the product that arises from adequate and well-controlled investigations. And the safety of the product must also be established to the FDA by adequate tests by all methods reasonably applicable to show whether or not the drug is safe for such use. I'm sure most people in the room know that in order to get a drug approved and to demonstrate efficacy and safety, the testing requires preclinical studies as well as phase one, two, and three clinical trials, which are the different increasing complexity of clinical trials that an investigational drug goes through. Phase one trials, usually in healthy volunteers or single-dose studies, providing some evidence of the pharmacokinetics and pharmacodynamics related to the drug. Phase two trials are in a larger population, usually in the patients with the disease indicated to treat. It may provide some evidence of some initial look into the efficacy of the product. And in the phase three clinical trials are these larger, much more complex comparative studies that provide evidence of the efficacy of the product. And then after this information is collected, it's submitted to the FDA where determining whether a drug's benefit outweighs its risk can involve dozens of FDA scientists pouring over extensive databases of studies in animals, toxicologic evaluations, and clinical trials. Now the FDA doesn't always do all of this internally. In fact, there is this process where the FDA can call on outside advisory committees, which are technical committees of outside experts to comment on the data as well. These days, in addition to technical experts, advisory committees usually include at least one consumer representative and now usually include at least one patient representative as well. The advisory committees provide advice on the approvability of specific products, scientific and policy issues they confront. And it provides a public forum for the discussion of controversial issues. There are 51 different advisory committees across the FDA. And these advisory committees make non-binding recommendations to the FDA about the questions that the FDA has put to it regarding the data that emerges on the trial. And there's some data suggesting that the FDA follows advisory committee recommendations about three quarters of the time. In addition to these expert advisory committees, there are a number of ways now that there are intersections between patients and the drug approval process. So one might be through patient-reported outcomes, which are measures that patients may provide in the course of studying a drug, the sort of the patient's assessment of the medical products effects, and can be used as a way of examining the outcomes of the drug. The FDA released some guidance on this relatively recently describing what sort of patient-reported outcomes are how they're integrated into the clinical investigation process. And in recent years, from about 20 submissions a year, that might include patient-reported outcomes prior to this. Currently, about over 100 different submissions to the FDA a year include some amount of patient-reported outcomes as part of their clinical, part of the sort of clinical information that's provided to the FDA in evaluating investigational drugs. Other ways that the FDA intersects with patients in the review process is through the Patient-Focused Drug Development Initiative, which was established in 2012 to try to systematically gather patients' perspectives on their condition and available therapies to treat their condition. And as part of that, the FDA has held two dozen different meetings with different patient groups over the last five years. There's the FDA Patient Network, in which the FDA puts out information to patients. And there's a publication called Patient Network News. There's a Patient Engagement Advisory Committee that advises the FDA commissioner on complex issues related to regulations, so it's basically an advisory committee made up entirely of patient stakeholders. And then, of course, in addition to this more specific process about patients' involvement in the evaluation of particular products, there are other ways that patients can become involved in the regulatory process. For example, when the FDA puts out a new rule that may affect the availability of products, there is a notice and comment process where anyone from the public can provide commentary on the rule, and then the FDA has to respond to the commentary as part of the rulemaking procedure. And then there are things called citizen petitions that allow people to raise issues to the FDA that they think the FDA needs to evaluate about their regulatory process, and then the FDA has to respond to them. So what we want to do is talk about this intersection and dynamic between the FDA and patients in the context of two recent cases. The first being philbancerin, which, well, Eve will provide a lot more detail on, but I just wanted to introduce very quickly. Philbancerin was a failed antidepressant that was repurposed to treat a condition called hypoactive sexual desire disorder in premenopausal women, a condition for which there are no other approved treatments. However, the first two efficacy studies failed to show any benefit, and along with substantial safety concerns, the initial application was not approved. At that point, the drug was acquired by a small manufacturer, from the large manufacturer of Boring or Engelheim who was studying it, it was acquired by a smaller manufacturer called Sprout, and conducted a new efficacy trial showing a statistically significant, but numerically small treatment differences compared with placebo, and there you can see the level of the difference that the newer study showed, an increase of 0.3 on this female sexual function index desire subscale that they used, again the range of the subscale was from 1.2 to 6, and an increase of about 0.5 satisfying sexual events per month on average. In addition to conducting these additional studies, Sprout launched a new advocacy group called Even the Score, which conducted an intense promotional campaign directed a journalist women congress in the FDA, including sending the FDA more than two dozen letters, some accusing the FDA of sexism from various advocacy organizations and researchers, and so, and then they of course they had an advisory committee meeting regarding this product, and here is a report from the New York Times on what that advisory committee meeting was like. That campaign, which packed the advisory committee meeting room with the drug supporters, apparently helped tip the balance for philbancerin, which had been rejected twice by the FDA. Thursday's meeting included nearly two hours of testimony from the public, an unusually large amount, with most speakers urging approval. Many were from organizations in the Even the Score coalition. Their remarks were loudly applauded as was the final vote in favor of approval. And then the second case is the case of Edna Plurson, which is an investigational product intended to treat Duchenne muscular dystrophy. I would point out that the FDA ultimately did end up approving philbancer. So Edna Plurson, Duchenne muscular dystrophy is a degenerative fatal disease of mostly male children, and is of course a very substantial unmet medical need. Edna Plurson, it's caused by a failure of the protein dystrophy as a result of a stop codon in the production of it. And the Edna Plurson product was intended to be an exon skipping drug that would allow the muscle to recover some functional dystrophy protein and therefore to prevent the loss of activity in the kids. So the initial, the clinical trials that were completed and submitted to FDA included an initial proof of concept, phase one style study, single dose study, and a dose finding study in 19 patients range from 0.5 to 20 makes per kid for the product. And then a final single double blind placebo controlled phase two study that lasted initially 24 to 48 weeks within included 12 patients. And then an open label extension that followed these patients for up to four years of four patients receiving the 30 make per kid dose, four patients receiving the 50 make per kid dose, and four patients receiving a placebo. So the data on Edna Plurson was controversial. And so the primary end points of the trial were generally not met. There was no change in dystrophy in production at 12 weeks in the 50 make per kid cohort, although there was a statistically significant increase at 24 weeks for the 30 make per kid cohort. And then the functional endpoint did not show a statistically significant difference in the six minute walk test at 48 weeks. However, within that kind of general negative picture, there were some positive signs that this drug may have some effect. So if you excluded two of the patients that lost functional activity pretty quickly in their process, and you can see that on the graph on the left, actually the difference was, there was a statistically significant difference at 48 weeks, although of course this is a post hoc change to the study design which has its own problems. And then of course if you look at the four year data as compared to a set of historical controls, the blue line being Edna Plurson line and the yellow line being the historical control, you can see that there does seem to be some evidence of benefit in terms of maintained six minute walk distance for the patients who had been receiving this product for this time. But of course there were a number of questions related to the data collection process, questions about the assay process, questions about the correlation between the amount of dystrophin observed and the six minute walk test results which you can see on the right there, that there isn't really a correlation between increases in dystrophin level and six minute walk test. And then the FDA presented some re-analyses of the historical control data, which if you account for the age of the patient at the time that they started treatment and actually shows that the differences may actually not be there. Because of this controversy about the data, a advisory committee was organized, I was one of the members of the advisory committee. And the advisory committee included 52 public commentators that went on for, we talked for about, the public comment session went on for about four or four plus hours, 51 of those commentators spoke in favor of approval. And here's just a report from the STAT News article about the committee. They reported that the session was easily one of the most politically charged FDA advisory panel meetings. Among them was one boy who appeared with his mother and testified before the crowd.  He told the panel prompting cheers and applause from the overflow audience. FDA, please don't let me die early. One Wall Street analyst believes no one should be surprised if the FDA does approve the drug. Everyone was visibly moved by the audience in their testimony and that is not going to go unnoticed, said Stephen Brozak. It's not just political pressure, it's advocacy pressure of the sort that I've not seen since the days of HIV and AIDS. So this drug remains unapproved as of yet. More data was requested by the FDA of the sponsor. So our charge for today is to talk a little bit about this process and this, maybe we're seeing a trend, maybe we're not of involvement of patients and patient advocacy groups in the FDA the liberation process and whether or not it is being optimally organized and how it can be optimally organized. And I'm hoping that this can be a very interactive discussion after a few initial comments from our expert panel. We're gonna try to get conversation, sort of moderated conversation with the audience early on so that your questions aren't just held until the very end. But what I don't wanna do is I don't wanna relitigate the approval decision or talk about whether or not it was correct for the FDA to approve or not to prove full banter, but to talk about the process by which the FDA gets there. And these are some of the questions that I want people to think about as Walid and Diana talked. How should patients contribute to the FDA evaluation of investigational drugs? What are the different points in the drug review process when patient voices are most useful? How should individual patient voices be weighed against accumulated data about a drug? How should we consider financial relationships between patient advocacy groups and industry sponsors? And should we be trying to more proactively educating patients to equip them to engage in discussions about trial design statistics and evaluation? So with that introduction, I wanted to bring Walid up who is gonna talk about his expertise and experience with the full banter and discussion. All right, well, it's great to be here. This is a great crowd and it's a great topic. So I'm really happy to be here. I'm just gonna give a little bit more background on full banter and a little bit over what Erin had said and then just a few of my thoughts to get people started thinking about the role of patients with phlebancerin. So phlebancerin, and I'm gonna go, Erin mentioned some of these, so I'll go quickly, but phlebancerin, the brand name is Addie, it was approved a year ago for the treatment of premenopausal women with HSDD, which is a hypoactive sexual desire disorder. And it was quite controversial. And in the end, the approval required very strict REMS, which are risk evaluation mitigation strategies with ITASU is what the FDA name is, which is Elements to Assure Safe Use. And so the FDA required both prescribers and pharmacies to be certified before they could prescribe this drug. And they also had a black box warning for alcohol. So it was quite strict. And I think there have been very, very few prescriptions since it's been prescribed. But there was a controversy whether or not phlebancerin was long overdue or a grievous error on behalf of the FDA. So let me just give you a little more detail about the regulatory history. In 2009, Erin mentioned some of this. It was the first FDA submission and it failed in the efficacy trials, at least one of the primary endpoints. There were two primary endpoints. One was positive, one was not, and the secondary endpoints were positive. The advisory committee met, voted 11-0-no, and the drug was rejected. It was then sold and resubmitted with one new efficacy trial in 2013, using what had been a secondary endpoint now as a primary endpoint. And so both co-primary endpoints were now positive, statistically significant. But the FDA raised still safety concerns. And overall the issue was the drug had modest benefits, lots of safety concerns, and so it was rejected. There was an appeal, and as you can see, the appeal was rejected. So the FDA came back in its complete response letter and said, come back with some more safety data, we'll have an advisory committee meeting and we'll see what happens. So I'll get to 2015 in a moment. This is the efficacy data. The leader of the endocrine division at the FDA actually wrote a perspective, Hilton Jaffee, who actually was a resident, I believe, at the Brigham. And they went over a lot of the data. And here's the efficacy data for co-bancering. And here's the satisfying sexual events. So this is over the three phase three trials. From a baseline of two to three over placebo, there's 0.5 to one additional satisfying sexual event per month. And then you can see again these other scales related to desire and distress, which had statistically significant effects, but very small. Okay, so that's the bottom line about the efficacy. The FDA does what they call a responder analysis. And what they found is after accounting for placebo, about eight to 13% of women were at least much improved. Okay, so that gives you a different way of looking at efficacy than just the average effects. About one in 10 women reported being much improved after taking this drug. So that's an important point to remember. There were a lot of safety effects. The risk of sedation was quite high. Or more than a quarter of women taking the drug had some sedation-related side effect, which is why it's taken at night. There was a high, a lot of concern about the risk of syncope or passing out, but the rates were actually quite low in the pit of a little trials and the same between flu-bancer and placebo. But the problem was these are healthy women who otherwise all of a sudden could just pass out. It could happen right after the med, it could happen a few hours after the medicine. There was a significant interaction with alcohol. And like any FDA, any trial for a new drug, they were limited to healthy women, not on other sedating meds. The duration of the trial was short and there were a lot of concerns about all-slabel use. And we'll talk more about the all-slabel use later. Okay, so in 2015, the drug was resubmitted to the FDA. There was no new efficacy data. There were two safety studies. One was a next-day driving study and they found that there was no impairment in driving the next day. And the other was this interaction, this alcohol interaction study where they gave 25 people alcohol and they saw it greatly increase the risk of side effects. It happened that 23 of those were men, which was one of the problems with the data. But the advisory committee voted 18 to six, this time, yes, to approve the drug. And the drug went on to be approved after actually a lot of debate within the FDA. So what happened between 2013 and 2015? And Aaron mentioned some of these. In 2014 was his FDA meeting on female sexual dysfunction, which I'll get into. And in 20, and after it was rejected in 2013, there was a new advocacy group called Even the Score that formed that did a lot of advocacy around getting this drug passed, which we'll talk about. Now, I'm not saying there's a cause and effect, I'm just saying this happened, okay? So the three items when I think about the role that patients played actually a large role with Fulbanes, when one was this workshop, this patient-focused drug development and public meeting on female sexual dysfunction. And I'll talk more about that in a minute. Even the square advocacy campaign and then the actual advisory committee meeting was written in public testimony, which I'll just give you some thoughts about from my own experience. So this drug development public meeting, this was actually the FDA agreed in the reauthorization of PDUFA, which is the Prescription Drug User Fee Act, they agreed to hold a bunch of these meetings about conditions for which there were not great therapies or for which the therapies didn't really impact the symptoms that patients feel. So they looked at Chagas disease, they looked at a lot of others, which I can't remember, but Fibromyalgia, psoriasis, and then female sexual dysfunction was one of these. So, and you can look at the report if you want online. This is, they wrote a report that came out in June of last year and here are the topics.  And then what are their perspectives on treatment? What have they tried? What do they look for in a treatment that were, what's their ideal treatment? So, this was a summary from the report that I thought would be useful for us to read, for me to read and for you to read also, but I think it's telling. So what this was a section from the report. So many participants and docket commentators described a distinct transition from a prior fulfilling sex life to a total loss of interest or arousal, regardless of their varying beliefs about the causes or triggers of their interest, participant stress and submitting the stress that caused. They described never initiating sexual contact, the measures they took to avoid intimacy and persistent anxiety, okay?  And so when you think, and so you're in a meeting and this is now a voice, a person telling you this, which is very different from you reading in a textbook saying that this bothers women, okay? And I think that's one of the lessons and the dynamic that comes out of these meetings. So there was the even the score campaign.  Here's the logo. They got together many groups that were interested in this topic. The National Consumers League was included, many women's health groups. And what I love is they also had the manufacture of the drug here in the purple, which is probably by design. But they organized consumers and women's group, they were very effective. Letters to Congress, letters to the FDA, personal appeals, op-eds. And their main message was this, was that there are 26 treatments for men for sexual dysfunction and one for women. And it's time to even the score. That was the gist of the campaign. So completely disingenuous, the FDA hated this. And I think it's disingenuous. The problem is there are actually no approved treatments for men for sexual dysfunction. Sorry, for decreased desire in men. And there are many of these 26 drugs for men are different preparations of testosterone. So it was kind of a, it was a disingenuous claim, but it was very effective. And so we had the public meeting, the even the score campaign and the advisory committee meeting. And just a few thoughts on this. There was a lot of written testimony which advisory committee members were asked to read beforehand. And it was extensive. And I'll say in some cases it was really quite personal. And the public testimony the same. And I remember a man coming to talk about how this condition had destroyed his marriage and how he was worried it was gonna destroy his daughter's marriage because his daughter had the same thing.  And in some cases you knew that Sprout had paid for them to come talk. And in other cases, at least one I can remember out of the many speakers it was clear that Sprout did not pay for them to come talk. But there was heartfelt emotion, I will tell you among the speakers. And there was courage to really talk about this topic in front of the entire world. So in some cases the public speakers spoke directly to the male panelists. And this is something maybe we can talk about the dynamic between having the public involved and having people making decisions. But this was very interesting to have someone standing there looking at you saying, you know, think about what you're doing. There was silent but visible disagreement. And I'll tell you initially I had on here snarky comments but my wife told me, you know, put this. But that's essentially what it was, was you would look out in the audience when someone was talking and then you would see you would see the faces that people would make if someone was criticizing the drug. The kinda, you know, the sarcastic looks in some cases hisses. I mean it was a really interesting dynamic when you looked at people speaking and when people disagreed. And of course there was an applause that Erin had mentioned. So just a few, to end just a few thoughts maybe for the discussion later. So what did patient involvement do? It without a doubt increased attention to the drug. It did provide an important point of view. It gave a sense of the level of risk some women were willing to take to have the treatment available.  And it gave a human element for what a statistically significant benefit on average means. And I think that was really the point of having the patient there involved in the meeting. It has the possibility and it did influence the perception of benefit. Ultimately the FDA is making a risk benefit trade off. And if there are senses that there's more benefit then that helps with the approval of the drug. And I think that was the point of having women come and talk. Obviously we'll talk about whether it provides a selective point of view, whether it biases or pressures the FDA and panel participants. And here's, so I'll end with this which I thought, when I thought about phlebancer and I thought this is one of the issues that we should probably talk about. So for the committee, off-label use was a huge concern. This drug is gonna be approved. It's gonna be used by lots of women who don't really need it.  But for a given patient who needs the drug why do they care? And I think when you have patients saying that they're willing to take the risk then the issue about off-label use is really about what physicians are doing. And it's maybe not so much about what patients care about. And so I think it's just one of the issues where having the patient involved gave a different sense of how people should be thinking about the drug. Okay, so I'll leave it there. We can talk more later, but yeah, great. All right, so I guess I'd ask people to hold on their questions. We're gonna have Diana come up and give her presentation and then we're gonna bring both panelists up and have some discussion after that. Thanks very much. I'll start out by saying that my center was one of the few that had speakers at the Fulbanseran hearings speaking about evidence-based medicine and our concern that this was a drug that not only should not be approved but that it would set a dangerous precedent. But I'm not gonna talk more about Fulbanseran. I'm gonna talk more about the general issues of how FDA approval works. What's the role of the advisory committees? What's the role of patients? And I'm actually gonna go a little further and talk about what your role could be, not necessarily as a patient, but as a health professional or as a researcher or as someone who cares about the qualities of products that are approved by the FDA. So to go over this very briefly, as you've heard, the FDA makes decisions on whether the benefits outweigh the risks for most patients. And then the question is who decides what's the, how important is the benefit? How important is the risk? How do you quantify that? And the fact that your voice, whether you're a patient or a family member or a health professional or a researcher can be part of that process. So just to go over the process briefly, companies pay big shots to conduct the studies. The companies pay academic researchers from Harvard if they possibly can. And if they can't get Harvard, they'll settle for Yale or Stanford or someplace else to do the studies for them. But the companies are not just paying for the studies, they're controlling them to some extent. They're doing the design. They're making sure that the analysis is what they want it to be. And then the FDA scientists review the data. They don't review the raw data. That's really an honor system. They're reviewing the data that are provided to them by the company. And that's when, after that analysis is done, that is when, if there's going to be a public advisory committee, that's when it takes place. For a couple of hours, the company's gonna praise their product and present all their data and PowerPoints and say how great their product is. Then the FDA scientists have a couple of hours that to either agree or disagree with what the company has said. Sometimes they've reanalyzed the data and they interpret it differently. Sometimes they reanalyze the data and they agree completely or somewhere in between. And then there's usually about one hour of public comments. At the full bancer hearing, it was two hours. I've actually been to a meeting where it was an entire day of public comments, but it's usually an hour or less. And usually each person speaking gets five minutes or less, sometimes it's three minutes. And that could be time for me to speak. I've done that more times than I care to mention. Or it could be any of you or it can be patients. Anybody can sign up to give a public comment. You are supposed to sign up in advance, but most of them are gonna have a conflict of interest. The vast majority of speakers have been brought there. Who wants to go to Silver Spring now? It's a very nice town. But are you gonna fly in at your own expense to talk about a product? Probably not. But if somebody pays you and that's off in the company or somebody working with the company, such as the eating the sport campaign, if they'll pay your way in and pay for your hotel and train you on how to do this, you're just much more likely to do it. So then after those presentations, there's approximately two hours. And this is all approximate because these meetings are sometimes two days long. They're sometimes a whole day. Sometimes they're less than a day. There'll be about two hours for advisory committee members to ask questions, to discuss what they've heard. And then at the end, almost always they will vote. And there'll be several votes usually, but one of them is taken as whether they recommend or approval or not. Although usually the votes described as do you think the benefits outweigh the risk for most patients? And there also can be votes that are quite specific about restrictions. And you heard about REMs, which are restrictions usually on training. Or what should there be a black box warning? So it can be quite nuanced. So there's one vote and that's what the media usually pays attention to. Now, is this a good product or not? Is this worthy of approval or not? And then there are caveats with full banser just to say this is a drug that has to be taken every day for the rest of your life if you've got this problem. So it's not like Viagra where you take it once whenever you need it. This is taken every day. If you're not supposed to drink, that means you can't drink alcohol every day. If it giving you a risk on being sleepy the next day, it's every day. So they try to have restrictions that are quite clear. Is this something that if you take this drug and you shouldn't do something else for full banserin, part of the training is, I was told about a three minute video that the doctors have to watch on how they should tell their patients you must not drink alcohol. Then as part of the meeting, oh, so to consider this, how can the patients or the public, they can express their views during the meeting. They can write written comments that are provided to the advisory committee before the meeting. Some advisory committee members are gonna read those comments. A lot of them are not. The joke about FDA advisory committee meetings is the people who are best prepared on the panel are the ones that had the longest flight to the meeting. They're the ones and it's the local folks who might not have read much of anything. The media also expresses their views. You saw some examples of that. And the public can also talk to Congress about what Congress should do. Should Congress get involved in the case of full banserin, for example, and in the case of the drug for Duchenne's muscular dystrophy, quite a few members of Congress have in fact contacted the FDA saying our constituents say you should approve this drug. And so Congress does that, does tell FDA what they think and usually that's based entirely on what their constituents have told them. If their constituents are patients, that's one thing. If their constituents are faculty members at Harvard, that is something else. But whoever their constituents are, Congress will usually pass along those concerns. And then FDA makes its decision whether to approve the product or not. In our case, our center frequently speaks at these public meetings and we are the ones talking about the science during those public comment periods. We talk about how generalizable the data are. Do the data include enough women? Do they include enough people of color? Are they including people over 65 who are not in the case of full banserin but in other cases are the people most likely to be using the drugs and also are gonna be the basis of whether Medicare is going to reimburse for it. Are the data long-term enough? Are they studying a drug for three months that's gonna lower cholesterol for example and it's gonna be taken by people for decades? And are there enough patients in the study? Not just enough women or people of color but enough patients at all. In some cases there might be thousands of patients in the study. In other cases there might be 60. I've seen studies with 24 and actually the one with the Shenz muscular dystrophy as you know had 12. So these numbers really vary and we talk about that. And then we also talk about how meaningful the outcomes were. And that should be meaningful not just to patients but to scientists, it should be both. If you're looking at what are called surrogate endpoints or biomarkers, we're kind of used to that but it's not the same thing as looking at length of survival in the case of a cancer drug for example, looking at quality of life, looking at a number of days in the hospital. And now with the FDA being pushed a lot to approve drugs more quickly, there are various fast tracks or expedited reviews which often rely on just one study instead of two clinical trials and often rely on biomarkers such as cholesterol levels, glucose levels, bone mineral density and tumor shrinkage. We're so used to the now that we think of them as health measures but they're not really meaningful health measures to the patient unless it affects their actual health. And how can we be sure that a biomarker is going to translate to improved health? A recent study published in JAMA Oncology and authored by someone from the National Cancer Institute found that most new cancer drugs that were approved because they shrunk tumors but when they did studies after the product was on the market, most of these did not improve survival at all. I'm gonna just, if I can figure out how to do this, gonna show you just an example of not just an ad but the way that patients are being... For adults with an advanced lung cancer called squamous non-small cell previously treated with platinum-based chemotherapy, it's not every day something this big comes along. A Chance to Live Longer with Optivo, Navalabad. Optivo is the first and only immunotherapy FDA approved based on a clinical trial demonstrating longer life for these patients. In fact, Optivo significantly increased the chance of living longer versus chemotherapy. So I won't do the whole thing but I feel better already, you know? This is a chance to live longer, I will take that. I don't have lung cancer but it's just very moving and this is part of the way the emotions are used of patients who are told this drug, whatever it is, will give you that chance and that chance might be a 1% chance, it might be a 90% chance. They're not necessarily quantifying the chance. So as a result of this kind of emotion we've seen this a few times where the advisory committee very frequently says, yes, we want to approve this product, FDA should approve this product. Often the physicians on the panel are saying we need more options. If there's a patient representative on the panel usually the patient's gonna say we need options. The consumer representatives often are also saying that our center actually did a study comparing who votes what way. For the most of the time at least 80% of the drugs and advisory committee and devices they recommend approval. And then they sometimes compromise when the data aren't very good with some restrictions such as training requirements but a lot of times those restrictions are unenforceable. I'll just give one example, Accutane which is for Acne had a restriction that it could not be given to teenage girls who might be pregnant and that if you wanted to prescribe it for a teenage, well actually any young woman, you had to have, she had to have a pregnancy test. And what they found was there were doctors that were changing the name of the women to make it look like they were boys and they were checking instead of female they were checking male so that they wouldn't have to require a pregnancy test and they would prescribe a drug anyway. So, gee it's hard to believe doctors would do such a thing but sometimes they do. So when you have a drug approved that's still really experimental and you don't know exactly how safe and effective it is the patients are basically paying for that drug to be guinea pigs. The company has little incentive to complete any post market required studies and these ineffective new products can cost a lot of money. With these cancer drugs that are now known to not improve survival, the most expensive of them was $169,000 a year and most of the average cost was about $95,000 a year. Just quickly to say device approval criteria are even worse. These low risk devices aren't tested at all. The moderate risk devices are not tested in clinical trials. It's only the highest risk devices such as a pacemaker, an artificial heart and infusion pump. These are the ones that are tested in clinical trials and only one, not two. And sometimes those trials lack a control group. This has life and death implications. Almost half a billion of these moderate devices that weren't tested in clinical trials were recalled as high risk a year ago including the contaminated alcohol swabs that killed this little boy.  The gold standard of two double blind randomized clinical trials studying patients' actual health outcomes is an endangered species. The outcome measures are often surrogate outcomes and the FDA often hears from industry and they hear from patients affiliated with industry but they're not hearing from you. So we want you. We want healthcare professionals as well as patients and ethicists and researchers to bring their perspectives to this process. And if you have no financial ties to the product that's being considered and if you focus on the evidence, you're almost a unique voice in that room but a very, very important one. And as they used to say in the 60s, if you're not part of the solution, you're part of the problem. Thank you very much. All right, so I'd like to invite our panelists to come up to the front and I want to engage our first set of predetermined questions Okay, so I want us first to consider citizens' roles at the FDA and I found a nice review article about the ways that public can participate in democracy and there are three questions of institutional design related to citizen participants and governance. The first question is who should go to a citizens you're going to engage with the FDA or any governance? And so this is kind of a spectrum of different ways that citizens can involve the FDA. So with a more inclusive side of the spectrum, you have the kind of open self-selection process which I take it as kind of what we are now with regard to patient participation with the FDA advisory. But you can also imagine that there is an open targeted recruiting process where you try to make sure that there are particular people there based on how you advertise or do you advertise the availability of these committees too or other ways that you can imagine trying to recruit particular people, you can try to randomly select a population through some, again, lottery process where you can designate lay stakeholders or professional stakeholders who are educated, maybe unpaid citizens who have a deep interest, think about people who are involved in a neighborhood association or a school council or professional stakeholders with those people who are paid for those conditions. The second major for us is how are the views expressed in those settings? So again, another spectrum of different activities in ways you can imagine expressing views. So on the least intense side of the spectrum, then people can just, there could be interaction between public and government and people can just listen as a spectator or they can express their preferences which I think is again where we sort of are right now. You can imagine a process where we develop people's preferences by providing them materials and organizing them into smaller groups and having them discuss and consider alternatives. And then you can imagine the more intense situations where people are aggregating and bargaining with each other and coming up with a single public voice that would then be expressed. And then the final section or the final prompt here is what is the impact of public participation? Do people simply get public the personal benefits of being involved, learning about the process, feeling their civic obligations to talk? Or is there some kind of a community of influence where the members of government are actually trying to get information from? Or is there some more formal advisory and consulting process where opinions need to be more formally listened to even as far as like a co-governance process in sort of a classic New England town hall meeting where the citizens all come together and with the people elected in front and make a group decision about what they want? So thinking about those different aspects of institutional design, I just want to have Pauline and Diana comment on what they think, where they think we are right now and where they think we could be and then open up to questions and comments from the audience about where you guys think the optimal situation should be. Okay, sure. I'll start. I will just say one of the things that our center is now doing is that we are training patients to understand science. Our hope is that when patients participate as advisory committee members or are speaking, that they will bring their perspective, which is unique to them, but also very important. One of the things I think that's really important about patient perspectives is that, it's one thing to say, as Walid said, it's one thing to say, this is ruining my marriage. It's another thing to hear somebody talk about what that means. So we think that these are very important and I'll use an example of antidepressants actually. There was a public meeting on antidepressants trying to get certain antidepressants either off the market or not allowed for children. And the issue was that some kids were killing themselves or becoming very violent as a result of taking SSRIs. And they found that in the data, it said that the kids were getting volatile or that their emotions were labile. And one of the examples of a child whose emotions were labile was a teenage girl who set herself on fire. Now personally, I would call that a suicide attempt. She survived. But when she was asked, why did you set yourself on fire? She said, I don't know. So they called that labile, you know. And that doesn't convey, that's an example where the data are not conveying what's happening. So we think that these perspectives are really important but they need to be in the context of understanding that just because this drug was good for you or bad for you, doesn't mean that that's true for most people and understanding what a significance test means. So we've explained to people what a double-blind randomized clinical trial is and why that's important. We've also explained how sometimes you can't do that. Trying to teach patients to understand the science. Yeah, I would say the more I thought about this, the more I realized I didn't really know what the answer is and maybe that's why it's good for a seminar like this and to hear other people's thoughts. But I came up with the fundamental principle when I thought about it, which is we want to allow any person who is willing and able to have the opportunity to present. That may be one principle you might think about. The issue is I think like everything else in public involvement in the US, the more money you have and the more organized you are, the more likely you are to be heard. And that's a reality at the FDA. It's a reality in Congress. It's a reality everywhere. And so I think all public participation has the same problems. But fundamentally you want anyone who wants to present to be able to present. And advisory committee members, we know people who are speaking sometimes are paid. And so I think if you can't have a broad representation of everyone, then at least you know who they're representing. And so that's the way I think about it is let's let everyone present who can, understanding that not everybody will and as long as we realize that that's okay. That's one way to approach it. There are a lot of other ideas about randomly selecting people like juries. Maybe everyone who's in the clinical trial should agree that at the time of approval, they could be randomly called if they're willing to present their experiences. I just wanted to add one thing and that is that, that we know that one of the tricks at these FDA public meetings is that because public comment speakers are asked, basically do you have conflicts of interest? We've seen situations where a patient will say, I paid my way to get here today, but they're reimbursed the next day. Something like that, certainly one of the things that always amuses me are physicians who say, I have no conflicts of interest, but my institution gets $5 million from this company. I mean, they don't say that part. So conflicts of interest are not just in the eye of the beholder, they're also easily disguised if that's a goal. And that's a real problem, I think. So yeah, I mean, we'll now open it up to sort of commentary from the audience and I see Dan's hand. I've been on the receiving end of these public comments. I was on a three-person committee that had supposed to list the three days of the stuff. And it quickly became apparent what was going on. Passion patients and after a while, we could figure out who paid to them and who wrote with their lines, all the stuff. So of course, it had made no impression on us. But we couldn't say that because we didn't have any evidence. It was perfectly obvious, but we dismissed it. Now, I think you both said that these advisory committees are being swayed. I just wonder what you think is going on in the minds of the members of the committee. Don't they see through this? And if they do, how could they be swayed by it? Do they feel that they have an obligation to be swayed because of how other people will be perceiving this? I mean, who's not in on the con? I think I can only speak for myself. I will say that I agree that I think everyone on the committee knows what's happening. You know, the question is, is there anything to be gained still from what people say during the meetings? And I think knowing what you know that they were paid to come, is there still some value to it? I don't think people are being conned. But when someone stands in front of you, looks at you, says, my life has been ruined, this drug saved me. That does something to you, even if you know that they were paid by the company to come tell you that. Yeah, and I would say just, and then imagine that sort of times four hours of sort of consistent five minutes at five minutes in five minutes. And you get a sense of what the Ed Eplersen committee was like also with the committee members because the room was too big. There was no big enough room at the FDA. I think it was at a held at a hotel. And so there were literally a thousand people in the audience who were kind of watching the interactions with the advisory committee. And I do think that it is important to listen to patients. And I think that they provide a lot of important color to the data as Walid was saying. And can provide some insights on various topics. But at some point, is there a sort of diminishing return from the same point of view being said over and over again? And is there some benefit to trying to make sure that there is other points of view expressed that may not be able to reach there, maybe through some Skype or whatever else we were able to set up to try to recruit as broadly a coalition as possible? And I would just like to add to that. I haven't been on an FDA advisory committee, but I am on the Medicare advisory committee. And it's hard to speak up when you're the only voice. And when you have a roomful of people cheering when somebody says this drug is great and booing as they did for the Duchenne's drug, booing when you say something they don't like. Not everybody has the fortitude to say what they really think, let alone vote, the way they like to vote. So it really, you know, I'm pretty tough. And I've been a lone voice in the wilderness at a lot of these meetings. And I will say at the Duchenne's meeting, there were 50 public comment speakers all saying this drug is lifesaving. You must approve it. And one person saying the science isn't there. That one person worked for me. And it was a horrible experience for her. And when she walked out of that room, some man followed her out swearing at her. And we had another staff member who was with her who was, you know, walking with her. And this man threatened him and said, I should punch you in the face, old man. I mean, this was a volatile situation.  But it's hard to have staff people who will do this work if they're going to be treated that way. And we were the only voice. And for the Phlebanceran meeting, I think we were one of four or five, even though the other colleagues who agreed with us were all the women's health experts in the country on FDA issues, including the former head of the Office for Women's Health at FDA. So although there were a lot of nonprofits in this, even the SCORE campaign, it did not include the foremost experts on how FDA makes decisions on treatment. And even so, it ain't fun being one of four people. Hi, so in full disclosure, my son is one of the 12 kids in the Etiplerston study and I was at the ADCOM and I was one of the 51 speakers who spoke in favor of the drug. And I just have a question for everybody in the room. If you're shopping for a product, who better to hear from than a person who has used that product? And so I kinda wanna speak directly to you about the member of your group that was there and was the only one who spoke against the drug and about the caliber of the folks who spoke for the drug. I think when we're talking about patients, we're talking about patients who are experienced on the drug. We were very careful as advocacy groups to make sure these weren't just patients who were pleading for something. Duchenne is fatal. We know Duchenne is fatal. That's not what you need to hear at an ADCOM. What you need to hear is that this drug changed the course of the disease for my child. And 11 of the 12 children in the trial were represented not only by parents who potentially were seeing what they wanted to see, but by their treating clinicians who had treated these children from before they were on the drug to during treatment on the drug and currently. And these were world renowned Duchenne experts. I guess I'm feeling for you that you were one of the only negative groups in the room, but I think the reason for that is because there was great support for the drug. And why would somebody who had no experience with a particular drug want to speak during an open public forum? So I guess that's where I believe patient voice is important. Clinician voice is important. Real stakeholder voice is important. Trained speakers to look at science. That's something that occurs elsewhere. So I think the right people were at that meeting speaking at that right time. And I'm very sorry that you're, that the woman who you sent was harassed. I don't think that's representative of our community, but my son included was one of the people that made a lot of noise at that event. And I hope you understand that it was incredibly emotional because this is saving their lives. Yeah, well, thanks for that. Did you want to come? Yeah, no, of course we understand that. And any parent would understand that. I guess what I would say about our center is that we believe that if the company's data were that good, they would have provided data from the more than 90 patients that they were required to add to the study that had already been in the study at the time of the meeting. So why only talk about the data for the 12 patients when they already had data for 90 additional patients? So I'm really speaking from a scientific point of view. If you only have 12 people in the study, there is a difference in terms of how well people react to certain drugs. Two people of the 12 actually had a very bad experience and the data will look a lot better when you take those people out in a post-hoc way, but that's not the scientific way. And again, I don't think we have the time or opportunity here to kind of get back into the weeds of the data themselves, but go ahead, Paulie. Yeah, and this just tells you the difficult job the FDA has. But I mean, patients have a very different experience with the drug than people looking at the data. I mean, I think that's what the bottom line is. And I think as advisors, it's important to understand both sides of it. I mean, a patient, for an individual patient, it works or it doesn't. You know, the issue is the FDA is not really deciding whether for you the drug works or not. And that's the tension. You know, but I think that it is valuable and at a pluricent flow of answer on whatever it might be gets at this issue. I think it is important for advisors to understand that tension in the course of their evaluation, even while they look at objective data. But I think that's important. We have some comments over here, Paulie. Thank you very much. I'm very interested in what the two or three of you think about this question I have. Why would the FDA hold a public hearing on studies that included such small numbers of patients? That's number one question. Number two question, isn't there some government regulation that should prohibit television advertising for medications that have no scientific background? Thinking particularly of that drug that improves your memory. I'm in that age group analysis. So I pay attention to that. But I tried to get the original studies that show this dramatic increase in improvement in memory and the data are not available. Why does the FDA or the government allow that kind of advertising? So again, on your first question, I don't wanna get into the weeds of why choices were made. I mean, this is a rare disease. And so it's hard to, you need to be flexible in terms of how many people you expect in trials. But I think it does, I think I would like to try to rephrase your first question to ask, well, why is the FDA holding these hearings and what are they expecting to get and what contribution are they hoping that the patient voice will make? And maybe I could rephrase that slightly and pose that as well as if you did wanna touch on the post-approval use and the commercial speech advertising issues from drugs based on FDA decision-making. Sure, and I'm sympathetic to this question. It used to be that the FDA only had public meetings when a drug was really controversial, where it could go either way. And normally, I've never known them to ever have a public meeting for a drug that was only studied on 12 people without a control group. I've never known that to happen before. So I think it's a sign of the times and it is an issue that I want people to think about because it isn't just this one drug. How many drugs do you want to be considered when they are studied on a small number of people, even though Duchenne's is a rare disease, other companies who have made products for the same condition had 100 patients or more in their study. So it's not like you can only get 12. So the FDA is under tremendous pressure from Congress now and this has been going on for years and they have been lowering their standards. That's why it used to be two double-blind, randomized clinical trials now, it's frequently one. Those of you who are scientists, you know science depends on replication. If you've got one study, you don't have replication. So the standards have gotten lower. Every time FDA approves a drug, they put out a press release cheering how great this is. Somebody said that recently they've been approving 96% of the drugs that they're looking at, which would be great if these are all effective drugs, but not so great if they're based on preliminary studies. So, and the advertising thing, I'm not a lawyer, but the issue, we are one of two countries in the world that allow direct to consumer advertising. New Zealand being the other one and supposedly it's a free speech issue and that is why that changed some years ago where before, it wasn't allowed and now it is. There's a lot of questions, I'm happy to. That's fine. I'm happy to sort of take on more questions as well. So, yeah, go ahead. Thank you very much. I'm Charlize Norman. I actually was on a medical devices advisory panel for about six or seven years. And so I wanted to comment on the earlier comment about listening to the public and the fact that all the advisors, at least on the committee I was on, appreciated the comments, but weren't fooled by some things. And I will state that I think most people didn't even think that if somebody was paid by a company, a patient to come and testify, so to speak, we didn't think that they were lying. We knew that they really believed that. And so I just wanted to make that clear. It's not that we're thinking, oh, they're all just doing that. And I think it's very important, part of the process for the panel members to hear that, because it is true, you look at the data. I'm a statistician, by the way, so I'm probably one of the few that don't have any conflicts on the actual advisory panel. I wanted to ask a question along the different lines, but related to the patient's voice in the product approval. And that is, I wanted to get the opinion of how you feel about the patients providing more input or more sway into the types of outcomes that are collected. You, Diana, refer to that in terms of quality of life. And we know these are competing risks, because if you want to expedite a clinical trial, you use a biomarker, because you can measure that more quickly than an endpoint. But I personally would think there would be the design and the usefulness of many of these regulatory trials would be more raised to a higher level if we had those types of outcomes. And so I was curious of your response to that, whether you think there's more hope for that and whether or not that's going to happen in the future. Thank you. Well, I'll just say, I mean, that was the point, and you probably know this of the 20, 25 meetings that the FDA held on patient-focused drug development. This is a commitment from the FDA to figure out what are the symptoms for those diseases for which there are no treatments or for those diseases in which the treatments do not actually make you feel any different. What are the outcomes that would be of benefit? So I think that those have been valuable. And I think, certainly if you think about phlebancerin, there were a number of different outcomes, very controversial, but this issue of satisfying sexual events is a patient-reported outcome. And I think it's hard for any panel member to really conceptualize what that means, and that's where really hearing from a patient can be helpful. Again, even if you know they were paid by the company, they were probably one of the 10% who responded, and you just take that for what it is. Yeah, and I agree that there's nothing like hearing from a patient to know what the benefit really is and what that feels like and what the risk really is and what the complications are and what that feels like. And what really concerns me about what FDA is doing now is because of the nature of who comes. And I agree with you, it's not like they're lying, it's just that you're only hearing one side of the story because that's who's there. So what concerns me about what the FDA is doing is that they are paying a lot of attention to the benefits and to patients saying, I'm willing to take any risk and I don't care for this potential benefit. But I hear from a lot of patients who thought that until they had a bad response and then they're really angry and saying, how could the FDA allow this on the market?  I'm struck by the dilemma that the FDA and the committees have a herculean task. One of the things I've been looking at with a group of people here, certainly in forensic psychiatry, is how do people make decisions in setting of uncertainty? And we found that judges and jurors who are to evaluate the facts don't base their decision-making all the time. In fact, mostly in rational ways. The emotions always play a role. And so Dan's question I think is that it's really hard even when you're sitting on a committee and you're faced with a bunch of people who have a bias, oftentimes, to not listen with an empathic voice even though you're trying to listen to a statistical evaluation. And the question I have is how much effort, I think what you have an epistemological problem set up by virtue of having the public come. And that is, by and large, as you suggested, the public is coming with an emotional response and advocacy position with an expectation. And the expectation is that they will win. We've seen a lot of this in the current political scene here. Strong voices, big voices, bombastic voices with a single mind sway a lot of people. And I think that they come with an expectation. If we plea our case, how can you not be empathic? And so they're looking for an emotional response to what's going on. And the committee is sitting there trying to make a scientific evaluative decision based on what are considered facts. And that can often be discussed in a variety of ways. I wonder if anyone, whenever there's a hearing, sets up directly before the discussion takes place what the parameters are, that in fact people are coming with different perspectives. And here is what the committee does. We're happy to listen to what you say. But clearly we know you have a perspective and we honor that, we trust it, we wanna hear it. At the same time understand that our decision making is gonna be based on the following. Does that happen regularly or does that just not happen? Yeah, I would just quickly say that I've been to many, many advisory committee meetings where if somebody started to cheer or clap or boo, they were told by the chair, there will be no clapping, there will be no booing. And if you continue this, you're gonna be escorted out of the room. And I've never until recently had the experience of the kind of meetings where it's allowed for people to cheer and boo, because I do think it's very harmful to the scientific integrity of the meeting. And yes, people are emotional, but there are a lot of things in life where you're not allowed to cheer and boo loudly. It's not a football game. Why don't you just remain young at the room first? Oh yeah, they do do that. Yeah, they'll say that, but then it doesn't matter if they say that, if the emotions take over the room. Good, yeah. I just wanna say one thing just for the sake of just thinking broadly, and I guess what I'll do is push back against this idea that we are objectively without emotion looking at facts and numbers. Because we are on paper, but ultimately it's still about risk-benefit. And the thing about the patient voice, and I'll just go back to Phlebanceran, the fact is that it increases it by one sexually satisfying sexual event per month. That is a fact, but is that a benefit or not? And is that a good enough benefit or not? And so that's where we'd like to think that we're just looking at numbers, studies, what's the outcome, but ultimately you still have to decide a benefit. And even MI, heart attack may be of different value to one person versus another. I think just for the sake of... And on just kind of, and I'm sorry, and on just sort of a more sort of specific answer, like I think that there is right now very little framing of the advisory committee session and the patient voice and the advisory committee session to try to, you know, relatively little is sort of charged to the advisory committee members about what it is that the patients are supposed to add or what they're supposed to get out of it. And so it can, you know, I'm sure it affects, therefore affects different people and different thoughts in different ways. Yeah, and I'm sorry, I just wanna add, you know, the other side that they're not hearing is, you know, if a patient took the drug and drank alcohol and they thought, well, I just had one glass of wine and then, you know, they went, they were driving and, you know, they passed out and died.  So that's the part you're not hearing. Yeah, I think that's true. And I was just, I was gonna react to Erin's comment. I think that is true. We weren't given a charge for how to think about the patient voice. And, you know, I was thinking more about this. I was thinking, should the patient speakers really be there when we're voting and deliberating? I mean, because part of this is the interaction between and you can hear someone's opinion, but maybe you don't need them looking at you, you know, when you're trying to decide, yeah. So I have other discussion questions, but we also have only about five minutes left. So I mean, I think we still have some people who want to express opinion, so I think we should just keep going around. So, Mike. The value in maybe delaying when the vote actually occurs because it's so emotionally charged or a decision about, so what component, how much should today's proceedings influence our overall decision? Has there been thought about that? The one thing I would say is, remember, the FDA is not deciding that day, but it is true, the committee is voting, but it is a good idea. I mean, maybe it should be separated by at least a night to sleep on it or something. I mean, there would certainly be complaints about this is a public meeting, everybody's entitled to be there and we should see the vote. But you're right. If the vote were delayed, it could still be on line or something.  There's a question hand in the back. Yeah. Hi, I actually had a question for Dr. Zuckerberg. Earlier you had mentioned that you're starting to talk with patients and do some education with them. I was wondering if you could talk a little bit about how you're doing that education and if you find it effective. Yeah, and I have some information I can hand out afterwards too. We got a grant from PCORI, the Patient-Centered Outcome Research Institute, which was created by Congress as part of the Affordable Care Act to look at research comparative effectiveness research and evidence-based medicine with the idea that if you had better evidence, healthcare would be more affordable and insurance would be more affordable. But so we're training about 100 patients, it's open, anybody can apply. They have to understand that the goal is to help people who want to be patient advocates and want to learn about the science. And those of us in this room, we take a lot of things for granted in terms of what people know and what people don't know. So we've learned, we evaluate our workshops to find out what they've learned and what they don't. And they're all learning and some are learning more than others. But they certainly have an appreciation. I think what's important to us is they get that appreciation of why their own personal experience is important, but it is not the only thing. And how important it is to understand that a really horrible side effect that only affects 5% of the people using the treatment, if it kills them or ruins their life, that's something that really has to be taken into consideration. And that's why you need a decent number of people in any study to get those rare but horrible safety issues. Yep. First of all, thank you to the panelists. It's been a very interesting talk. I had a question regarding the Public Advisory Committee in terms of the sense of intimidation you felt some people speaking against newly sort of controversial drugs might feel. Do you think that there would be any value in there being the possibility to present one's thoughts in person but in a private setting, sort of just with the panelists so that the sense of intimidation and fear against speaking against popular opinion can be eliminated? I mean, it's an interesting idea. Are you suggesting that someone might come one at a time rather than having a whole room? I mean, that would change the dynamic and that would still allow individuals to participate. And that's an interesting idea. I hadn't thought about that. I mean, what I'll say is I hope that any person agreeing to be on an advisory panel is confident enough to know that they can look at the data and also take in the patient perspective. But I know we're all human. But that is an interesting idea that I hadn't thought of. Yeah, I mean, I think that's an interesting idea too. I think some of the other ideas that were expressed today on that vein were also interesting. I mean, I think that if the things continue in the, if the phlebancering experience and the other experiences are signals of where this is now going to be in the future. And I think that those sorts of ideas may need to be more seriously taken up by the organizers of the committees and people will need to try to come to grips with trying to direct the passion and information that can be obtained in a way that makes it most effective. I mean, it's also theoretically possible that the sort of the model from the phlebancering provokes the exact opposite reaction in the way that Dan had explained earlier of people thinking that this is all sort of a setup. And so it actually works in the opposite direction. So I mean, I think that ultimately if this starts to become more and more of a social trend, then I think that we'll need to start to think about it and analyze it and consider these other options. So we're running towards the end of our time. I wanted to thank the panelists for coming and thank all of you for being here as well. And just to remind you about the upcoming ones in October and November. And be happy to hear any feedback as well on the format so that we can continue to take public's opinion and adapt it into our expert decisions of how to handle these decisions. So thank you very much.