Hey, hey, everyone. This is Carlos and the founder and CEO at Product School. Today I'm here with another prop star. That's how we call people that work at Product School. Her name is Mariana Abdala and she's the VP of Product at Product School. Welcome to the show. Thank you for having me, Carlos. We've had so many of these conversations private. So excited to do these for the first time in public. Yes, absolutely. As am I. So Mariana, the title VP of Product at Product School is very meta. Let me just quickly explain what that means and of course I would love to learn more about your background. Basically we teach product and best people to teach are practitioners in my opinion. So all our instructors, Mariana, are VP's of product or were VP's of products at other organizations. In her case, she worked at companies like Dending Club or William Sonoma. And yeah, let's see. I would love to learn more about Mariana, what was kind of your journey and then what kind of led you to now help others build product instead of build it yourself. Sure. So my journey in product management has been kind of a winding road. And I never thought I would go into product management. I didn't know what it was 15 years ago when I took my first role as a product manager. I didn't even know what a product manager was supposed to do. So in short, I learned on the job and made many mistakes along the way, but also found good mentorship from former Yahoo and eBay execs who had founded their own companies and were now the CEOs of startups that I was working at and could have been there for the first time with Eyes Wide Open. So along the way, I got a lot of help, but it made me realize during throughout my trajectory in tech and in product, it made me realize how people need coaching and they need mentorship in their careers and product school has been transformative for me because that's one of the things I love to do the most is to instruct, teach, coach and mentor product professionals or aspiring product professionals. I didn't have an MBA background. I don't have a business school background. I don't have an engineering background. So I learned on the job and I learned from other talent that I met along the way. And yeah, and it's been a great journey. Product is a fantastic career to go into for so many reasons. So one of the things that we learned as we continue to grow the business is that this is not just an individual problem. It's been a lot of people like you earlier in their careers where they realize that they needed mentoring or training to continue growing. In many cases, they would pay out of pocket to do it and find different avenues to educate themselves if they couldn't have the right mentors that work or they needed something else. But to me, one of the breakthroughs as we evolved as an organization was to realize that, in fact, a lot of the students that were taking our public trainings were being remorse by their employers. There were some sort of vocation stipend or budget that these companies would give to employees to go and pick the right training for themselves. That was around five years ago. So we started a company ten years ago mostly focused on what we would call B2C public trainings. And then around five years ago, it made a huge push to start training companies privately. And that was the classic bottoms up approach where we have a lot of those students who work at those companies. In many cases they're being remorse and they would introduce us to whoever made the decision to approve that budget. And it was funny because a lot of times those decision makers didn't even know that they were our clients. They would just give these people a budget and then we'd go and say, Hey, do you know that this five, 10, 15 people in your company that are swiping your credit card? They're like, Oh, no, well, but yet it's happening. And I think that's the best introduction to a company. Instead of just knocking the door called you just to explain that it's already happening. And also these employees where our students are having a great experience. So it was a very straightforward path for us to then discuss the opportunity to do something more private just for the employees of a company, but also custom. And that's the piece that I want to discuss with you because we don't know that every company thinks of product in a slightly different way. And there's no one size fit all solution yet. Shouldn't be an excuse for a company to say, Well, then I'm not going to try to learn some of the best practices or I'm going to continue doing my thing. So how do you think about custom? Like in your experience working at these different companies, what was the main things that you saw from from product or to product or the main things I saw were that there were highly competent, talented and motivated individuals who were working as individuals and were not reading from the same book, essentially, or reading from the same singing from the same song sheet, if you will. So there were people coming from all walks of life and they're all trying to do the best thing for the company, do the best thing for the user. But the how was not aligned across the company. And there was a lack of, I think, organizational empowerment to really do all kind of be doing the same thing for the same reasons. And that's because, you know, we are all unique individuals who come with unique skill sets and talents. And when it comes to product, we all are shipping different products. And there are different types of product teams within these companies. There are consumer facing products that you and I as, you know, customers utilize. And then there are B2B products. And there are internal products as well with internal employees as the end customer. So it's to say that for all of these different flavors of product teams, there are at a high level, these kind of tried and true frameworks and tools that every product manager and their stakeholders and their partners need in their toolkit for the day to day, so that we can all be working with the same types of frameworks and tools together. And it's, you know, it yields higher productivity. And then the customization pieces. Well, who are we solving for? Who was who is the user? Who are the customers? What are the problem spaces that we're going to together solve for? And that's where it starts to get custom. And to me, I've heard that word custom many times and as an excuse to not put in the work. It's always much easier to criticize a standard framework, a standard tool, a standard book and say, well, my company is different. Yes, every company is different. Every individual is different. And I believe that learning more about best practices and give you a head start. I also agree that what work for one company doesn't necessarily need to work exactly for the company, for the other company. But yes, having a good understanding of, okay, what are my options? What are the frameworks, as you mentioned, or templates or best practices and how other people were really smart, went about fixing something and hopefully identify patterns. Because at the end of the day, it's a lot of pattern recognition. And then think the art of tweaking those details. One thing that I also noticed is that this product thing that you know, I've been doing for a long time, is not just a tech thing. If anything, tech is now not really an industry. Every company is becoming a tech enabled company at least. So we're noticing more and more organizations in financial services, or healthcare or consumer package goods that used to think of product as something physical. Now hiding a lot of having huge product organizations that are focused on data, data processes for the internal productivity, data services for their clients and so many other options for them. So when you go and train some of those large organizations that, you know, traditionally don't come from what I quote, high tech, what are some of those common patterns that you see in terms of how they think about product and the opportunity that you see to help them act more like a tech service without losing their identity. So I will also provide some background on, you know, the most recent company I worked at was William Sonoma. And when people think of William Sonoma, they think, oh, it's a furniture retail company. It's just a furniture company. William Sonoma on top of being a parent company to many other brands who are all kind of working in some way autonomously, like Pottery Barn, West Elm, Rejuvenation. These are these are all kind of individual companies kind of coming up into William Sonoma. So here you have many furniture companies as they might be seen by the consumer all working together. And it doesn't sound like there's any tech there, or at first. But it's mind blowing, the tech muscle that William Sonoma has had to build over the last 10 years. And they've gone through several digital transformations, and they will go through some other digital transformation. And that's, I think, just a buzzword to say, how do we continue to be relevant, be innovative, attract good talent in to enable our technology to reach more customers, delight those customers and generate revenue, especially in companies like retail, or even, and this extends to, you know, health care, finance, insurance, these are companies that are to your point, Carlos, they're not tech companies, but they have tech enablement running throughout their organization. And if they don't, there's tons of opportunity and low hanging fruit opportunities. I'll give you an example of one of my recent clients that, you know, that came through product schools enterprise program. They had their human resources division, take product school training. And you think, well, why are why is like, what is HR going to do with tech? What they're going to do is start thinking leaner. There's going to start organizing themselves in a different way. They're going to start thinking about not just the processes that they have to put into place like payroll, new employee onboarding, promotions, like, you know, compliance training, they're not thinking about those things, they're thinking about what do our people need? Because that's their user. And think about how much more fulfilled, not only they're going to be, you know, building better products and really empathizing with who's using their products. But how much more delightful will it be for those employees at that company to be heard by HR. So and to be seen and to get problems solved by HR. So it's all about how we're doing the work and and how we can be innovative, even in these kind of more, you know, day to day processes that we have at our at our companies. And it's not just about like, Oh, how can we, you know, code things faster and automate things. It's really like, how can we transform our organizations to be more efficient, think more about the end user. And in the end, just deliver more value to the company. I think that is the key work. At the end of the day, organization is meant to return value to their shareholders and to their customers. And obviously customers first, because if the vast customers don't see value, they're not willing to pay for it. And there's nothing else for shareholders. But I think that connection between those business outcomes to what the product team can do is key. And sometimes missing, I won't say in many cases, when when we started working with some of our enterprise clients, and they went through previous transformations. I think that war didn't do justice sometimes and created some confusion. Some companies would hire a digital transformation leader or would say, okay, we're going to transformation. And they would, I mean, good intentions, but ultimately, they would apply that just for the engineers to use some sort of less waterfall methodology. But still, product wasn't considered strategic arm for the organization. It was still under IT and their tech and still supposed to be behind the wall waiting for marketing and sales teams to close a deal and then go and deliver. And the shift that we are seeing and we are contributing to is elevating the product role as a business function that can really drive business outcomes. And I think the example you gave of, hey, first of all, this is not just, product is not just for tech companies, the same way product management is not just for product managers. I think it's key is you can apply that same mindset to different functions and in the case of talent acquisition, find how draw a line between how you can improve certain processes to obviously provide a better candidate experience, but ultimately increase revenue because you're going to be able to close candidates faster and hopefully you're going to be able to retain them longer. Right, exactly. Sometimes it's as simple, like, and you mentioned this, sometimes it's just as simple as reorganizing and shifting the roles around so that you have people with a product mindset sitting closer to the business and being that Lynch pin with the technology teams. Sometimes it's, you know, bringing in additional frameworks or best practices. Sometimes it's going deeper and understanding, you know, but having a little more definition around what the products are and who the users are for a specific company. In the end, those are kind of examples of what we say when we say customizations, like those might be areas where we double click in to see what, you know, what how we might customize product training, our product school training for these companies and their needs. And that is how it's different from digital transformation. Because I feel like digital transformation from what I've seen, having been at a company that is not a tech company, but has a strong technology division, like William Sonoma digital transformation oftentimes means, let's get out of the business of having an IT org and, and, you know, have more of a kind of tech, like a tech group or a tech organization. And there might be kind of more high level operational things that are being done. But this is a lot deeper. We're going deeper. And we're going, we're going also farther across the organization. Well, let's talk about that. And the depth and the length of this type of new transformation or or upskilling for teams. When we talk about depth, and you tell us more about what is the approach that that you are taking when when training large companies that have hundreds or thousands of employees distributed across the world. Yeah, so one example of something that we do is, first of all, with just the instruction style that we have, we really do try to understand the backgrounds of the individuals in the room. So we don't have just product managers taking like we don't have product managers who are at a bank or a, you know, fintech company sitting in our courses and being taught, you know, best practices for product management to then be taken back to their teams. We have people in our courses who are engineers, engineering managers, analysts, designers, that they are HR partners, they are from all over the organization. So that captures breath because they are all getting this training and then they will disseminate the learnings to their respective teams. So that's breath. And then in terms of depth, we really focus on partnerships. The one thing about product management that I it's the biggest takeaway for me. And it's what I oftentimes emphasize in our trainings. You can be really good at analytics, you can be a rock star at, you know, design sprints, you can really like you may have certifications in scrum. And then you can, you know, you also might have like a UX background, you could be really awesome at utilizing all the different frameworks and templates. And you're an awesome roadmapper. But if you don't build relationships across your organization, you will not be successful. And so the relationship building across the organization, the alignment, the ability to softly influence the outcomes is the, you know, the shining moment for a product manager. And we talk about that a lot in our trainings with a group of cross functional folks. And that goes deep, because relationships are unique at every organization. Yeah. And it's also a relatively new function. Yes, I have been taking 10, 20, no more than 30 years for a lot of companies. And it's normal for a lot of people to not know what is this person doing? Like now suddenly we have a chief product officer in a company that is not digital first. And we have a bunch of product managers and product marketers and other roles that have the word product in it. So I resonate with that type of friction that can be created and agree with the opportunity to approach this as an ally who's coming here to help you be more successful at your job, not as someone who's coming here to replace you to take your commission or do anything, anything in that in that way. And I've seen a lot of pms that are actually very talented because in order to become a PM, you actually need to be pretty good first. So yeah, but I think that keeping that eye on like, how can others can perceive this new role and treat this in a way that can be collaborative is absolutely critical. Yeah, so to add to that actually, I can give an example from my own experience in my career, Lending Club, which is, you know, I think if you kind of take a step back, it looks like a tech company, they call themselves a fintech company, and they are they have no physical location in terms of, you know, their bank and they work all online, they do everything online. And they started off as an online lender. And their competition was other banks, but they had a competitive advantage. They had a unique product that banks didn't have. They were offering people who couldn't get loans, unsecured loans, they couldn't get loans at banks, and they came in and said, Okay, there's a there's an opportunity here. And they had a business team, they had their credit risk team, which really was working on the core product, which was this underwritten loan. And then they had a marketing team. And there was no product team at the beginning. And then they started hiring project managers with MBAs. And then one day, someone came in and said, Oh, I work, I met your product manager. And he's great. He, he, he took my designs from the marketing team. And he gave it to the visual designer. And we've got a new website now. So fast forward through their, you know, hyper growth stage, and they're about to, you know, go and file for IPO, they had to do some serious reorging. So then and restructuring. And that's where the product team came in. The product team was introduced, because they needed a linchpin, a partnership or an ally for marketing, the business and design, and the user, the customer. And that's where that's where product management came in for lending club and completely transformed their business. And now they're not just loans, they're a bank, they offer true lifetime value and financial wellness to people. And they, they did years of this exploration of how product management and mostly a product mindset can transform the business and upscale everyone across the org. I can tell you that when they introduced a new SVP of product, because they hadn't had an SVP of product before, that individual had a lot of things working against him, and he persevered and he transformed the business. That is exactly what we are finding when we go into this type of transformations with some of our clients. These organizations didn't have a chief product officer or an SVP of product, and they brought someone like that. And that person is our biggest champion, because they've seen the value of elevating the thinking from project management to product management. And they also know there's a lot of project managers that have, they can be upscale, they can be set up for success in a true product network. The same way they can also acquire new talent, but in any case, they finding that blueprint that helps leverage what they have in house in terms of processes as well as their team, and bringing in a new perspective on how they can make product a true partner for business, for driving business outcomes, including increasing revenue is key. And I know it takes time, especially when talking clients like JP Morgan, Walmart, like we're talking thousands of thousands of people, not just in the product or across the entire company. So also curious to know from your experience, how you are not just customizing a program for clients, but specifically segmenting that at a level that specific groups of people can get what they need, and also ultimately measure success of the impact that all of these trainings are having for the business success. Well, one of the things that our training and the way that we set up training helps it to kind of go in and adjust the whole product mindset training to different individuals with different roles and functions at large, huge enterprises. We I mentioned the relationship building piece. And to kind of build on that. We look at products from different angles, and we really do a lot of product definition. And that requires problem and problem and context definition. So I think a lot of these companies are working really quickly, they're they're they're working hard, and they're they're often working in a siloed way. So they don't all share common definitions. And so what I see happen in our courses in in the class, when people are having aha moments, and these are people like I mentioned from all across the organization, different roles, different functions, not just product managers. Actually, it's seldom product managers. Sometimes there are business analysts and, you know, program managers and and other folks that are in the business ops world. But when they're together, and we're really understanding together what it means to develop a persona, who are the personas? What are their problems? What are their pain points? And then we go really deep into customer development. And that's just one example, of course, like we go really deep into customer development. Suddenly, there's an unlock, everybody is now understanding whether you're from risk, compliance, the business, tech, you're now understanding and you have a common, common definition of who the customer is, what they need from your business. There's also the next set like the next level to that is understanding how we track success by setting up metrics. We spend lots of time in our courses talking about how to set North star metrics and more kind of vision focused metrics that drive the vision and the mission of the business, and then how we set up success metrics and okay ours, and those roll down and trickle down to the individual teams. And they all kind of roll back up to support this greater reason for being for that company. The reason that's important is because suddenly there's not only a common vocabulary, but you know, shared among the different roles and functions, but there's also this understanding that the objective and key results can be different across the different teams. The success metrics are measuring your outcome. And the roles and functions have different outcomes, but they all lead up to that one company goal. And that empowers the people in those classes that are sitting in class. It empowers them to go and say, okay, now I understand how I need to measure my success on my team and my output, my output. Yeah. And I think it also applies to how we measure our own success in the training, because a lot of companies that do training, that type of training usually falls under learning development or human resources. And we found that all of those decision makers would just try to find a solution that was kind of good enough for everybody. And their success metrics sometimes would be as the NPS, user satisfaction. And that's just not enough. That, I mean, it's always good to keep people happy and get me wrong. But when we talk about functional, deep, custom training that drives business outcomes, it's you just need something else. And examples that you gave before, for when we did training for talent team could be, okay, time to reducing time to hire or increase increasing and produce tension. That's an outcome. Exactly. Not like a tool. Yes. And I, something where I repeat a lot, and I'm glad when it's already happening in organizations where the CPO or the makers run, product person in the order would say, hey, I'm either going to do this, right? Or I'm not going to do it. I can stop sending my people towards videos and things like that. So it's, I think it starts from the top. There has to be real buy-in. And we need to take responsibility of co-creating those outcomes up front. So we can make necessary adjustments, customizations to ensure that at the end of the day, this company, these people are set up for success. They're really not just happy because they had a good time in class. They're really, the executives are happy because they are providing more value to users and ultimately more return to their shareholders. Yes. And you just reminded me, one of the recent pieces of feedback I heard from our current client of ours in healthcare, they are measuring success. And I think in turn, we can measure the success that they're looking at, okay, of the people who took the course, how, how many things like how, how often are they referencing product school material that they've learned, learnings and implementing them into what they're currently doing, which is their 2024 activation of their operational planning. So they had, you know, spent a lot of time, probably, you know, the end of last year, 2023, setting up their annual roadmaps and alignments and everything. And they were doing that kind of at the top administrative level. So now they're going, they're going back in our, the people we partnered with to create this customized coursework for this healthcare organization is going back in and saying, okay, let's resurvey you, how often are you using this material? And it's like, I mean, it's 75% of the time, they're seeing that people that the people who have been trained are changing the way they're working and bringing in, you know, something like a prioritization framework that they all agreed was a good idea for their type of team. They're bringing in an outcome based roadmap instead of just a roadmap that has features with some timelines. So those are really like, they sound like small things. But the fact that people are going and within a quarter, within a couple months, they're using the learnings and putting them into motion and other people are seeing what that means, right? So they're, you know, consequently, there are other people wanting to sign up for product school to say, like, okay, well, that looks like something I want my team doing. Yeah, well, in startups, that's not a problem at the beginning because usually a founding team has a sense of product or they're also smaller, so they can coordinate in a more easy way, right? But I agree, like when we're talking about training thousands of people across the world that are using knows what type of artifact for creating roadmaps or analyzing data for deciding what is the priority that sometimes as you said, creating creating that short vocabulary and starting from creating that alignment. So we understand the rules of the game is huge victory because just a small optimization at that level creates a massive impact given the size and then obviously there are levels, right? Of maturity and I like to always start with an assessment, like we do in product, you try to assess the problem and then pick the priorities. In this case, one thing that I notice is that regardless of the level of maturity of the work, there are also different levels of different types of training and a lot of individual contributors or first time managers are really hungry and appreciative to be in a classroom environment, creating workshops and participating with others. But as we were as we were hiring the seniority, we see a lot of senior managers say, I don't need this, I already know. And so I'm curious to know how you found success getting buy-in from mid-level managers and above to ensure that they do not jeopardize this. They are also become allies and they can continue seeing value. I think, yeah, I think I mean, it's different for every company, right? And for every organization, but the buy-in happens when you can really apply the best practices to their space. So we have a finance, large finance client that is really wanting to understand how they can stay, for them, it's efficiencies, right? Like they're trying to gain efficiencies. They don't have any issue staying relevant and competitive. They're too large of a bank. They're almost too big to fail. But there's an opportunity that the middle management sees for efficiency gains and cost cutting and innovation above all because you're too big to fail in the current state. But we all know that trends like AI and, you know, other things that are happening in the tech space, like if we don't find out how we might innovate for that now, tomorrow is too late. So what happens with middle management a lot of the time is they're kind of at the crux of the, you know, upper management wants to move fast. And so to move fast, sometimes they are giving the solution downstream and saying, we're just going to do this. And then there's the middle management saying, well, that might not be the right thing to build. And so how do we build the right thing to get, you know, to solve the problem? But how do we build the right thing? Yeah, and that conversation is that the how do we build the right thing is how I think people slowly get it the buy in. And it's so tricky, especially in large companies that already have something that is working. Yes.  Obviously, it's important, but that's not always sound critical because you have something that is working that core business. Right. So it usually takes some sort of wake up call as someone who can really drive influence to make people really start taking action. Obviously, upskilling is one of the actions, as you mentioned, investing in other technologies, such as AIs, other actions and and there are others. So what I what I what I saw that usually works with executives is to speak the language in a smaller environment. Because nobody wants to be in. I don't see many VPs that say, oh, I want to be in the same class. Even though they might need it, they don't want to be in the same class as other people who are two, three levels below. So this is to me, they sometimes creating the right environment to make them feel safe to say, I don't know something. Tell me more about the risk. Tell me more about the opportunity to do it. Tell me more about what is a how concrete and outcome driven roadmap, things like that, me like creating those and sometimes that means one to one, by the way. And that's that's something that I that I love that we can customize to to ensure that at the end of the day, this doesn't. Feel like a nice initiative to keep individual contributors happy. It's true and strategic initiative. We buying from the top. Yeah, absolutely. So yes, I was looking at looking at more of it like, OK, how do we, you know, empower the middle management? But I think that for yes, for executives who just they they kind of just want the the TLDR of like, tell me, you know, give give me the insights. Take me to dinner and give me the insights. So yeah, yeah, that's how much more money can I make? Right, right, right, right. Exactly. Yeah. Yeah, it's empowerment at all levels. Totally. I mean, at the end of the day, it comes down to that, right? Like, OK, you either make me greedy or you make me scared. And that's kind of how people tend to react myself, included. And you just reminded me of some example of maybe something that I hope a lot of companies decide not to do or not to do for too long, which is this, what we're going to create this innovation team or this labs team. These are my like my rockstars, my up and cameras, the tech team. We're going to give them training. We're going to give them resources because they're our going to be our champions. We're going to spread the love and show good results. That can go either way, because I've seen examples of companies that do that, but those teams are isolated. The rest of the company is very jealous of them. And ultimately, it's just like putting water and oil together. In my opinion, in my experience, I've seen these be more successful when we still customize, but treat this as the same problem and as the same team. I couldn't agree with you more, Carlos. I have colleagues from, I mean, from graduate school all the way through, you know, up until my recent product management career and my different roles. I have colleagues working at companies right now. They are on innovation teams and they're like, we we are overloaded and there are too many other areas in the organization that need more of us. And I really hope they don't just spin up more siloed innovation teams because then nobody's talking. Nobody's singing from the same book like it's and they are seeing that. So yes, word of caution, like there's something that can happen that's more holistic and across the organization that's going to actually yield, I think, better results, right? And more value. Yeah, Mariana, we need to to do more of this because we have more to discuss. And I really enjoyed the conversation. Thank you so much for sharing more about what you are seen from the trenches and thank you for the work that you're doing, helping large organizations transform and skill their teams to increase revenue ultimately. Thank you. Thank you. This has been great. Thanks for the opportunity.


Thank you for having me today. I have to confess I know almost nothing about the topic as I was asked to talk about financial impact of genomic program implementation, and the organizers were kind enough to allow me to talk about something I do. Oops, I'm not on. Okay. All right. The organizers were kind enough to allow me to talk about comparative effectiveness, something I am familiar with, and I do appreciate Ned and Sean for setting up the prelude to this. It will tie around to matters of finance and economics, because I think how we think about comparative effectiveness will be central to our ability to maintain a sustainable healthcare system. So I want to go beyond cost, finance, because ultimately we need to think about sustainability and affordability. Just a little bit, for an overview, I'll tell you a little bit about the Technology Evaluation Center at Blue Cross and Blue Shield Association. And then thinking about the introduction of diagnostic tests, perhaps we are engaging in deja vu all over again. I'll draw some examples from the world of imaging that may be informative as we think about genomics, talk a little bit about comparative effectiveness as a concept and a broad concept, and then touch very lightly on what I think of as the third rail of comparative effectiveness, which is cost and cost effectiveness. So we'll touch that gingerly. Blue Cross and Blue Shield Organization is comprised of 38 independent companies who make their own coverage and financial and other decisions. The Blue Cross and Blue Shield Association provides them with core services and representation, one of which is the Technology Evaluation Center. Collectively, the Blues have almost 100 million beneficiaries, about one in three in the U.S. Technology Evaluation Center has been in existence since 1986 and represents our systems commitment to using evidence-based in decision-making around health benefits. The core of our tech assessment process is rigorous review of clinical evidence, systematic review, and the ultimate question, of course, is does this technology improve health? We have an independent medical advisory panel that has scientific and clinical authority over our reports. We are proud to have Marin as one of our members. That medical advisory panel is of 19 members, only three are affiliated with health plans or maybe four. We have a dedicated professional staff within tech of physicians, epidemiologists, pharmacists who develop our reports, and we're proud to be an evidence-based practice center of the Agency for Healthcare Research and Quality, and we're designated as the comparative effectiveness review center for cancer and infectious disease. And I have been honored myself to have been appointed to the methodology committee of the Patient-Centered Outcomes Research Institute. So what I'll present today is really an amalgam of thoughts from these various vantage points and experiences. I do want to speak with clarity to one issue that is often confusing outside of the health care insurance pair world, which is what are we talking about when we talk about technology assessment? We see the world contractually as being in medical policy, coverage policy, and payment policy. Coverage policy, what is actually covered, is determined by the contract with the beneficiary. Typically it is a contract with the employer who provides health care benefits, and that outlines very broadly what the beneficiary is entitled to under the contract. And it may consider matters of cost or cost effectiveness with respect to including or excluding certain types of benefits. And how much the reimbursement is is another contractual matter, that's a contract between the health plan and the provider, whether it be institution, the clinical laboratory, and so on. And that determines the payment. The part of the world that we sit in at tech is what we would call medical policy. This is very much a clinical enterprise. It gives medical directors in our plans the clinical analyses to support their decisions with respect to medical policy. And contractually that means administering the medical necessity and investigational provisions of the plan. So just to understand our vantage point is the system. So as I think about genomics, I'm very influenced, Sean and I were both at a symposium on comparative effectiveness in imaging, where some fantastic work is being done. And it reminds us that in some ways what we can learn from the experience of radiology and imaging, I think, can be very informative for genomics. The advanced imaging that we now have both entered a different world than we now exist in, in terms of the healthcare system, in the issues of cost, affordability, and access. And to some degree, also helped to create this world because of the 80s was really a surge of high tech interventions and diagnostics, whether this be CT scanning, or whether this be organ transplants, there was a tremendous rapidity of innovation surges and capabilities and surges of cost in that, in that era. So we are, we inherit the era, the lessons of that era and still some of the dilemmas of that era. But I, I have here the diagnostic model for a continuum of efficacy. This was published by Frybeck and Collex in 91, and it was with regard to imaging, but it outlines three levels of efficacy, starting with the technical, which we tend to think of as pretty pictures, and pretty pictures are really not enough to establish clinical value going to societal efficacy. And I find it of interest because I think as you look at it, those of you who are familiar as most of you are with the ACC framework will see here a progenitor and how in various fields of diagnostics we really have been grappling with this tension between more information and the clinical use of that information for several decades now, trying to get the right applications and the balance. This is a, from a paper by Golden colleagues that traced about five technical innovations in diagnostic imaging in breast cancer. This is the timeline for MRI, but in general I think it is sobering and informative. What their overall finding was that utilization of new imaging technologies is driven by regulatory approval and reimbursement by payers rather than evidence that we provide benefit to patients. And so I go to many conferences and meetings of many stakeholders and often the question in the hand is how do payers cover, what will they pay, why do they pay, when do they pay, how do they pay, and the real question we have to ask is how are patients benefiting and what are we creating in our healthcare system to maximize our ability to provide benefit and improve outcomes to the population. This is illustrative again, this is from the world of imaging, but it just shows how the number of scans in CT and MRI has risen along with capacity and demand has grown in lockstep with the actual capacity that's out there as you are probably well aware. Healthcare tends to violate the basic law of economics which is that if you have more capacity and more supply in relation to demand, you ought to have some moderation of prices, but it seems in healthcare that actually capacity drives both demand and can be price increasing. And we need to be very cognizant of this as we think of a world where affordability in our own healthcare system is critical. We have 53 million individuals who are uninsured, locked out except for emergent situations from the healthcare system. We are trying to bring them into the system and into the tent and balance all of these variables for access and affordability. So I have to thank Peter Bach for this slide and he thanks Dr. Seuss, but we will call this medical suesonomics which illustrates that if you have a capacity, you will have throughput. If you have throughput, you will have revenue generation. What we don't know is whether you will have a health outcome benefit and that's the challenge to us to achieve this. So thinking about, I call this diagnostics, prognostics, agnostics because I think we need to take an agnostic stance, imaging, testing information in and of itself as Ned has pointed out is neither good nor bad. It has the potential for both beneficial and adverse effects. I think as we begin to think about how in the age of comparative effectiveness, we're evaluating evidence, we have been in the age of technology assessment. And I think we actually need to move beyond that. We tend to start from the test. I've got this test. Does it work? Can I do something with it? What can I do with what I've got? Well, maybe I'll compare it to another test. But we really have to talk about starting from the patient and about the clinical decision at hand and saying, given this clinical situation, given this context, given this nexus, given these attributes, what is really important to making a decision and to management? And what are the strategies available to accomplish the best we can for these individuals? So I would like to invite us, particularly as we're really on the doorway in genomics to a new area, to also have a new paradigm and a new way of thinking about evidence evaluation to contextualize it. And to be working from the clinical problem rather than working from the test that we would like to introduce or get reimbursed and that we hope will improve outcomes. But I think if we start from the clinical setting, from the clinical question, we have better prospects for achieving that. I think we need to realize too that in comparing test to test, another part of the context that we may lose is that test results are rarely definitive, they are often suggestive, and very often additive in the context of other clinical information. So that when we think of the test, per se, we really need to think more broadly about the setting of use for the test and what the test represents in terms of all the information that goes into the strategy. So I'd like to suggest that we think that clinical management strategies have a context for evaluation, that tests and treatments are the components that help, that display an analytic framework. All the steps that take you from a diagnostic information to an outcome involve treatment decisions, treatment interventions, the knowledge and behavior of providers, the knowledge and behavior of recipients of patients, it's a very long and complicated chain to get from information to outcome, and in that chain we have a cascade of consequences, some of them for the best, some of them perhaps for unnecessary testing, misdiagnosis or harm. Certainly we've become more aware of this in the area of radiology with advanced imaging, the remarkable exposure from relatively young ages to radiation that has simply become part of our medical practice, and which I think has been taken for granted and not appreciated the potential for harm until in the last three, I'd say the five or six years there's been more documentation of this and more awareness of this. So diagnostic results in themselves are intermediate outcomes, they depend on treatment outcomes, and all the treatment outcomes are called direct in the language of technology evaluation, technology assessment. In the real world, in the world of true clinical effectiveness, they're mediated by operator skill, the health delivery system, and patient adherence. So it's really complicated. So that's why I think it's important to understand that comparative effectiveness is much more than A versus B, or as I've heard it put, it's more than the red pill versus the blue pill. Again, getting out to this idea of a context, the clinical situation, the decision and the strategy. So comparative effectiveness, and I draw here from the work of the IOM committee on comparative effectiveness research, and also the work of the federal coordinating committee on comparative effectiveness. But comparative effectiveness addresses strategies to manage your condition, taking into account real world practice, and variations in patient populations. And I found it quite telling that the Institute of Medicine, the top 100 priorities of those topics, they're not strictly A versus B, intervention versus intervention, but half compared the health care delivery system, that is, comparisons of how services are provided and what the results are, a third addressed racial and ethnic disparities, a fifth addressed functional limitations and disabilities. Now, this is side by side with the usual suspects for evaluation in our health care system, where the large utilization, the large populations, the big issues are cardiovascular disease, psychiatric and neurologic disorders, cancer, and others. But I find it very telling that it wasn't just about these clinical conditions. It was about the setting of treatment. It was about disparities. It was about function. And we also know that outcomes that are frequently measured. And Sean alluded to this in the issue of the evidence gaps and who was really creating the research and what other stakeholders would bring to how that research is approached. So it's the issue of what outcomes are really important to patients. And that, I think, is something that the Patient-Centered Outcomes Research Institute is very much bringing to the center, trying to move from the vantage point of the patient's decisions, such as given my personal characteristics, my conditions, my preferences, what will happen to me? What are my options? And I think we need to keep in mind that it is especially as we get into the chronic diseases and we get beyond midlife. When we talk about the patient and we talk about the condition, we are not talking about an individual with one condition. We are talking about individuals with comorbidities. Chronic conditions are rarely observed in less than a set of three clusters together, yet guidelines speak to osteoarthritis or they speak to heart failure without really good guidance as to how all of these intersect. And we need to start thinking in a cross-cutting way about real patients with real complexities in their situations. And so obviously, the patients want to know about potential benefits and harms, and we hope that patients will become empowered to make decisions and have an impact on their care by being informed and interfacing with the delivery system. Just to illustrate, this is not at all from the world of diagnostics. It just happens to be one of my favorite topics. Think about the worth of what stimulating agents we at Tech have thought about them for about 10 years now because ARC has come back to us a number of times to do comparative effectiveness reviews in this area related to management of anemia of cancer therapy. Now, it became quite evident with our first report that the issue is not a poet versus Darba poet, and there's some class differences, there's some convenience differences, there's some pharmacokinetic differences, but there's really not much difference from a clinical vantage point between EPO and DARB. And the real issue in understanding of the orthopedic stimulating agents is not a poet versus DARB. It's how do you manage anemia of cancer chemotherapy, and really there was a whole transition in thinking from, this is a safe, benign intervention that will raise hemoglobin, prevent anemia, prevent transfusions to understanding that there were real risks involved. So going from thinking this is something very safe, more may be better, sooner may be better to understanding that there's really truly unanticipated risks. To me, this characterizes the whole notion of comparative effectiveness research. It's not about EPO versus DARB, it's about the situation of an individual going cancer chemotherapy, potentially becoming anemic, and what are the options and the benefits and the risks in that setting. Comparative effectiveness is not a de novo enterprise. We have been in evidence-based medicine era for decades now, creating an evidence base for decision making and for practice. So comparative effectiveness to my mind, as Newton said, stands on the shoulders, Newton said if I have seen farther it's because I stand on the shoulders of giants. And so in the same way, comparative effectiveness research has to stand on the shoulders of present knowledge. We know in our understanding of the evidence base and our understanding of the interventions available to us that there are significant obstacles to assessing outcomes to really understanding the health benefits. This is not the topic here, but I do need to highlight these. We have a pervasiveness of reporting of outcome measures that don't really measure health. We have inconsistent and often absent reporting of adverse effects. We've got selective reporting and publication bias. We've got a gap between understanding efficacy versus effectiveness. I will give one of my favorite examples from the device world, the implantable cardiovascular defibrillator for primary prevention.  You can see that there is some benefit and you can calculate the number of needed to treat and the hazard ratios and the improvement in survival.  You get it out in the real world and then what happens? Defective leads and nobody knows when defective leads should be removed. They're not so good at how to remove them. A whole unanticipated cascade of events gap between efficacy versus effectiveness. What we knew from those trials in no way would anticipate some of the real world tough decisions that had to be made without much guidance given these implantation of these devices with manufacturer and defects. And as we move more to comparative effectiveness, we will be looking more to different kinds of studies. The randomized controlled trial has always been a gold standard that will continue to be an absolute platform of our knowledge, but we also need to be getting better using observational methods. I believe very much that modeling, modeling that has a robust empirical basis will be critical because there are questions of such tremendous complexity and such rapid evolution of circumstances and technologies that they really will not be able to be captured within the walls of an RCT. And it's how you use all of those methodological tools and how you understand and I think it is important to understand both the promises and the pitfalls of observational methods because just as in diagnosis and just as any other information, these observational methods offer us both benefits and possible harms. So let's think a little bit about cost. PCORI has actually been concluded from conducting any cost effectiveness analysis. We have a political culture which is very reluctant to address these matters and as I think of it, it is the third rail of comparative effectiveness. If your research institute touches it, you may be zapped. But the fact is we can't afford not to think about these things. This is the, everybody has their own projection but you can see the projections have in common that left to present trends. The healthcare spending could potentially eat up the entire gross domestic product that's obviously unsustainable. Our position in world markets has changed. We need some way of dealing with this. So in summary, we're looking at comparative effectiveness from the vantage point of managing conditions. It's different than moving from the technology. We need to think differently. It's a different paradigm. We have to take into account systems of care delivery. We have to think about organizational behavior, individual behavior, systems aspects. We need to think about the patient's voice in these decisions and the vantage point to benefits. We need to think about the tests and treatments as being components of the strategy but not necessarily the object of evaluation or assessment. We need to understand that our evidence base in general has certain deficiencies and that we need to improve them because if we are to stand on the shoulders of giants and to see further, then we need to improve that base. There are certain well-known deficiencies. We see that the politics around cost is very, very difficult in our culture but I think we need to grapple with it, that value and affordability are intertwined and that all stakeholders in this health care system are going to have to exercise stewardship in order to make it sustainable. Thank you. Thank you very much. I think we're running a little bit behind. So if there are burning questions, we'll take one or two and if there are no burning questions, either they're burning questions that everybody's ignoring because of time or it's sort of the post-lunch torpor. So we'll not figure out which one that is but we'll have the next talk instead.