Okay, I got a we better get going because we got 45 minutes. I am Chris Edwards. I'm a senior manager development manager at IHS I Am not here to sell anything. I'm not a consultant. I'm an agile practitioner I come to these conferences because I like talking to people. I like learning from people so After this talk, I encourage people to come up to me I want to understand more about what people's experiences are if you can take anything away from this How does this work in your context? So? disclaimer, this is not a talk on architecture I'm here today to talk about leadership What I'm really going to talk about is how this architect role fits into an agile organization So there's a lot of questions, you know, if you think you look at the agile manifesto It doesn't really talk a lot about this architect role talks about team self-organization. We want to empower teams We want to give them control That seems a little bit contradictory with this concept of this this Architectural architect sitting on his ivory tower making decisions generating large documents Encouraging large backs batch sizes So is there really a place for this architect role in agile organization? so I'm not here to talk about what I think your organization should do what I'm really talking about today is is our experience my experience in this role Hoping that some of the things that we tried you can take back and give a try on your own So you just I want to set the stage a little bit of our context We're relatively small group within our organization. I chess itself is 8,000 people But our part of the organization that was working on this product is about five teams It's a pretty good mix each team sort of had their own mix of their development their their process within their team We are a thick client desktop application It's a engineering platform for petroleum engineers written in C++ The code base is fairly old the product itself was fairly new But it's built on a code base. That's you know 30 some of the code in there was written 30 years ago and We recently adopted agile so this the story is a few years old now, but when but when it started we had just started doing an agile transformation and The main the main story revolves around a major project that we had which was to take our thick client and convert it to a distributed database platform multi-user platform So we started we started our journey With these this new agile transformation We really wanted to be pure to our agile principles. We we were we wanted to empower teams to self organize We wanted to give them control So our first approach to architecture was totally hands-off We we formed feature teams we gave them projects and we gave them full autonomy We believe these agile manifesto says that we should leaders should really step out of the way and give the teams the space to get the job done and We actually saw some some good results with this We had some projects that were they were pretty Well-defined they were in our they're in our skill set So, you know, maybe it was an enhancement to an additional feature or a new feature that was you know another engineering feature or but What we found was on this new project This new multi-user project required skills that we didn't have an organization We never had anyone that had worked in distributed systems. I didn't know no one in our organization had ever had to deal with things like database concurrency and The same approach of just handing this project the teams doesn't really work people people were actually getting frustrated The project was was having trouble getting off the ground and people were getting frustrated. They were actually screaming at us for help the agile manifesto This was a piece of the manifesto. I think we kind of ignored so build projects around motivated individuals Give them the environment and support they need and trust them to get the job done. So These people were not motivated. They were frustrated. They were the people actually dreaded working on this project And we were not giving them the support they needed so This was his first realization we had about this architect role is that this concept of empowerment We want to we want to just take our hands off the wheel and give teams the space There's a little bit more to it than that. We need to provide them with tools. We need to provide them with skills training And we need to figure out a way to do this without sacrificing our agile values Maybe maybe this really is a time we said for having someone taking a higher level view and Finding a way finding a path forward But how do we do this? without this person becoming a commanding control design authority Okay, so So I was actually tasked with doing this the reason for that was I had been on the project along I'd worked on this product a long time I had a pretty in-depth knowledge of the code base and I'd been involved with some of the lead-up of the project So I had a lot of the context so it kind of made sense that I'd be given this this task and I really needed to answer. How do I do this? I wanted to avoid this? I wanted to avoid me Doing a huge amount of upfront architecture work And then throwing this, you know large design document to the team and I also wanted to avoid becoming the the Boss the guy who specifies standards that everyone must follow What we really need realized is that rather than a you know military commander what we needed was someone who could scout ahead Someone who could look look down the backlog look for you know challenges that are coming down the pipe do some investigation And then feed a stream of information to the team someone who's a servant to the team someone who's not an authority figure So we arrived at this concept of the architecture scout and the scout and this idea is that that the team still own their designs the scout is there to provide information they are a service Now I did some large I'd say not large design, but I did do some documentation. This wasn't totally just a a Prototyping or information gathering thing. I did do some you know, I figured out. What are the layers of the system? I fed the team some design documents, you know, here's the responsibilities these layers But primarily the way I communicated information the team was through prototypes and examples The teams use these prototypes as templates to work from so whereas before they're struggling They they have no idea where to get going, but now they've got something to look at they can use these examples as templates for their next work and The teams really hit the ground running with this. We saw velocity improve almost immediately Morale was improving people were excited to work on the project. We were getting things done. This was really really great for a time so Here's the five teams and here's little me architect guy figuring all this out And this is a little bit of a representation of the communication structures that we had I Was feeding information to the teams? And if people had problems, they'd feed them through me. I would come up with some solutions I would help them out the teams themselves weren't really communicating Occasionally Teams would have to consume each other's work. We're building this system So we've got modules and they need to come consume the modules each other's a built But maybe one team has a different idea of how things should be designed than another and what we started seeing was Conflicts forming between the teams. So here's an example, you know One team consumes another team's work and he says oh, this is really poorly designed. It's got like a thousand classes here It's it's complex. I don't like this. It's hard to understand and the other team says well If I have just one giant class, that's not going to be maintainable. I Think there's elements to truth to both of these statements But the problem was they were looking to me to define some standard. Well, you're the architect Can you just tell us what the system you tell us how to design it and we'll design it? We don't we don't like this conflict Really what they're saying is work harder do more do more and What I started getting to the point was I needed to be really in a thousand places at once I I need to be looking at everyone's work I needed to be kind of running around and and and seeing you know, what are you doing? Oh, that's not consistent with what these eyes are doing and I need to to fix this problem and I remember at one point. I actually Realized that I was terrified of taking a vacation because if I left for any period of time Well, they're all these mistakes are gonna get made and and and I can't do that because because then I'm gonna have to fix them And we're gonna have to do all this rework and Yeah, this was not fun so What we really got into a situation was that I I'd become a single point of failure on the project. I'd become a bottleneck for decision-making and Not only that it relied on my skills and decisions To be really really good because if I was wrong, it's gonna be really costly So the other kind of question is what if I am wrong and if I was wrong, how would I even know and In fact, what we found out a few months into the project that there was a pretty major problem with our architectural approach is it was there was a part of the architecture was extremely unmaintainable and We realized that if we were to continue this approach in the long term It would be completely unsustainable and we had to do a ton of rework and and take a new direction And I had some conversations with some people about this and I was kind of like well like what do you think about this? And they're like, oh, yeah, yeah, this this doesn't make any sense Well, wait, how long have you known that I? Kind of suspected this a while ago Why didn't you bring it up? Well, you're the architect. I just assumed you knew something. I didn't I thought you had a really good reason for doing it This way. Oh Okay well What was happening was When I was doing the design when I'm doing the architecture, there was all sorts of assumptions that are being built in The teams weren't aware of these assumptions. They did and what I wasn't saying was the detailed work I wasn't in the weeds Realizing that all these assumptions I built in were being completely invalidated in the day-to-day work Because they weren't aware of the assumptions. They had no way of communicating to me or knowing that this was the thing that they I should question There was a study done recently. I think there's a TED talk on this that they found that Medical teams they did a study of a whole bunch of medical teams And they were trying to find out what was the key elements in the high-performing medical teams the teams that were really successful And there was a weird element to them that the teams that were the highest success were the ones that made the most mistakes That didn't quite seem right so they dug deeper and what they really found was it wasn't the teams that made a lot of mistakes It's the ones that brought up a lot of mistakes the ones that Communicated to their team because they understood what was going on and they didn't feel Uncomfortable telling the team these things, but I'd create an environment Where the architecture couldn't be questioned the architect couldn't be questioned and the team didn't understand the context of the work They were doing So what we really had here was a people problem This isn't a technical problem. This this is a leadership challenge and I I this was when I really started coming to this realization that The architect role is more than just a technical role that there's a real fundamental human people component to this and So I'm a I'm a software developer I'm My trade is to tell computers how to work solve technical problems. I Was finding technical solutions to what I thought were technical problems really what we have is a people problem. I Love this xkcd comic guys Engineer as well, and he's trying to solve the equation for love using math This is probably how I would do it too my normal approach is useless here so My view up until now on leadership was fairly one-dimensional I kind of thought that you either you know, you have the the drill sergeant who tells people exactly what to do Or we have our our agile Empowerment where leaders take their hands off the wheel and just let the teams self-organize and decide And but neither of these was working So I had this problem. How do I serve teams without just telling them what to do? I need to I need to help them build this architecture of the system, but when I just give answers People will just follow it whether they make sense or not Fortunately, I had a lot of leaders in my organization few could coach me they provided me a lot of resources I got a lot of support here one of the one of the resources that Was pretty pretty powerful pretty impactful on me was just looked by David Marquet turn the ship around so Marquet was a submarine commander and In 1999 he took command of a ship submarine called the Santa Fe And he had he had six months to get this ship ready for deployment The problem he had though was that he'd been given this assignment at the last minute He'd actually been preparing to take command of a completely different submarine one that had different technical specifications different standards and There's no way that he was going to be able to get that ship ready If it relied on his ability to give the right orders at the right time Because he just wasn't going to be able to understand the technical specifications of that ship fast enough so David understand understood that in organizations. We often funnel Information to those who are the decision-makers those in the authority positions Whether it's the you know the VP or the architect or the manager or the product manager or whoever we Funnel information to them so that they can make really good decisions You'll see you'll see leaders do this They'll go into meetings and they'll just sit there and they'll ask lots of caught lots of questions And then they'll make the decision meeting over so How instead rather than moving information to the authority How about we distribute the authority to those with the information because then we can make fast decisions? We can we can respond quickly to new information and those who have the information are going to make better decisions So I like this quote leadership is not what you achieve in the moment leadership is what others achieve the moment you leave so If you're terrified to go on vacation because you think everything's going to fall apart Then I think that that maybe you're not meeting this and and this was definitely the case for me So the other thing David realized was that if you want to take your hands off the wheel You want to give up control and give authority you need you need a little bit more because if we're in a situation Where if I take my hands off the wheel and everything falls apart Well, then empowerment is not going to give us a good result. We need something more so he identified these two pillars and In order in order to divest control we need to develop the technical competence and those doing the work And we need to create organizational clarity. So do people know? Do they have skills in good software design? Do they know TDD the day-to-day? Knowledge do they have knowledge of enterprise architecture of distributed systems of of database concurrency? They need to have all this knowledge And the other thing they need is to have the understanding of all the assumptions that are going into the architectural decisions They understand why we took this approach so that they can know when to question it And when to try something different Once you have those two things in place, then you can really hand over control here team So David he marked had he made a he made a pledge to never give another order Instead of giving orders. He replaced it with intent So he would communicate what the what his intent was and then rather than for permission to things They would communicate to him what they intend to do and this is pretty powerful And and actually I give you guys all a task next time you're thinking of asking your boss For permission to do something instead instead say I'm gonna intend to do this and here's why here's what here's what went into my decision and a lot of times When managers or leaders see people do that what they actually see is man This is someone who's really taking ownership of their work and this isn't this is actually something we don't encourage enough I think so Marquette's Marquette's book kind of opened up a new a new dimension to how I think about things So this is great. This is from Spotify. A lot of you have probably seen this I Kind of had this this one-dimensional view down here We either you know have 100% control or we have 100% autonomy. We knew that didn't work. So Marquette's Concept of organizational clarity is very similar to this concept of alignment What we really want to do is figure out how to get people up here Excellent technical skills. I'm aligned on what the system is and I have the autonomy to figure out how to deliver it so how do I do that and so It took a new approach to this architect we viewed me more rather than more as the scout now I'm more of a coaching role. I'm going to help Build that help develop the skills and understanding and the people who are doing the work so that they can make the decisions We want to divest the decision-making So one of the things we did we formed this daily design meeting We realized we needed to do some catch-up because the teams didn't have this knowledge So we invested 30 minutes every day where a technical lead from everyone Form each team would meet and we would talk about challenges that the teams are having and we'd work through them together As a team we would discuss how it how would you solve this problem? And what came up was all sorts of realizations that we were totally not in the same page and then Once once we came to a conclusion. We wanted to avoid that this team became the new New authority so the technical lead would bring there the one the technical lead that brought the challenge from their team They would bring it back to their team and they would repeat the exercise We wanted to reinforce that the teams themselves are the ones that own their decisions Okay, this was a this is this is a good suggestion from Marques book resist the urge to provide solutions So a big part of my job was usually I'd be sitting in my office somebody come by they have a problem. They have a question and I'm a problem solver. That's that's that's that's who I am that's kind of core to my nature So I want to solve problems. I mean you bring me a problem. I'm gonna give you an answer But every time I do that I'm communicating To the person across from me that I own the architecture. I'm the decision maker. I'm the decider So resist the urge to provide solutions, but we have to replace that with something So what I started doing was if someone came to me with a problem I'd I'd sit there and I'd I talked to them I'd I'd partner with them to solve the problem might understand, you know, what would what what solutions? Have you already considered? Why would you choose this one over that one? I might I might ask some questions about some things They hadn't thought of but what I really want them to do is I want them to walk away from that meeting owning the decision that they made Okay, and what's in what was really interesting about this is after after a while of doing this People would would already know that I was gonna ask these questions. They'd approach me in my office and They'd already have things prepared to say, okay. Here's a problem. I'm having Here's some things I've thought of You know, I really like this idea Because of these reasons I think this was one of the better ones and this is what I intend to do about it Not they're not asking me permission. They're telling me what they intend to do and We got to a point where I could just basically say yeah, that sounds great So in these meetings in any discussions in the meetings, I'd ask a lot of questions the questions I ask communicate values Which of these solutions is going to be the easiest to accommodate change in the future What are the impacts on on performance on reliability? What do I want you to think about? When you're when you're designing the system and the questions. I'm asking are communicated them What I think you should think about when you're building the system Okay so the last last kind of part about this is Going from a position of Building a system. I personally invested a lot in the development of this product And now I'm having to really really take my fingers out of the weeds and and let people make decisions that Maybe you're not always the best realizing that that mistakes are going to be made and letting go letting go of control was very hard for me, but I Know that I've been in meetings where I've done this facilitator thing and I've been asking the leading questions But I've really gotten the back of my mind like this is the answer. I want everyone in this room to come to Everyone in that room knows that I'm trying to get them to that answer. That's not really the same thing as letting go of control So, okay. One thing I want to say is that this design meeting didn't come without any problems We did get some jealousy from those kind of attend this kind of Became the new like expert authority meeting man. These are the smart guys. I want to be in that meeting Why can't I be in that meeting? And not only that we still had some conflict between teams We were divesting control to the teams themselves and the teams were taking ownership of their decisions But the teams themselves weren't always agreeing and you could see some conflicts And the last one is we were still having this problem that this design group was still kind of seen as a design authority So that was a challenge that was hard to overcome So the main thing I want to talk about now is the the conflict A lot of you are probably familiar with Conway's law Organizations which design systems are constrained to produce designs which are copies of the communication structure of the organization so If this is our communication structure and really now it's it's not me it's five people in the middle here Then can we really be surprised that we can we're gonna build a system that's that's you know Got modules being built that are all built very inconsistently or not and not communicating well with each other So how do we solve this problem? So One of the things we did and I would actually say that this was a this was a side effect This was not the intent that we were going for We we developed these sort of design steps. This was a framework to have discussions within our design meeting and The reason we created this is because a lot of our design meetings were turning into arguments People would you know get really attached to an idea they would have I'm sure everyone's been in a meeting where Someone's like arguing tooth and nail for the idea they came up with they're very attached to it I get pulled into this really easily What we tried to do was develop an algorithm I mean we're we're programmers we like to develop algorithms So we've created an algorithm to solve this problem This was a framework for having these meetings to pull us out of this this positional argument So so at the start of every you know someone would bring a problem. We'd make sure everyone understands What is the problem? We're trying to solve this is gonna keep us focused Because we go down rabbit holes Then we go in a solution mode idea mode Where everyone? Everyone is allowed to throw ideas onto the table. We'd actually put them on the whiteboard Everyone's ideas are on the whiteboard. They're all equal and there's actually something interesting about us about us all standing back Separated from ideas now. We're physically separated from the idea. I came up with and all the ideas are equal and The last thing is we we did objective Analysis of each of them. Why is this one a good idea? Why is this one a bad idea and Okay, so something I give you a try if somebody's if somebody's arguing with you and they're really arguing for a point ask them Well, what's a what what is one thing about that you don't like? Everyone had to talk about everything that's good about other people's ideas and everything is bad about their ideas and What this does is it pulls people into interest-based thinking we're thinking about what are we what are the outcomes? We're trying to accomplish rather than what what what is the idea? I want to win So this is something I actually realized recently is that we basically reproduced Edward de Bono's six Thinking hats minus the feelings one Which I actually think maybe could have helped So Edward de Bono's thinking at his objectives. Let's define the problem. Let's go into fat. Let's let's identify facts ideas Positive and negative outcomes and then arrive at a conclusion and agree together So what this does is it pulls people into parallel thinking and we we're all in the same mode and it's a conflict resolution framework so What's interesting is what we what we developed as a way to stop fighting in that meeting Actually turned into a way to help the teams themselves Get along better because what started happening was people the technical leads would go back to their teams They use the same framework And people started realizing that when we talk in this way We're not fighting anymore and everyone feels better And if we and I've actually been in meetings where we're me and another guy We'd be we'd be fighting and then and then someone would say Why are we not doing that thing again? This doesn't feel right and we kind of all tend to migrate back to this this way of discussing What's interesting is now this Framework of having conflict resolution was embedded within all of the teams and all of the teams had this common understanding That if we're going to disagree this is how we're going to resolve it One of the biggest reasons that people avoid conflict is because they don't know how the other person's going to react when I bring something up But if I know exactly how they're going to react if I know exactly the conversation We're going to have and I have a history of positive outcomes of having the conflict then The framework for having this this this common framework for resolving these became a Avenue to have better communication passed between the team. This was not something I expected to happen This was a surprise and I was very pleased about it Okay, this is kind of a I'm going faster than I thought Last point disagreement is a mechanism for organizational clarity. So Conflict is not just something we have to resolve but what we realized is when we disagree when we have deep Disagreement, that's actually when we we are the most Misaligned and these are opportunities for us to to get better and deeper understanding of the assumptions and reasons behind the architecture So I'll give you an example We had a problem one of the team members one of the teams brought to our design meeting and We realized there was really two options we could We could put some business logic in the domain layer or we put it in the database And we had this discussion the meeting and everyone's pretty well aligned within that group that business logic belongs in the domain layer That's the most cohesive for the system So we came we came to that agreement but the side technically brought it back to his team and the team repeated the exercise We want the team to own their decision great They came to the complete opposite conclusion And what we realized we wanted to deeper we wanted to understand what was going on here That there were there were members of that team that had a very strong database background And they felt very strongly that you know database integrity is the most important thing We should we should optimize that against everything else so My initial impulse reaction Well, we should just get them in line that you know the design teams decided we should we should enforce the standard the Architecture says business logic goes in the domain layer. You should put the business logic in the domain layer But if I if I look at this situation through my sort of new lens my new leadership lens my new way of thinking I'm realizing that really what we're missing is is one of our pillars There's this this lack of organizational clarity around what we should be thinking about it in our architecture So what we did the way we resolved this was we actually brought the members of that team to our design meeting We actually spent the next week in the meetings having pretty intense discussions about About all the things that led up to where the architecture is that we looked at all the things The reasons we made the decisions. We really wanted to make sure that we weren't missing something Maybe they were right. Maybe database. Maybe the database was the right way to put it What we came to the conclusion of everyone realized that that yes This was an important thing, but when weighed against the other factors, there was other things in our project that were more important So we did end up reinforcing our architectural approach But we what came out of it was multiple people who had a deeper understanding of the why behind it and We had much deeper commitment within our group and across teams in the approach we were taking Okay, so the end. Hey, we're all big happy family, right? So something interesting happened and I love surprises because They teach me something We've gotten to this point All the teams are gonna go along. We've got these great communication paths forming between the teams then there's little old me Am I even needed anymore Which it's a little scary Because does that mean I don't have a job anymore Actually, it turned out to be a really good thing So I'll get to that in a second, but we we realized that yeah my my role wasn't really needed We come to the point where we were having our daily design meetings and people were already Coming to conclusions. They were solving things in their team if they realized as they needed another team To solve it. They'd just you know, they'd pull the people in they'd have these design discussions They come to our daily meetings and really it just became a reporting mechanism. Hey, hey architect guy Here's the things we decided. What do you think and I just kind of became a yep looks good looks good looks good Well, you don't really need to pay someone to do that. So we migrated to this approach of Full full divestment of the architecture that of the control the architecture the teams themselves own the designs So okay, so what happened to me I actually You put yourself out of a job and opportunities open up So I'm actually in a leadership role now in a different group and part of the organization So this has been a really good this whole thing was a growing experience for me I learned a lot about leadership, which has now opened up new opportunities. So put yourself out of a job. It's a good thing Okay, so just kind of a review we started out here this idea that we need full autonomy, but we didn't really have this alignment or The skills I migrated to this low autonomy thing. I'm just gonna give you the answers What we tried now is is okay. We still have the architecture. This is still the approach But let's start building the skills and understanding of the team and eventually we migrated into our happy place Where the the design and ownership the ownership of design is shared amongst all the teams one thing I learned from this whole thing your approach has to evolve the The architecture scout role I believe served a purpose when we did it The architecture coach served another purpose eventually we realized we didn't need this role at all That doesn't mean that we would just jump straight to doing the architecture coach if we did this again Maybe maybe we have to resurrect these roles again in the future Maybe there's some new role that we hadn't even thought of that we're gonna need again But we needed to continuously inspect and adapt what we were doing and try different things So some learnings for me the empowerment more than just taking your hands off the wheel and letting go You need to give people support Lead through intent. Don't just don't just give answers build understanding Explain the why what why are we doing this? What is the architectural intent? Resist those urge to provide solutions Partner with people help them take ownership of the decisions they're making and help them come to those conclusions Create a framework for disagreement and the framework itself can help to navigate conflict But if it's common across the organization it can actually create this emergent behavior of better relationships across all the teams and the last one When when conflict happens when there's major disagreements celebrate that because you're about to you're about to delve deeper and get a deeper Understanding across your organization. It's a really good thing So I have what 10 minutes left for questions. Thank you everyone for listening Um There any questions? Do we have a do you have a like a hand mic? Go the architect should we know in the solution discussion or He needs to review a leader. I mean that better to know in the solution discussion Or it should be one's a solution let the team figure out the solution and let give you later I'm what is that better? I'm sorry I mean the solution I mean team will discuss among them and come up with a solution or We should I mean architect should we know all in that team discussion or he should be sit outside and review those Okay, so yes, I think the question is is should the architecture architect be involved in that team discussion? I said absolutely because that that is the opportunity that I had to to teach to build understanding To ask leading questions to communicate the values through the questions. I'm asking That is the key point that I did a lot the most of my work in building that Thanks Any of the architectural decision which you wanted you you you showed that okay You had to convince people for almost a week for going for some decision but During the phase of like no spending this week itself is like, you know, too long for them to digest that decision So how do we enforce them to like, you know that to any decision? You've forced decision Once we arrived at this it didn't Be a people People don't want to be misaligned. They don't want to be on a different page This is what I saw is that is that actually once you had the the way of Resolving these disagreements people naturally wanted to do that They the reason they avoided in the past is because conflict is uncomfortable conflict bad. I disagree I disagree with you. That's fine. I'll just do my thing because that's the easiest thing to do Once you create the framework so that they can resolve that I Never encountered a person who didn't want to do that who just like that. I had to tell them what to do it just sort of Got better That's been my experience Second question Being the architect lead You have to like not take care of like now how each team is progressing from their Thought process about what is the problem they are solving? So how do you keep that in line with what you are thinking from whatever product you are trying to build as architect? I had to lean heavily on the Technical leads and I wouldn't believe these were the people that had the strongest understanding The strongest technical skills. I can't be everywhere at once and and this is what you're saying Is it's really really hard to keep track of all these things? That's just not something one person can do I believe so the way we got there was was Step one was really developing the competencies of this smaller group of people so that they could go back and Then the teams themselves can sort of self self-regulate. I think I've got time for a couple more questions We talk about the system architects here, but there is a chain on business architect enterprise architect So how can build that connection? So it is team level I have Build architect team, right, but I don't know how the way business architect is thinking right how we can build those connections So here, I'll let you hold the mic. I don't need a mic. I've got one Um, I don't have a lot of experience outside of this context I haven't I haven't worked in like the enterprise architecture scale So I don't want to I don't want to give you an answer I don't under I don't really know but I believe that the principles of building skills and and and getting alignment explaining why Pushing that down as far down in the organization as you can I recognize that there's Times where not everyone can be involved in a decision you get to a point where that you that just is not feasible to do But I think the principle of divesting control as far down the organizational structure I I've seen this on With user stories as well Right, it's it's that here's that here's the requirements. I'll just implement them whether they make sense or not a Product manager a product owner who's communicating the intent behind the user story is going to have a much better long-term Outcome because the team can understand why they're making a lot of little decisions every day So I think that this concept Scales to a lot of different situations, but I don't have that exact experience. So Sure you first and then okay, we'll do a couple more So you you talked about that you have a design meeting. So my point there is You get away from your architect position and you created five more. So that now you already talked about that every team I think oh why that person is there why not I then How so I don't know whether you guys have solved that problem or is there any idea? How do I solve that problem? I don't know to create again these other people also go through the same way you went through So Marquette talked about this in a book because he said He wants to create a ship where every person on that ship thinks like the captain. I Would I would like a team where every single member of the team is thinking like the architect and that's a long-term goal It takes a lot of investment in people and people development and in skills development But that is the ultimate goal is to really have everyone having that solid understanding Did I ask your question? That The the problem you try to solve that not not single person dependency now another person getting to the same same So at least at least then we had one person You moved one step further, but yeah So the other thing I was developing was the leadership skills of those individuals so that they could repeat that task Yeah, that makes sense. Yeah, I'll agree Why don't we do one more okay? Simple question as you discussed you have a conflict regarding when we need to implement in on database level or module ever, right? Suppose you have decided that now you have decided that it will be implemented on module level So after that do we need to create some guidelines some document guidelines so that in the future? Also, we not know to need not to discuss same thing again. Yeah, we had some very light documentation about that No one read it Just didn't happen and I don't know if that's just because people don't like reading documentation Or the second one is just that the knowledge was so dispersed throughout the team that it wasn't needed That if you had a conversation with someone they'd be like oh remember remember that that's not where we put stuff Okay, yeah, you're right and it became like a self-sustaining Knowledge base that the team itself A lot of times I think we produce architecture because we're worried that if that person disappears that knowledge is gonna disappear Well, if everyone's got the knowledge and I know that I know that's not really like that's an extreme case There's gonna be so there's still gonna be some silos of knowledge, but let's let's avoid single points of failure Basically, it's happens with within our team I'm we discussed and we decided to go with this one and after that we don't have any Guideline over it. So after that we have discussed same thing again And we have decided like that again will do need to do this thing So in the future also some other developer comes in and then again we have a discussion like that So we need to have some guidelines, right so that we can refer it and Yeah, yeah, and I don't have a problem with that like that doesn't I think having some light documentation about you know the general like how did we arrive here and like a historical document, but I've never seen that done. Well if you if you if you see a way that that could be done But they always get lost like the people forget it exists. It doesn't get updated when new decisions are made Okay, thank you If I'm gonna stick around afterwards if people want to talk But I want to be respectful of people's lunchtime because lunch is starting now So I'll just be up at the front if people want to Talk about this more. I want to I'm curious to hear more about other people's experiences I want to learn if you think this would apply to you if you think it wouldn't that would be very beneficial to me So thank you everyone


Good afternoon and welcome everyone to the Green Mountain care board meeting. My name is Kevin Mullen chair of the board The first item on the agenda is the executive directors report Susan Barrett. Thank you, mr. Chair I don't have a Significant amount to report out only just to remind folks that they should just keep an eye on our Website and our Press release for upcoming meetings We have a couple of meetings next week and the week after that our TBD But we will certainly warn those Meetings if there anything if there's anything added to those agendas and will adjust the press release and I and I believe there Already was an addition this week. We added a primary care advisory group on Wednesday October 21st, so just keep an eye on that and that's all I have to report out Thank You Susan. The next item on the agenda are the minutes of September 29th and September 30th. Is there a motion? It's been moved It's been moved and seconded to approve the minutes of September 29th and September 30th without any additions deletions or corrections Is there any discussion? Hearing none all those in favor signify by saying aye Aye Those opposed signify by saying nay Thank you everyone So the first item on this afternoon's agenda will be a Discussion of the quality standards and for that I'm going to turn it over to Michelle degree to tee that up Michelle Thank You chair Mullen. So We are here today to discuss the final 2019 quality results For the one care payer specific contracts and so as GMCB staff I'll be wearing two hats today. I thought about actually like bringing hats, but I didn't So the first one will be as GMCB staff just overview of the program requirements And what we're looking at here today and the second sort of role that I'll play today is in reviewing the actual Medicare results for 2020 2019 not 2020 for 2019 So what I'm gonna do is Introduce everyone that I have with me here today. So from diva, I know that I have Pat Jones and Amy Coonrat on the line From Blue Cross Blue Shield. I have Andrew Garland and Tyler got here and joined from one care of Vermont So hello to everyone What I will do is share my screen and I will advance the slide throughout the presentation So once I'm done, I'll pass it over to Medicaid and just give me a signal and I'll I'll advance the slides for you so First things first just a quick agenda for today, but I just kind of went through but I'll be providing some background Just on the results that we're looking at here today Then we'll review each of the payer results. So we'll start with Medicare Then we'll go to Medicaid and commercial and then there's opportunity for the AC to provide comment This isn't the same format as we did this presentation last November Then we'll go to board questions in public comment Starting off with some background I Just want to kind of remind folks that today is about assessing performance on the ACO Quality measures that are set forth in the payer contracts between the ACO and the and the payers This is not an evaluation of the all-pair model quality performance as a reminder We measure all-pair model quality report performance annually It's very likely that those results will not be released until early 2021 For the 2019 year, we're just starting to get some of those data in obviously We have the payer specific data But in terms of the claims based measures that we need to evaluate we need a little more time for claims runout And for the 2019 year to be finalized So that said Again Just noting this is solely a reflection of the ACO's performance relative to payer contracts is not necessarily Reflect the ACO's contribution to the state's performance within the all-pair model agreement Though I will say we're currently exploring ways in which we can interpret these results in the context of APM performance And I'll talk a little bit more about our thoughts on that a little later in in the presentation So under the agreement just a reminder that the ACO is a legal organization of health care providers that agree to be accountable for the quality Cost and care of the beneficiaries assigned to it The ACO skills target qualifying programs must reasonably align in their design across payers, which includes ACO payer quality measures And so you'll recall that back in 2018 We did a significant amount of work With the HCA and one care around designing the Medicare measures and starting in the 2019 year So we'll see those results Here in a moment My famous crosswalk that I get to update every year So this year we're actually looking at the full suite of all-pair model measures here in the first column And you can really see that We're starting to see some pretty great deal of alignment across payer programs So you have the 2019 Medicaid next-generation program in the second column The third column is the 2019 Medicare initiative and the fourth column is that Blue Cross Blue Shield next-generation program I do want to point out a couple of quick notes here For example the initiation and engagement measure Blue Cross does treat those as a composite The all-pair model the Medicare initiative and Medicaid next-gen do treat those as separate measures And so anywhere that you see sort of that locked cell is just to indicate that the payer in that case That's those as a composite measure Another note here just before we get to into the weeds Caps measures. It is the consumer assessment of healthcare providers and systems. Those are the patient experience surveys And each program has some type of caps survey and that they Distribute but they're not necessarily all the same and so a couple of quick examples The all-pair model includes the cap survey Composite of timely care appointments and information for ACO attributed Medicare beneficiaries only But the Vermont Medicaid next-generation program Includes multiple caps patient-centered medical home composites you have The Medicare initiative using multiple ACO caps composites for attributed Medicare beneficiaries and similarly with Blue Cross They include care coordination composites and tobacco cessation questions from the caps patient-centered medical home group and just to consolidate this List a little bit. It was easier to make those sort of one lump But each payer and goes into detail about the caps specific measures within their programs So with that just moving forward While working towards pair lineman is a primary objective not all pair programs are equivalent in terms of quality requirements Again, you'll recall in 2018. We went through that work group to develop and propose the measure set in the Medicare initiative From 2019 to 2022 and so similarities across the programs are much more noticeable in the 2019 program here than they were in 2018 and differences that remain are primarily due to types of covered lives or aligned beneficiaries And and make quite good sense. So for example, there's a lot of a couple of adolescent measures for Blue Cross and Medicaid Populations but not necessarily for Medicare and that just makes sense based on their covered population So I know there's going to be a lot of Questions about what can we say year over year in terms of Advancements or changes in quality and so while we do now have two points in time Comparability is still a challenge for us and I apologize if you can hear my dogs So performance year one 2018 and performance year two 2019 data have been finalized We're working with our analytics team as well as some of our outside experts to dig into changes in quality associated with shifting populations and other factors And so just to sort of highlight that I wanted to show A quick snippet, you know, we've we've now produced two scale reports also And so this is just showing really that population shift so the increase in the Medicare and Medicaid populations and that Small decrease in the commercial program from 2018 to 2019 To just show how that shifting population has the potential to impact our results and our quality measures And again, that's something that we're looking into and how we can address to start to look at this more longitudinally Okay, switching my hat Medicare results for 2019 Uh, so in 2019 similar to 2018, there were four domains for Medicare We have patient and caregiver experience, which is worth 20 points I just want to note here. That's half of the total points and they're all based on those CAP survey questions. So that's patient experience Accounting for half of the total points available There's a care coordination and patient safety aspect Two measures there. There's a preventive health Section those there are four measures there A couple of notes on these so for preventive health in the at-risk population There were measures that were initially slated to be paid for Performance and they reverted to pay for reporting Um as all activities related to the quality measure validation audit for 2019 year were canceled And that's due to the public health emergency So that was a choice by cms to revert those measures to pay for reporting since they could not Perform the audits necessary to give a total performance for So here's the 2019 results I know this is kind of small, but I can leave it up on the screen for a bit while we start to talk through it I did want to note so I tried really hard to come up with a creative way to show 2018 compared to 2019 and this would be easier So every measure that has a star next to it was also reported in 2018 And so those will be part of the measures that we look into or that we work with one care on Thinking about how we start to look at these in a longitudinal manner How do we think about the population system of these and how do we You know start to to think about how to really trend this data Um, we don't want to make assumptions based on this because we know there was a pretty significant growth and that just that you know The overall denominator would have a pretty significant growth um So a couple of notes here for the for the medicare results specifically Caps measures combine responses to several questions and as such performance on caps don't represent an actual percentage But rather the acos mean or average Um, I can't say whether or not that's consistent across pairs I would imagine because they're all sort of on different planning scales But I just wanted to point that out. So in most cases, this is a percent score In the case of caps. It's actually just the average score The risk standardized and the all condition readmission measures So those both of those care coordination and patient safety measures are actually inverted So a lower score is going to be better there Um, and you'll see in a future slide. I made a mistake. So Abigail I will have to update our Our slides, but I will I promise I'll do that um and For this at risk population section down below here This is where you start to see the inclusion of those Those measures that are also part of our all pair models that that were not uh Asked of the aco in prior years. So that's where we look at follow-up after discharge form from the ed for mental health alcohol or other drug dependence um And then mental illness and then the initiation and engagement measure looking at that for the medicare population as well Okay So for 2019 quality results There were 20 measures total. So again two point maximum allows 40 possible points The aco's earned score was 36.75 which results in about a 92 percent quality score for 2019 and again just to note that those Related those that required a qmv audit were canceled And so there were five measures that the aco did receive full points just for reporting on Um, and again when these results when the all pair model quality results are final We'll work to figure out how these pair specific measure results impact the state's performance And we'll have more insight as to how to look at these longitudinally That's something that myself and the a team are really Interested in exploring and and trying to make sure that we can provide an accurate snapshot of performance year over year for the model I did want to give you a quick reminder of 2018. So jumping back a year We went from 29 measures to 20 in the 2019 year. So last year there were 58 points available and the aco earned a score of 82.4 percent And so here's where I made a mistake So of the 16 measures that were carried into the 2019 program improvement was noted in 14 of those measures not 12 so there were Three measures where a decline was noticed and it was very minimal Those were in the timely care appointments and information caps composite The um caps stewardship of patient resources And risk standardized all condition Re-admission and I just really want to stress how minimal the declines for um, and I I'm working with the MMI to see if They did any statistical significance testing on those results to see if there really is truly a statistical difference in those scores Something to just quickly talk about in terms of medicare Most notably kind of the exogenous specter here of COVID-19 and the public health emergency Really impacting some of the ability to score Appropriately for the 2019 year In addition we talked about this but the growing provider network and payer churn And then for for medicare specifically the Vermont population demographics. We know that we have An aging population and that Acuity and disease burden within that aging population is higher And these are just sort of caveats that we're looking to see if how we're Looking into more detail to see how these might contribute to our overall results And last but not least Back into my gmcb hat um I really discussed some of this ad nauseam already, but gmcb staff In the near term are going to dig into the impact of acl payer quality outcomes on our states 2019 quality performance under the model And we'll certainly be bringing those insights before the board again likely early in 2021 Uh, we'll be working to untangle year over year quality performance from changes in metrics due to increased scale or changes in the population um, and per the 2020 budget order the acl and the gmcb staff are working to develop a dashboard that include acl and hsa level quality results over time and for The gmcb staff are creating and publishing data visualization of cumulative apm quality results And this adds to our existing available resources On our tableau site for things like total cost of care and scale participation Looking forward to 2020 just a couple of things that I want to note that we've heard from medicare already Medicare at this point is slated to be paid for performance or for monitoring purposes only this is subject to change And then currently the cms proposed rule Suggest the removal of the caps requirement for all acos in 2020 So full credit would be awarded, but the survey itself would not be administered So again, this is in the proposed rule the cms rules don't become final until usually about mid november So we've got at least another month until we know about that um And Another note is just that utilization for 2020. We already know is pretty significantly down So we're going to have a pretty small n here to work with and so just thinking about What 2020 looks like in the grand scheme of you know, the agreement and the five-year term um, and then just another note that the federal evaluation of the all-paramodal agreement includes an analysis You know and quality outcomes across per month As well as at the aco level over the life of the agreement, but these results won't be available until at least 2023 So we've got some time before we would we would see those and our hope as stuff is to make sure that we can start to incorporate um Some of these analyses a little a little earlier down the line So with that, I'm going to turn it over to Medicaid Amy and pat if you just want to let me know when you'd like me to advance your slide Hi, thanks michelle. Can you hear me? Yeah, great. Okay. Hi. Um, so this is Amy kuhnrod, and I'm the director of operations for aco programs over at diva and I am here with pat jones to give a brief overview of The vermont medicaid next generation aco programs quality performance for the 2019 performance year If you'd like to advance the slide, that'd be great. Great. Thank you um So 2019 represented the third year of the vm ng aco program between diva and one care Which as michelle mentioned earlier involved 13 communities and an attributed population of around 79,000 medicaid members, which was a pretty significant increase in the attributed population from 2018, which was around 42,000 um As folks know as a value-based payment model the vm ng program campaigns of quality measures that and an associated value-based incentive fund part of the aco's fixed perspective payment is set aside every month into that fund and After the performance year that part of money is distributed to the network Based on the aco's performance on the quality measures that you see up here on the screen For the 2019 performance year the equivalent of two percent of the total fixed perspective payment for the aco from diva Was set aside into this value-based incentive fund That percentage has increased Every year since the program's inception I'll just give folks a refresh that in 2017 the value-based incentive fund Was 0.5 of the fixed perspective payment and in 2018 it was 1.5 of that payment that was set aside into this fund After the performance year the quality performance is calculated at the aco level And then a proportion of the incentive fund is distributed to one care's provider network based on its performance Half of those undistributed funds that are left over are reinvested by one care in quality improvement initiatives at the aco level And the other half is returned to diva during the financial reconciliation On this slide specifically as a reminder This table contains the aco's quality measure set for the 2019 performance year of the vmng program The set contains 10 payment measures and three reporting measures one of which is that caps patient experience survey These measures were selected to align as closely as possible with the quality measures for the aco's other payer programs As well as the overarching all-payer model quality measure set But there are differences as michelle's beautiful crosswalk chose To ensure that the measures are appropriate in the vmng program for the medicaid population Next slide please great So in terms of scoring there are a total of 20 points available for the 2019 performance year And each of the payment measures was weighted equally within the set and was scored individually Where possible One cares measure results were compared to national medicaid benchmarks, which were available for Eight of the 10 payment measures in the set If medicaid benchmarks weren't available at the national level or multi-state level The 2019 results were compared to one cares performance on those measures for the 2018 program year Which was the case for two of those 10 payment measures Beginning with the 2018 performance year as well and continuing into 2019 One care was also able to start earning bonus points for each measure That had available benchmarks or demonstrating statistically significant improvement over their prior year's performance Um, I went through that really quickly, but I would like to now Turn it over to my colleague pat jones Who will more specifically to one cares actual 2019 quality performance? Great. Thank you. Amy and good afternoon to everybody So i'm going to give a high level summary of the aco's performance in 2019 and then in our last slide I'll really dig into detail on how they performed on each of the measures So for 2019 the aco's overall quality score was 95 percent For the 10 payment measures and that compares to a score of 85 percent in in 2018 As amy mentioned there are eight measures for which we had national benchmarks and for three of those measures One cares performance succeeded the national 90th percentile and we tend to think of 90th percentile is The highest achievable benchmark. I mean that that that's a very high level of performance So that was the case for three of the measures There was one measure and that's the developmental screening and the first three years of life measure Where there is no 90th national percentile The measure comes out of or again It's used widely by cms and by many states But the highest published benchmark we have for that is the 75th percentile and in the case of that measure One cares performance succeeded that national 75th percentile For another measure One cares performance was between the national 75th and 90th percentile For two measures their performance was between the national 50th And 75th percentile And then for one measure it was between the national 25th and 50th percentile And I'll get into detail on which measures fell into which category As Amy mentioned national benchmarks were not available for the remaining two payment measures So in lieu of of that one cares 2019 performance Was compared to their 2018 performance and for both of those measures performance improved in 2019 For five measures In this set there was statistically significant improvement from 2018 to 2019 And that includes one of the measures one of the two measures for which there was not a national benchmark So michelle if you could advance Great. Thank you So I want to start with the key Because this describes How points are assigned for performance for each of the measures So for measures where performance is equal to or below the 25th percentile No points would be awarded For measures between the 25th and 50th percentile a half a point Would be awarded above the 50th percentile one point Above the 75th percentile one and a half points with one exception Which is that measure the developmental screening measure where there is no 90th percentile In that case, um, they were awarded two points for performance above the 75th percentile And then above the 90th percentile also two points We don't um, we don't Put our measures into domains the way that medicare does but I do want to note that The measures Broadly follow some of the areas the high level goals in the all payer model So we have um mental health measures We have two of those out of the 10 measures We have three measures that speak to treatment for substance use disorder We have three measures that speak to Treatment or really in the case of these measures It's heading into the territory of outcome measures For chronic conditions And then we have um a couple of more preventive care type measures as well So if you look at the um at the table What we've shown is a brief description of the measure The numerator and denominator We show the 2019 rate and that's where the key comes into play. That's where we show You know this the scoring The 2018 rate is provided for reference When we have national benchmarks, we provide those at the 25th 50th 75th and 90th percentile We then show the points that are awarded for performance And then bonus points awarded if there is statistically significant improvement So the first measure is 30 day follow-up after discharge From the emergency department for alcohol and other drug abuse or dependence For that measure, um the aco was in fact Above the 90th percentile benchmark It was an improvement over the 2018 rate And in fact, it was a statistically significant improvement so that um resulted in one bonus point A companion measure to that is the 30 day follow-up after discharge from the ed for mental health And again, in this case, the aco performed in excess of the 90th percentile It was an improvement over 2018, but it was not a statistically significant improvement And so there were no bonus points for that measure The third measure is adolescent well care visits This is our measure with the largest denominator by far In this case, the aco's performance was between the 50th and 75th percentile And so they were awarded one point for that performance It was a slight increase over 2018 But not enough to achieve statistically significant improvement The fourth measure all cause Planned admissions for patients with multiple chronic conditions There are nine chronic conditions outlined in this measure So people with two or more of those chronic conditions are considered as eligible And it's a risk adjusted measure that looks at whether there are planned admissions This is a measure where a lower rate is better And so in this case, the aco did see improvement The rate went down from 2018 to 2019 This is also a measure where we have no national benchmarks And so in this case, even though the aco improved It was not a statistically significant improvement And so they were awarded one point instead of two points for their performance on this measure The next measure I've already talked about This is a developmental screening in the first three years of life This is a measure that does not have the 90th percentile benchmark The aco's performance is quite a bit above the 75th percentile It also represents a statistically significant improvement from 18 to 19 So they attain the two points for performance and the additional bonus point Diabetes, hemoglobin A1C, poor control This is one of our chronic illness measures It is considered an outcome type measure This is a measure where also a lower number is better on this measure And so once again, the aco improved from 2018 to 2019 The rate was better than the 90th percentile And the change was statistically significant So in this case again, two points for performance and the additional bonus point Hypertension, another chronic illness measure, controlling high blood pressure This measure, the aco performed between the 50th and 75th percentile This is a measure where there was a slight decline in performance from 2018 to 2019 Not statistically significant, so they did not lose points for this As a result of their performance, they achieved one point Initiation of alcohol and other drug abuse or dependence treatment This is a measure where the aco performed between the 25th and 50th percentile This is a measure that we've been working on intensely in Vermont It's a challenging one Because of the aco's performance on this measure, they received only a half a point And the companion measure to that, Michelle mentioned this in her comments But some payers look at this as a composite In our case, we look at this as two separate measures And it looks at after sort of the initiation visit That's the person receive at least two more visits in the next 34 days After a new diagnosis of substance use disorder And so in this case, the aco's performance improved from 18 to 19 And it was a statistically significant improvement So they got the bonus point Their performance was between the 75th and 90th percentile And so they got one and a half points for this And then screening for clinical depression and follow-up plan This is a measure where, again, the aco improved from 18 to 19 It's also a measure where we don't have the national benchmark So we looked at change in performance over time This was a statistically significant improvement And so they received two points So the total points with the performance points And then the statistically significant improvement Resulted in 19 points out of 20 potential points So that was, again, an overall performance of 95 percent I just want to say, you know, from our perspective This is strong and very encouraging performance To have this many measures above the 90th percentile Above the 75th percentile Seeing this level of improvement from year to year These are really encouraging results And I think reflective of the really hard work And good work that providers in Vermont Are doing on behalf of Medicaid beneficiaries So that does it for our presentation And thank you very much for your time and interest Thanks, Michelle, can you hear me okay? Yes Great, okay My screen is not synced up with my audio here But I think it's working all right So hi everybody, I'm Andrew Garland I'm the Vice President of Client Relations and External Affairs From Blue Cross to Shield of Vermont I have to admit I'm somewhat pinch hitting today This isn't my area of expertise But the person who would normally give this presentation Is unavailable today Joining me is Micah Devers One of my colleagues He's in our quality improvement area He's really an expert in these measures And Micah agreed to join today In case you have a detailed question So I'll take us through the high level And then we'll see where your questions go So with that, Michelle I think you can just turn us to the next slide And I'll start just by saying That this is really important to us And I'm thankful to the Green Mountain Care Board And to Medicare and Medicaid And all of the folks on the line today For the dialogue The all-payer model And the work that the ACO is doing Is directly aligned with our mission And our vision for a healthier Vermont And a healthier healthcare system for Vermont So thanks everybody for this dialogue I'm excited to be talking with you today So I thought as a next step On the next slide, Michelle We can talk a little bit About what our program looks like in 2019 These are just some of the high levels And I'll slow down on the quality Aspects of the program So this is a qualified health plan program In 2019 You probably know we had a pretty dramatic expansion In 2020 bringing a number of large clients Into the program But this analysis of course predates that The basic financial arrangement in 2020 Was shared savings As Michelle showed in her great slide Our metrics are a subset of the larger set And we selected metrics that really resonate For a commercial population As others have noted That the needs and challenges Of our population are a little bit different Than those that Medicare And Medicare are facing Medicare and Medicaid are facing Our program works in a slightly different way We ask one care of Vermont To set aside every year Dollars that equate to about 1.5% Of the total cost of care And then they either distribute those dollars Out to the providers in their network Or they retain the dollars To reinvest in quality program Depending on their achievement On the scorecard each year So for points they achieve That frees up dollars to Distribute out to the network And for points that are missed Those, that share of the dollars Or the dollars that correspond to those points Stay with one care To be reinvested in next year's quality program So the idea is Whether we're hitting the measures Or missing them Or not hitting them as hard as we want Either way we're reinvesting In the network's ability To deliver quality improvement to our members We do have in our contract Some collaboration requirements So we really try to stay close To one care as they're doing this work We recognize that Blue Cross with Sheila Vermont Is a very important supporter So we pay close attention to what's going on And we ask it every step What can we do to help And then I'll just note that There was a pilot program going on in 2019 Actually continues to this day With the University of Vermont Medical Center As our first ASO client To be participating in the all fair model With a relationship with one care Vermont I won't be reporting on those results today That's a very, very small pool And I don't think we would want to equate their results With the QHP and reporting on them independently I think it's not going to tell us much Just because the number of members is so small And then on the next slide Before we dive into the details I thought it was just worth reflecting a little bit As Kelly did when she made this update last year On what some of the bright spots and challenges are That we face When we look back at the 2019 Arrangement with one Karen Some going forward You'll see I snuck a 2020 on there at the end of the challenges Because I have the life space to do so Maybe most important on this slide Are those first two bullets under bright spots And really taken together To me they say that the Blue Cross team And the one care team Have a great collaborative approach to doing this work You know, if I look back at this arrangement And I've been working on payment reform in Vermont Since I started on payer-related work Or provider-related work back in the mid-2000s I would say that up until two or three years ago Most of our work in this space Could still be characterized by the word negotiation When we worked with the ACO Or other provider organizations before this The tone of the conversations in both directions Was really dictated from a contracting point of view Always negotiating on behalf of our organizations But I feel like in the last year or two with one care We have really moved past that dynamic And fostered a new dynamic that would be better Characterized by the word collaboration And as we've encountered a number of challenges Sort of external to the program But the challenges that nonetheless have impacted The program the most obvious being the pandemic We've been able to respond together quickly And in virtual lockstep in a way that I just think We wouldn't have been able to do for five years ago So that's really remarkable It's an extremely strong base to build from Another really great accomplishment for us in 2019 Though it didn't technically go live until April of 2020 Was the work we did together to build the first commercial Prospective payment system Sorry about that title Prospective payment system And one of the first that we know of Of its kind in the nation And that was a really challenging undertaking Particularly on the psychology side And we're really thrilled with what we accomplished together On the challenges side Again, I would take those two bullets The first two bullets together I think they're largely saying something very similar Which is it's still difficult for us to look at All the data that we have on our members And clearly distinguish the line between those that are in The one care model and those that are not So one of the challenges of using this Type of quality scorecard program Is that it's not as directly tied To the work that the providers and the folks at one care Are doing every day as we would like And these are pretty bottom line measures And there's an awful lot of things that can affect them Between the work that the providers are doing And the time that we actually get around to measure this But you'll see it's not got a bright point in there Under that first challenge We've already worked with one care To come up with a new approach To our scorecard program for 2020 So that we're not relying so heavily on Measures which are so far downstream But will have more built into the program That really allow us to see One care's work plan And you know look more closely at the results That are coming directly out of their actions So that's pretty exciting stuff And then a final point You know I heard the host Sarah Michelle makes this point Which is talking about the Medicare program COVID-19 a major disruptor Both on our ability to measure quality results But of course as we move through 2020 and into 2021 A major disruptor to providers And their ability to engage with us On any new quality improvement initiatives Whether they're through the all-payer model Or things that Blue Cross and Shield of Vermont Would be working directly on So I think we'll obviously have a very different discussion When we're together next year To look at the results of 2020 Okay so let's Oh right a few other things I wanted to point out There were some really exciting Collaborative initiatives that we worked on together In 2019 and some more that we have underway in 2020 I mean the two most exciting for 2019 here on this page One was us really working differently with one care To try to provide to them actionable practice level data And rather than relying only on high level data extracts Our quality team packaged some information To make it easier for them to serve out Things that could be given directly To the provider population To help move the needle both on the mental health Substance use disorder side And then also as you'll see And data related to some of the quality metrics On the scorecard And then we also started working with them in 2019 On a really exciting program That's a little bit aside from the quality scorecard Or much higher up to get more folks into primary care One of the things that our quality team That our nurses work on all the time Are interventions designed to find those members On our books who are not accessing their Their primary care position services regularly Trying to make contact with them And encourage them to go in and use Those preventive benefits that are available to them So in this initiative we've worked with one care To try to move some of that communication And encouragement from Blue Cross Blue Shield Of Vermont directly to the primary care position office With the idea that people are more likely To pay attention or respond If they're getting that encouragement directly from their position So that's pretty exciting work And then on the next slide I just listed out some other chronic condition Some other chronic condition management initiatives That we had conceived and begun talking to One care about even before the pandemic And you'll see that they all involve Some form of telemedicine or telemonitoring So we're encouraged that These are initiatives that we'll be able to continue to pursue Through 2020 and 2021 Even in the face of this very different pattern Of utilization that we're seeing across Vermont Because these still involve Our telemedicine or telemonitoring So they're things that can go on even in the absence Of the same kind of in-person care That we are used to seeing before the pandemic So I've deferred a big finish here A lot of suspense to the actual scorecard So our scorecard has just nine measures on it That are considered for payment So these are the nine measures That affect that half a percent That one care sets apart As Michelle indicated at the beginning Of the presentation We also look at a number of other measures Caps measures primarily Those are reporting only measures You can see here that Where we've indicated a check mark One care achieved full quality measures On a particular measure And they did that by exceeding That national 90th percentile that Pat mentioned And we did make a change to our program In 2019, I believe So that in order to achieve full points On a measure Even if it's a measure that's as tough as say That initiation engagement of alcohol And other drug dependence treatment Where our starting point is way down Below the 25th percentile Or around the 25th percentile In order to achieve full points They have to get to the national 90th So in one care was willing to take on that challenge Really to recognize that we shouldn't be satisfied And say that we've fully accomplished What we needed to accomplish on any particular measure Until we're the best in the nation Our program also allows them to earn some bonus points For having statistically significant improvement In at least two measures In any particular measure And they did that on two measures last year So I didn't indicate this on the slide I'm not sure why I apologize They earned 14.5 points out of a potential 18 Because of those bonus points Which I think equates to about 81 percent Of potential points And you can see measures like that Initiation engagement of alcohol And other drug dependence treatment It's a long haul for us to get that up To the national 90th So we would not have expected them to get to 100 percent That's a really, really high bar to accomplish But still the way we've structured the program The dollars that don't flow back To the provider network again Remain with one care To reinvest in work on those measures That aren't at the national 90th this year Another thing you've probably noticed on this slide It's kind of jumped out There's the ups and downs I think it's important to keep in mind That even though this is a fairly large pool It's still a fairly small pool So the denominators on some of these measures Are relatively smaller I think are going to be significantly smaller Than what we would see either in the Medicare pool Or the Medicaid pool So we would expect some more ups and downs And then I did take a few minutes and look back at the I looked back at the 2018 scorecard Just to remind myself And though the point system was a little bit different The overall accomplishment is pretty significant We had three measures in 2018 That were at the national 90th Statistically significant improvements In a couple of measures And then a couple that swung in a different direction So you know consistently good work here And we're really optimistic about House government operations committee We are gathering here this morning To and beyond as we move into this new approach That takes more of a work plan approach And the scorecard is de-emphasized a little bit We'll still of course measure all of this stuff You know that's extremely important But moving the payment The payment part of the program To be more directly tied to the transformative work Will be a really interesting experiment And we're thrilled that one care was willing to go there That's all I had I'm not unlike Pat I can't take you through these measures one at a time But I think she covered all of them in her slides So thank you Thank you Andrew So with that we'll turn it over to board questions And you've got the full panel here Including Tyler from One Care To jump in and answer any questions That you might have for them as well Or any questions that us payers can't answer So I'll start off Pat as really the mother of the quality Measures in the state of Vermont Is there anything that jumps out at you that is alarming you? No, I don't think so I do think that you're seeing where some of the challenges are Initiating in particular And even engaging people in substance use disorder treatment Is a real challenge And I'm sure there are multifaceted reasons for that That have everything to do with capacity, geography But those are even the engagement measure Where we tend to do well If you look at the absolute rates Both for Vermont and nationally They're very low And so just continuing to work in those areas Where we know that treatment helps And we've done a lot in this state In terms of access to treatment And types of treatment But that continues to be a concern On the plus side Seeing some of the improvements And some of the outcome measures Like the diabetes for control measure And just seeing the across-the-board improvement And the rather high level of improvement Anything above 75th percentile is strong performance And so it's really again I just want to say it's a reflection Certainly of the work that people have done together The ACO payers, aligning measures But also the providers really doing the hard work Of quality improvement That to me is very encouraging So I saw much more to be encouraged about Than to be alarmed about Any questions for board members? I had one question for Pat as well When you look at the quality measures Where we were saying the achievement Was I think 19 out of 20 But when I look at it I'm looking at it, it was like 15 out of 20 And then there were four bonus points So how do we look at that? Because wouldn't that really be There were 24 possible points? There's more than that But they achieved some bonus points So there was 15 out of 20 And then they got four of the bonus points So even if I said it was 15 out of 24 I just want to make sure we're reflecting The right percentage of how we're doing And obviously we're making improvements In a lot of places But I just wanted your reaction to that Right, in keeping the idea of the bonus point Was to allow them to earn more We did not change the denominator So if you look strictly at performance It was 15 points for performance I will note that if they had statistically Significantly decline They could have lost some of those bonus points as well But there's no question that was an opportunity To for them to increase the score I, you know, I was I think we were pleased And surprised at the level to which Those improvements were statistically significant Because that's a, you know, reasonably high bar Yeah, I'm not trying to penalize I'm just trying to look at it Yeah You know, from different ways of the math So thank you Yeah So I'm just kind of looking at slide 14 You don't have to go there But it's the verbiage was that of the 16 measures carried into 2019 Improvement improvement was noted in 13 measures And I'm just wondering what does improvement mean Because when we get down into Medicaid And I have a question there But in Medicaid they're talking about statistical Significant differences And in Medicare its improvement was noted And so I assume that's on that 90, you know Percent on that percentile And was there a spread that was Assumed that if the measure increased by a certain number in the spread It would be considered an improvement Yeah, so for Medicare We do not yet have the statistically significant analysis done From our federal partners And so that's something I'm waiting on So like I said in those measures where we did not see an improvement We saw a decline in the score It's so minimal in so many cases that we're I'm not sure if it's statistically significant or not And the same goes for the noted improvements There are some where I could make an educated guess And say that I think that they would have statistically significantly improved Just based on that sort of gap But without that analysis I'm not comfortable in answering that on their behalf Once we do have that analysis I'll be sure to pass it along to the board And we can certainly do an update to this presentation Unless one care happened to have that And I do not But I do think if you just compare the two And see sort of that prior year performance rate And then against the 2019 performance rate But again trying to keep in mind That something that we're working through is taking into consideration Those massive changes in the scale population And changes in the denominator overall And sort of what impact that has on performance Okay and then so looking then at Medicaid And it's a you know five measures There was statistically significant improvement And there was the point of the bonus point approach But what can you talk a little bit more about What statistically significant means in that context I mean what is the hypothesis that X caused Y And so there is a spread in the numbers But is it just the fact that it's associated with the ACO Or my phone's ringing in the back I'm sorry I don't know if you can hear it But so I just if you could have a few more sentences about I mean you know how can we be assured That the statistically significance is aligned With these measures in the ACO program Yeah thank you Tom I you know I think all we can say When you're looking at statistically significant improvement Is all you can say is you know the measure improved And probably at a level that was not due to chance You know what the factors were in that improvement Whether we're reaching a level of alignment of measures And a critical mass of attributed folks That you know that that provider that allows providers to really focus on this I don't know that we can say that But you know to have year over year improvement That is statistically significant Seems like we're really going in the good direction I mean most of our measures improved We saw improvement But half of those payment measures saw improvement As statistically significant level You know I think as Michelle said There's a lot of digging into as to the why And how best to capture this but Directionally things look good at this point Yeah I would agree that I mean the flow is in the right direction And we have two or three more years at least within the all pair model to kind of Continue to track this and learn more and and have more data I just I just you know because of some of the controversy around the ACO Aligning and linking these quality improvements and this positive flow To the ACO I think is an important Consideration as opposed to saying some of the Medicaid numbers are better Because over in the education department You know for kids on Medicaid they improve nutrition You know and it's kind of like you know in a different arena So you know it's obviously the directions In you know in a positive way and that's a good thing The final my final thing is For Blue Cross Blue Shield now that Blue Cross Blue Shield is Engaging with the ACO I I wonder if And I could be totally off base on this but but to me the benchmark plan for the QHP population Hasn't been kind of reconsidered since its inception which I think is in like 2012 or 2013 was The basis for that plan and and one of the areas for example that we see here is diabetes And one of the areas you know in the benchmark plan is diabetes But there's no organized program in the benchmark plan to avoid the diabetes There's no organized pre-diabetic program and I'm just wondering if Blue Cross Blue Shield might kind of urge the ACO and other partners and participants to revisit that benchmark plan In order to make sure it's as aligned as possible With the goals of of the quality goals that we're talking about here I you know I do understand that some who were there when the benchmark plan was first crafted that It was kind of a food fight with everybody trying to get their you know their their Service Engaged but you know, I think you know folks can say there's a limit here We're just looking at prevention now in terms of the restructuring of this benchmark plan And move forward in that regard Yeah, thanks. We we have introduced some optional plans that are really focused on some of the issues that you That you described Tom, but I I'm sure the folks on on our team who are working every day to to Design benefits would would be thrilled to sit down, you know with other stakeholders and look again At the benchmark designs and ask, you know, you know now that we know what we know Where the network is able to engage? How can we do this? How can we do this better or differently? To make it easier for members to make those connections and that that's a The question that's on our minds all the time in fact, I have a task force working on this This question of why why why are people not taking advantage of zero cost share services? Which we know they need and and we tell them they need and they acknowledge and still say well We're not going to go because we're worried about cost So I think those are really important issues and and I will absolutely pass that engagement along Thank you other questions from the board Hi, this is robin. Um, I had a follow-up question for andrew related to The work plan approach. I was interested in hearing a little bit more granularity about how you're thinking of approaching that and um Just maybe giving a couple of examples about how you would use that for the quality measurement Yeah, well, I will admit that we are still working on the specific So if I don't offer them, it's not that I'm being evasive, but the the idea is that instead of saying You know, let's let's move the money around based on the results for adolescent well care visits We want to move that question upstream a little bit or downstream I guess depending on your perspective and we start by asking the aco Okay, well, what are the plans for 2020 for 2021 to move the needle on adolescent well care visits? Let's show us that work plan and let us Let us sit down together and agree on a point structure that is tied to some of those activities That you specifically hope to to put in place and let's find some measurable Between the work and the very very sort of downstream quality measure To see if we can get a better sense of how the action Is is turning into results because to tom's really good question and pat's great answer It's hard to tell. I mean this says as everybody knows there are so many dynamics at play in this healthcare system And you know when you push on something Whether or not the thing that's you know 15 dominoes down the down the line actually falls Can be really hard to know, you know, did it fall because of that push or did something else come in and and influence it So so that's the change we're trying to affect there. I think we'll have a better sense Over the next couple of months what that's going to look like specifically But we were we were really thrilled that the DACO was open to making that kind of change Yeah, it sounds like an interesting approach because I I do think one of the challenges as we've talked about is And that's pat alluded to when referencing scale is that the quality measurement results here at the individual provider level And so there could be a lot of different things going on in an individual provider's office that may affect those So that's interesting Um, I guess the other question that I had that I just wanted to ask The diva team and the blue cross team is Based on the quality measurement in In 2019 Where are the areas that you think it's most fruitful to focus on moving forward in the quality arena Given all the dynamics around The different areas I'm just curious if in your opinion on that I'll start with that. This is pat I you know, I think I alluded to it earlier, but I think continued work In the substance use disorder treatment arena Is is going to be really important you know and and partly because it It it rolls up into The tragedy of deaths from drug overdose and other morbidities From substance use disorder. So, um, you know given and it's a challenge. It's an all hands on deck Approach it's not that we can expect the providers to do this alone It's not that we Can expect the state to do it alone the payers to do it alone This is a multi-year All hands on deck effort And it's one of the really unique elements of our approach to quality, which is that, you know, we're looking at Areas that are You know, it's hard to to Improve in silos They involve public health efforts as well as clinical interventions and having capacity for treatment and Giving people the support they need to get treatment. But to me again when I