Thank you all for making time to be with us here today to discuss this critical topic. I'm Iris Palmer and I'm a senior advisor for higher education in the workforce here at New America. And right now we have an unprecedented need for community colleges to help connect people back to the workforce. After the catastrophic economic shocks of March and April and a quick recovery after that the economic recovery has continued to slow. We still have almost 10 million fewer jobs than we had in February of 2020. Last Friday we experienced the fifth straight month of hiring tapering off. And while temporary layoffs continue to fall permanent job losses are on the rise. And the percentage of people leaving the labor market is at a level we haven't seen since the Great Recession. But people still aren't returning to retraining. There's many reasons for that because of labor market uncertainty fear for their health and the lack of childcare. But we're seeing enrollment in community colleges fall. So it's down almost 10% this last fall and a recent survey found that 43% of prospective students for one and two year programs are looking to delay their enrollment. Because of this and a new administration where the first lady actually teaches at a community college and did her dissertation work on community college student success. We have an opportunity and an obligation to provide additional federal support to our community college system. And we can learn a lot about how to structure such an investment from the $2 billion federal investment in community colleges that we did make in the last recession. So over the last two and a half years, our team at New America and Bragg and Associates have researched the outcomes and implementation of the trade adjustment assistance community college and career training program. From here on will refer to that program is tax because it's easier to say. And we learned a lot about the impact of this program. We looked at how the grants actually improved completion and job outcomes for for participants. We learned about how possibly to design future investments in community colleges that improve their capacity. We looked at the benefits of coaches and prior learning assessment. And today we're releasing our latest paper, which explores how colleges funded to the tech grants built online and simulated learning in collaboration with each other. And lessons from that and lessons from what they did right can really help. I think our community colleges right now as they face this difficult unprecedented time and pandemic. Other tax studies and evaluations have advanced the field and what we know about adult friendly workforce focused and education and training more broadly. And now it's time to capitalize on that improved knowledge base. And that's why I'm so excited to hear from these experts who have spent so much time pulling together this new volume of new directions and community colleges. And with that, I'm pleased to turn it over to Dr Pamela Eddie. Thanks very much Iris. I'm excited to be with you all today. I'm a professor at William and Mary, and I'm the editor in chief of new directions for community colleges. This journal has been around for over 45 years and really looks at the community college context to try to change and look at research to put it in the hands of practitioners. The mission of the community college with open access and workforce development has been historic and has been profound in term of the effect on America. So as we look at transfer economic development and access, I'm really excited about this upcoming volume that's going to be coming out of new directions, it should be out sometime this week so stay tuned for that. The title of it is going to be large scale change lessons learned from tack. And I'm really excited to turn it over to two of the editors of the volume and the third joining shortly. So Michelle Van Noy, Heather McKay are going to speak next and Deborah Bragg was also an editor on that. I'm very excited and hope you access that. Thank you. So this is Michelle Van Noy associate director of the education and employment research center at Rutgers. I'm very excited to have this opportunity working closely with Heather McKay the director of ERC at Rutgers and also working closely with Deborah Bragg to work on this volume. I just want to spend a couple of minutes to share with you kind of the history of the volume and how this kind of came to be. You don't know a lot about tact. One of the distinctive features of it was that it had required evaluations for most of the rounds of the grants. And those evaluations included both a focus on implementation of the grant activities as well as outcomes to actually try and use some rigorous methods to look at outcomes on for students in terms of completion and employment. So one of the real opportunities that we saw as all of us working as evaluators and many of us on this call today is that we could really learn a lot from tact and so we really saw the opportunity to do that and as we pursued our evaluation work. Deborah Bragg was leading the transformative change initiative and she really I think is really the spark between behind all of this is that you know she had been working along with others and to hold convenings through the transformative transformative change initiative that brought together, you know practitioners but also that focus on bringing together evaluators and those of us who are doing that work of evaluating the tact grants and so we would come together through this this network of evaluators that that had been convening and discussed sort of lessons that we were learning and shared, you know, the work that we were doing on our valuations which I think caused the creation of a very fertile exchange of ideas and a very good community evaluators and through those conversations. We just felt like we were learning an awful lot and we were working closely with our college partners to learn a lot and we saw the opportunity to try and share more broadly with the field what we were learning. And so through that network and through those conversations this really kind of came together in this idea to to put together this this volume and so Heather and Deborah and I decided to sort of take some of the work and you know through those those TCI convenings that we never had convened people had already we already had some some papers that had been written and so we really saw this as being a nice opportunity to build off of that work and to to put this into a volume so that's sort of the legacy I suppose of what what happened and how we came to this volume and so we're really so thrilled to have today actually this culmination of all the people that have been working together with us to share these ideas and and have this volume is something that you know is hopefully well poised to to take what we learned through all of our good work on TAC and and bring it to the field and to a to a broader audience because as I think we're all working on these evaluations we saw the the richness of what we were learning and the opportunities to share from that so that's all say I'll hand it off to Heather who can speak more about TAC and queue us up for what we what we've learned. Thank you so much for having us today and and for joining us today to talk about tact a little bit I think many of you know quite a bit about tact already who are who are on the on the attendee list here but I'll just provide a brief introduction. So the trade adjustment assistance Community College and career training grants tact, as Iris told us earlier, we're launched by the United States Department of Labor at really the height of the great recession 2008 to 2010. So as we think about today, and we think about kind of the unprecedented economic challenges we're in with the pandemic. Many of the lessons we've learned here, we think can be applied to what we're going through right now and and thinking about how both workers and institutions can emerge from this crisis. There were. This was really an unprecedented amount of money put into Community Colleges over four rounds it was almost $2 billion. It was focused on building capacity for Community Colleges to train unemployed incumbent and new workers for high demand industries and occupations. The program sought to achieve the goals of short term increases in student outcomes and long term institutional change, and the authors on today's webinar will address many of the topics that tact address including career pathways, online and technology enabled learning, transferability and articulation of credit alignment with the workforce system employer and industry engagement. And of course we are all third party evaluators who looked at these projects with that lens. I'm excited today to be a part of this and for you all to hear about this tremendous amount of work that was done by the evaluation teams and by these projects. So thanks so much. Great. Thank you all so much. I am more and I started a senior fellow at the Urban Institute and I'm going to be talking today with Deb Bragg who is we're co authors on the concluding chapter of of the volume, and we're really excited to talk about the lessons we've we've learned I think if you want to go to the next slide I we, you know really tacked was about building a body of evidence of Community College. We did a number of initiatives innovations. We really did that through multiple ways. I was project director, as well as cookie I on the national evaluation for all four rounds we worked across multiple organizations, including the Associates George Washington University Capital Research Corporation and North at the University of Chicago. And it was a large effort to look at the implementation of the grants across all four rounds conduct synthesis of the third party evaluation findings, upon which are our lessons today. There's an outcome study that that's part of round four that will be released sometime this month is my understanding and a lot on special topics so we looked at employer perspectives we did some briefs on career pathways and systems as well so all of that is coming out on the Department of Labor website I know we can put that in the chat function as well as well as on the urban website the rounds one through three reports are out so they are available for folks to review and round before coming as well. And then, as Michelle mentioned the third party evaluations there were over 200 evaluations that is a lot of evaluations to be able to draw from, and look across an entire initiative and our synthesis relied on that as well as the ones that new America did. And, and then the work as well as the transformative change initiative so just a huge body of research coming out of the tact grants. Oh, and yes, thank you somebody put in the chat function you can get all the third party evaluation reports on skills commons.org. So I just wanted to just say a few words here about how we think about community college systems change and the Urban Institute has an explainer on this and we did some events but you know we really do think of it in three buckets. Around accelerating learning college persistence and completion and connections to employment, and you know many of the community colleges really were focused on changing their systems the main goal of the grants was building capacity for serving adult learners. And, and so they did this in many ways, and they did this within colleges across colleges and across state systems so there was a lot of work done on different levels. Next slide. So, you know, while there were many challenges to the grants we did learn a lot, especially about implementing systems change. And, you know, we have five insights that specifically come from our round three synthesis of the implementation findings we took. We took a strong look at systems change and the role played in the grants through the lens of the third party evaluators and round four evaluation also has a focus on systems change as well so there's more to come. But first really go back Ivy, go back one. So, one of the first insights we have is that embedding collaboration is a core element of the initiative really helped to support community colleges capacity building efforts they had internal partnerships, you know with different departments like student services, technology, external partnerships, which could be with public workforce system, community based organizations. And, you know, we did see a lot of employer engagement and so a lot of our work as well as a chapter in the, in the volume really does speak to that that aspect of of tact and lots that can be learned about that as well. And then there was a lot of collaboration across colleges so colleges were learning from each other they could have been part of a consortium. They could have been part of the transformative change initiative. They could have been talking to each other through coordinated efforts through the Department of Labor. So there was just a lot of collaboration going on across the grants. We did see a lot of grantees creating a continuum of supports. So, you know for students to really be able to succeed and the continuum went from enrollment through to employment. And there are a lot of places where students could be reached and be supported. And one of the chapters in the volume really does highlight the role of what we learned about advisor navigator counselor roles, especially around career navigation that supported educational workforce success. Next slide. Grantees also really created a comprehensive, many created comprehensive approach to career pathways. And what we mean by that is that there were many elements that went into a career pathway, not just one component of it so we saw pathway or comprehensive approaches that included curriculum stacked credentials transfer and articulation really that continuum, and that pathway was well spelled out, and there was mechanisms in place to support that advancement. And we do have several chapters in the volume that describe comprehensive career pathways, and can really provide some insight on how that worked. Fourth, we did see the grant grantees benefited from efforts to evaluate the grant projects implementation and improve it throughout the grant period. The evaluators help them create logic models monitor implementation and participant success. We saw one of the chapters in the volume talked a lot about developmental evaluation and you know what and the authors of the chapter really described how they used that method that for six third party evaluations, but evaluation can really be a challenge for grantees to really understand the use and value of those evaluations, as well as understand the especially the more rigorous impact methods. Next slide. I also wanted to highlight the role of state education and workforce agencies and the important role they could play in tax grants. They really did help colleges develop data sharing agreements so they could attain education records, and wait records and you know what was very challenging just data on their, their, their, their tax students, but also data on comparison group members and it just it was, it was very challenging if a college and a grantee didn't have a formal role, or for the state agencies that hold that data. The state agencies also and especially state system offices could really support creating statewide policies and practice within the colleges as a part of a tax grant to support adult learners. Next slide. And finally, I just wanted to say a little bit about the evaluation and saying it very broadly in terms of all, all of these multiple efforts going on. You know, I think there was a lot that went into, you know, through transformative change initiative for Department of Labor, providing evaluation technical assistance to support more rigorous impact evaluations. I think that was a really big challenge for colleges and, and their evaluators to really do well. And, you know, we, we saw fewer than half of the grantees doing more rigorous quasi experimental methods, although the, the prevalence of them did increase over over time and over rounds. You know, there also might be some consideration for incentives for conducting rigorous evaluations really carving out a particular aspect of a community college initiative that you want to test, maybe prior learning assessment. You know, there's just a lot of different things that could, could really be informative but doesn't have to over, you know, be everything within the initiative. And just as I spoke, the formal role for state education labor agent agencies to really support access to administrative data could really help improve the rigor of evaluations in the future. I've heard a lot from third party evaluators through their through different events and as well in their reports that, you know, they really did need a longer follow up period for capturing the impact of the grants on participants, especially their employment. It was often too short of a window to really understand what happened after they participated in training. And finally, you know, I think, you know, the other piece of this is making sure these findings gets get in the hands of both policymakers and practitioners, and we're doing a lot of that now by having this webinar, we're, you know, putting out briefs. We had had different events blogs across all these evaluation efforts but you know, considering where we are right now in a recession, and the need, you know, coming for retraining people. You know, how can we take what we've learned from tech and really get it into the hands of the decision makers and make it usable for them. So with that, I'm going to turn it over to Deb to talk about the rest of our chapter and some of the lessons. Thank you, Lauren. It's really a pleasure to be here today. I'm Deborah Bragg. I am the president of bragging associates at consulting firm that's committed to more equitable outcomes for our learners. I have also been working as a fellow with New America over the last year and I want to thank my new America colleagues for being such great supporters of this work. And build on what Lauren has talked about the final chapter of the new directions, kind of takes a step back and looks at all of the chapters that were contributed by the authors, as well as, you know, our own work. In the course of the tack grants, I was an evaluator for six tack grants, mostly consortium, but there was a single institution as well. I also had the opportunity to direct the transformative change initiative for a number of years with mid-Mindy Feldbaum of the Collaboratory and that was a really interesting and rewarding event, you know, and in my career. I thought about all of that and thought about how I could approach what I wanted to say about tack and decided to go to kind of an old, I want to say old, an experienced and esteemed scholar on educational change, Michael Follin. Some of you probably know Michael Follin's work and in K-12 higher ed and he also does work in the private sector. He has a very new book on leading in a culture of change. And I thought it would be interesting to apply his framework for what he is saying really is important to educational change and look at our work in tack and figure out, you know, do we see evidence that we were engaging in evaluation as evaluators of community college change that was focusing on a culture of change. So there are five areas that Michael Follin talks about and those are the ones you see on the screen about purpose, nuance, relationship building, deep learning and coherent change. So I'm going to use those that framework and speak to each one just very briefly to give you a sense of what I think we have to say about that from the tack perspective so idea if you'll go to the next slide. So moral purpose. I think a really critical point that Michael Follin makes is that change and when I when I'm talking about change and when I think about the tech grants. We saw a change or the potential for change on many different levels. We saw programs potentially change. We saw community colleges potentially change we saw systems change. We saw partnerships and relationships change. We saw policies. So I'm thinking of it in that very big sense. And what Follin is saying is it's really important to focus on what is changing, but also how change is happening. The process of getting to change really matters and leadership matters. And that is so critical. It's so critically important to root that in what he calls moral purpose, and he talks directly to educators to say moral purpose should be so easy, so obvious for those of us in education, because we're all about transforming people and and making, you know, changes in in lives. And you can see that in the tech grants you can you can see that in the way the grant proposals were written. And we the evaluators talk about what community college leaders were trying to accomplish. So the, the importance of moral purpose, particularly in the context of tact, which was to help people overcome the challenges of the great recession and move you know sustaining wage employment is is very, very prevalent as a core purpose. And our colleague who you'll hear from later to share a gale. I thought captured this so beautifully in her chapter. When she makes the connection between the work intact and advancing and sustaining equitable educational structures. You know that are linked to communities that really are in need. And that's what tack was really about. And we see that over and over again in in the evaluation so I think moral purpose is one that's quite prominent in the tech grants. The next slide then is about the nuance of change. I love this statement in Michael fullens book that organizational cultural change is more a learning problem than an execution problem. Our authors speak to that they speak to the evaluators, what they saw grantees trying to do. We saw a lot of folks struggling with execution but we also saw a lot of learning and a good example of that is colleague here Heather McKay and and her colleagues Suzanne Michael, right about developmental education reform in the in the community college system and their evaluation and their chapter is very honest about the successes and failures that the struggles and the complexity of carrying out developmental education reform in the community college. So they bring our attention to the ways in which processes were put into place that enable those who are part of the grant to learn from one another. And one, despite the challenges, one of the successes was, as they say here, you know, when they were able to find faculty and staff, you know, who really this this initiative met their needs and interest, then progress was made and when that didn't happen reform lingered installed. So the importance of learning and I'll come back to that. The third is about relationship building. And Lauren you spoke to this a little bit. The importance of relationships to leverage collective action that enables change to happen. One of the things you know you probably heard and I've written is no one changes alone. These kind of large scale initiatives require a lot of collaboration and cooperation. In the chapter that John Cosgrove and Maggie Crosgrove wrote about employer engagement. You know, I think really embodies this message. They talk about how they saw the community colleges in Missouri, really, not all of them, but some of them transform long standing program advisory committees that supported their, their CTE programs workforce training, which has evolved into strategic partnerships. Those partnerships, often extended to the workforce system as well, although not always and that is an area where we saw, you know, sometimes change sometimes not. At their point. I like very much their statement that the goal is not to conduct employer engagement activities, but to build employer relationships. And so that chapter has a lot to offer. Those of you who may be looking to improve relationships, you know, despite the complexities. And then the next slide. The focus is on again learning that change is fundamentally about learning. So as a leader of change, it's about creating and sustaining a learning environment where those who are involved in change can really execute the change can make it happen. So here I found some inspiration from Kate Dunham's chapter, who you will hear from here in the use of developmental evaluation and Lauren mentioned this as well. There's a lot to the importance of the evaluators sharing just in time data, being part of decision making where data is really active and alive and accessible to community college practitioners as they were carrying out the grants. And that being so critical to advancing organizational change and we've asked Kate to tell you a little bit more about that so she's going to do that. In just a few minutes. The next slide then is about the fifth is about coherent change. It refers to structures policies and processes that come together in a coherent way to stimulate learning and increase capacity building which is absolutely critical in attack grants. I will say having read over 200 third party evaluations, not every tax grant, I would say produce transformative change, but some of them did. And there were, there are some real lessons about what brings that about. And that some of the factors that we found in the transformative change initiative have to do with the clarity of purpose. Not just the leader but a broad based understanding of what it is we're trying to achieve an alignment of the rationale and the expectations. So folks have clarity around what to do, and the strategies are continuously improved to produce the intended outcomes so that importance of the data piece is always there. So that gives you a bit of a sense of the application of Michael Fallon's framework to the tax grants and with that I'm going to turn it back over to Ivy. Hi everyone. I'm so glad you're here with us today it's really an honor to have you with us this afternoon to learn about all we know about tax so far and what that might mean for us moving forward. So, I'm Ivy love I'm a senior policy analyst in the Center on Education and Labor at New America, and I've been part of the team that has been researching the tax program for the last few years. So excited to share some thoughts with you on what our learnings mean for a future federal investment in Community College. I want to back up for a second to last year when we published with Deb Bragg and her team a meta analysis of the tax program, looking at rigorous quasi experimental studies and what that meant for tax participants. What we found was that taxed participants compared to other participants were almost twice as likely to complete their programs, and they were around 30% more likely to either get a job or see a wage bump after their participation in a tax program. We found that tax empowered colleges to help students. And right now our communities colleges and our workers really need some strong supports, a lot of people are hurting right now are dealing with job loss trade change and need some more economic security. So what we need moving forward is an investment that works, and what we know from our research is that tactic work it did help students achieve their goals and get the training that they needed. So what I'm going to share with you right now is some thoughts based on our research on what those lessons mean for shaping an investment in Community College is moving forward. So I want to first look at the size and scale of the program how much are we talking about here. So in tact, there were three different kinds of grantees we have our single institutions are consortia of institutions that are all within the same state, and then folks who are institutions in a variety of states working together. We feel like a future investment should focus on those single state consortia. One of the things that we've learned from our research is that when groups of schools got together in a consortium to take on new initiatives, they sort of became more than the sum of their parts, they were able to learn from each other take strengths from each other and build something really wonderful to support their communities. We do think there's potentially some space for a large single institution that has the bandwidth to take something like this on just by itself, or potentially a multi state grant where all the schools are in the same local labor market that happens to go across state lines. There certainly be a place for that. One of the things that was sometimes a challenge in in tact was being an institution going this alone, especially small institutions so we feel like bringing them into the fold to work with each other is going to be more effective for everyone. In terms of the grant size, just to give you some scale the smallest tax grants were around $2 million, and the largest were around 20. So we feel like maybe going up to 20 might not be necessary but we definitely feel like the floor needs to be lifted. And I will say that we were advocating for this before the pandemic ever hit but right now especially $2 million the single digits is maybe not enough to support colleges like they need to be supported right now to achieve their goals so we feel like moving grants between 10 and 15 million would be appropriate. In terms of the timing of the grants, the four year time span for tax seems appropriate to us that seem to work well for lots of grantees. One thing harkening back to what Lauren Eister was saying earlier in the presentation on evaluations, extra time to collect data is really going to be important for a future investment, especially for colleges that started an associate degree program through tact or a longer term certificate. It was really hard to collect outcomes on those students with such a short time frame. So if we really want to understand the impact on those students, lengthening that window to get data will be really important. In terms of the way the program is administered we felt like having the Department of Labor and the Education Department working together on tact was a strength. And so there is some precedent for the creation of an interagency office that has detailed staff from multiple agencies, working together on one program. We feel like this is an appropriate space to have such an office in the future. So there's a lot of investment coming off the top to support those staff and those in that office. The school to work program that some may be familiar with had a similar office comprised of detailed staff from multiple agencies so we think this might be time to bring that sort of structure back. There also should be a state role in this as with tact states are greatly important I'm going to hop to the third item in that little list there which is data. I mentioned before me, sometimes getting the data on labor market outcomes for tact participants turned out to be a real challenge for grantees. So we feel like thinking about that from the beginning and setting aside some resources in the in each grant to go to a state agency that handles those matters could go a long way to facilitating evaluators work. And then in terms of moving beyond the four year grant period states have a lot of power to convene to scale and to keep good initiatives going so we would like to see some resources directed towards state agencies toward that end. In terms of what folks do with the grants that they would receive from a new initiative. Just on this first point here. Last year, Senators Cain and young introduced a bill on investment in community colleges, and they suggested that it should be structured where the first 25% of each grant would go to student services. We think this is really wise and we pick we are picking up on that here we have borrowed that from them. One of the things we learned from tact is that having a success coach or a navigator and good wraparound services made all the difference for lots of students. I can't tell you how many tact evaluations we've read well I can we read 220 but I can't tell you how many of them include interviews with students who said had it not been for my coach I would not have completed. Had it not been for my advisor I would not have made it through this program, and so having those personal relationships and the support resources. The structures within and across institutions to make that work is a really critical and initiative that should be prioritized in a future investment. In terms of what happens beyond that we think that having folks focus on one or two initiatives that they really want to do and do deeply is going to be most effective. Sometimes intact. If grantees tried to do seven or eight new initiatives that turned out to be a little bit much so they struggled with timeline or just juggling all of those priorities so we think scaling that down to a few things and really allowing grantees to focus on those initiatives that make the most sense for them right then is what will turn out to be most effective. And then the second point here is in some ways related to the paper that Iris Palmer and I just published today looking at online initiatives and simulated learning initiatives through tact. The four profiles that we wrote about tact grantees who had successful initiatives in these areas are still going. In some form or fashion they are carrying on. So we feel like a future investment should allow grantees to use resources to keep good work going to scale out initiatives to new communities to new institutions in their state. Initiatives that are supported through this grant we don't feel like they necessarily need to be shiny and new they just need to work. And if we know something that works, we think more power to the grantees to use resources to continue that. And finally I am going to share a couple thoughts and evaluation that are very much in line with what Lauren was sharing earlier. So we know that all tax grantees had, or at least the last rounds that's a fine point, did evaluations with third party evaluators. And some were able to do rigorous quasi experimental design impact studies and some were not. However, even the descriptive work, even the narrative work, the implementation studies have so much knowledge and wisdom to share with us as we're trying to figure out how to move community psychologists forward today. So we feel like in a future investment, having all grantees complete an implementation evaluation, and to take a look at sustainability in an evaluation would be really valuable without requiring that they complete a quasi experimental study that just may not be possible for everybody. However, as Lauren mentioned we think some incentives could go a long way in getting some grantees to work with evaluators to do such a study. In the blueprint for federal investment in community colleges that Iris Palmer and I published in March, we recommended that around 30% of grantees be able to access additional funding to share with evaluators who could complete a rigorous quasi experimental impact study. So we can have a good idea about the impact on participants without requiring everyone to do that and allowing other grantees to focus on the descriptives, the implementation, the qualitative work that is again still richly useful to researchers to institutions and to policymakers. So that's it for me. I'm going to stop sharing my screen. I will drop my email address in the chat Iris and I would be delighted to talk with any of you, we've been thinking and talking about tact a lot we would love to think and talk about and with that I'm going to hand it over to back to Deborah Bragg. We're going back and forth here who will be moderating our panel of experts. So thank you Deb. Thank you Ivy thank you for doing such a great job of wrapping together some of what we learned with some thoughts about future policy. That's a great place for us to queue up our awesome panel for for you here today. I had the pleasure of moderating this panel and I want to start with introductions. We have six panel members with us today. We have three speakers in the work at the at the federal level and also a number, mostly chapter authors and third party evaluators so bear with me. This is such an impressive group I just want you all to to know who these folks are before we start firing questions at them. I do have the introductions in about the order that I'm anticipating that people will respond to questions. So I'm going to start by introducing Cheryl Martin. Cheryl is a program manager for competitive grants at the United States Department of Labor in the employment and training administration division of strategic investment. Cheryl has managed the federal tax grant program since 2015. And as you all might suspect, she's played a pretty critical role in leadership of administration of the tax grants that you that we've been talking about here today so thanks for joining us Cheryl really appreciate it. Our second panelist is Derek Price Derek is the principal and founder of DVP praxis praxis. He is he is a supremely experienced tax evaluator, having evaluated as many of our speakers but tax grants across the country and especially work very closely with the state of Wisconsin and the technical college system they're working with Brandon Roberts and others on their team to lead that evaluation work. Derek's firm provides formative and summative evaluation services strategic advising and technical assistance and training across the post secondary education and training system so thank you Derek for joining us today. Our third panelist is Tashira Gale. Tashira is the director of evaluation services at education higher ed insight. Tashira was part of the third party evaluation teams for the national STEM evaluation, which was around one evaluation, but she was also part of six other evaluations, and if I have this right. She was part of it, five evaluations in round four. And I think she must have been the busiest evaluator on the planet from about 2014 to 2018. Thank you for being with us to share we really appreciate your input. Our fourth panelist is Jennifer Freeman. Jennifer is the senior director, a senior director jobs for the future. She led the technical assistance project at JFF for the US Department of Labor ETA. In that contract Jennifer was responsible for doing a lot of the documentation, convening and technical assistance, along with the Department of Labor of those, all of those grant grantees so another very, very busy person. I'm also really thrilled to have John Cosgrove on the panel with us today. John is a senior partner with Cosgrove and associates a consulting firm that he and his partner Maggie Cosgrove have led over the last many years. Starting Cosgrove and associates John worked in the community college sector as a director of research and academic vice chancellor and a faculty member and obviously brings all of that rich experience to his evaluation practice and we appreciate that John. Our sixth presenter or panelist then is Kate Dunham. Kate is the principal and director of project operations and the director of workforce and human services for the social policy research associates. Kate has over two decades of experience conducting research and evaluation on employment and training programs and the public workforce system. And she shared with me that she just has recently served as a co principal investigator of a study that looked at the implementation of DOL workforce innovation and Opportunity Act or WIOA. And a component of that was to look at the connections between post secondary CTE and WIOA. Thanks for being with us Kate. So, clearly, this is the A team. And we're just so thrilled to have you with us and just to remind you a couple of things. Just about all these folks have chapters in our new direction so you can read even more about some of their thoughts as they respond to questions, but I also want to remind you that we are taking questions. If you want to enter questions into that chat box, we're monitoring those and we hope to get to those. So I'm going to start with Cheryl. You know, there's so much that was learned Cheryl we're very interested in your perspective. So what do you think are some of the most important lessons from TAC. Well, thank you Deborah and thank you so much for including us even though we didn't write a chapter in the book. We thought you were tired. Yeah. That and a couple of other things we've been doing since then right. Yeah, but anyway we're just really excited about the research and the book and all of this and so I thought that I would focus you know you all are the evaluators and the experts on that. There are so many important lessons already just today. So I decided that I thought I would focus on two things where DOL, you know, sort of in our lane as a DOL program office and one of those is first some observations about technical assistance. And secondly, how DOL uses the findings from evaluations to shape future grant opportunities, because I thought you might want to know. Let me start first with some observations about technical assistance. This is related to the TCI transformative principles four and five see I read those things, which speak to peer communication and learning. And so both of those of course are critical components of TA learning from each other. And they say, you know, they say that you really only appreciate something when it's not there. And we can talk a lot about that in the last nine months right, but that could definitely apply to technical assistance for tact because while from the beginning there was a very small but mighty team and I'm talking about like four people who supported tact as feds. And then, you know, and that was before I came on, and then we must also acknowledge the critical role of all the DOL FPOs that federal project officers and providing TA. During the first half of tact when federal dollars were not available for various reasons, we observed a really high value to grantees in the TA that the Gates and Joyce and Lumina Foundation stepped into fund through the TCI initiative. One could just pause and think, what if they hadn't. So, you know, that would have been a really big lost opportunity I think for peer communication and learning, both around program design, but also laying those early foundations for evaluation, and bringing together evaluators and learning them learning from each other. So what we're doing here today and with this book is a real direct outgrowth of that. So, there we go. I wasn't working on tact during the TCI years and unfortunately but I came on right after round four was announced and when JFF and skills commons support was announced. And when we were also able to expand significantly on the national evaluation because that all kind of came around at the same time. We got three years into the seven year period where we had active grants, and I can say that we observed considerable value again in the federally funded TA that DOL was able to engage for the last four years of those active grants. So sometime we'll have a conversation about whether we could have had more continuity there I suppose but anyway, for tact that was not only key to helping identify the grantees that we're having challenges and that's one of the things we always do with technical assistance and you know providing them some extra support and things like that, but also to documenting learnings before the evaluations came along.  The last grants. But before that, you know, and of course some of the information has come out before this but you know you heard about some of those learnings early from JFF and chapter two in the book but this was or you will. But this wasn't just topical TA such as strategic employer engagement or credit for prior learning or success in student supports or a dozen other topics that we, you know, that were covered, but also identifying the factors and the processes that were the underpinnings for success in those systemic changes, because tax with tact was such a complex mashup of, you know, change in certain specific areas like those areas, and then those process changes that it took underneath that to make that happen. So anyway, the technical assistance gave the opportunity to write about some of that while it was happening it was kind of kind of a form of developmental evaluation if you will I mean not quite maybe but anyway it was it was while it was happening that we were talking about it. Another critical form of TA and support that was really successful and perhaps one could even say revolutionary was requiring all the curriculum and other materials that developed with tap funding to be licensed as open educational resources, and then creating the ability to fund the development of skills Commons, the repository for all that effort because without a place to put all of that and the place to go to to find it, you know, how useful would it be. So that effort is one of those things that I call a ripple effect of the changes on the specific campuses so changes happened on campuses, they happened across states and consortiums they happened across the nation and multi state consortiums. But then there was this ripple effect of all these changes that went out, you know, nationwide and worldwide frankly, the fact that skills Commons is still seeing 200,000 downloads a quarter. That tells you the value of making those resources available, and of, and the effort, you know that that was involved in that so that's another piece of it. So yeah, so that's one of the things I wanted to talk about and but besides learning about those changes on. Sorry, I wanted to move into a second part about how do you see DOL how do I see DOL using the findings from the evaluations and its future efforts. And I'll just go through that pretty quickly here. We're using the findings in all our grants, not just those for community colleges. So no tact, as we talked about was focused on supporting adult learners, coming back to college after being in the workforce and helping unemployed adults gain skills to make them re employable. This has always been a big emphasis at DOL. And it's especially so again as we seek to recover from the latest, you know, economic challenges that we're having with coven just like the nation was coming out of the great recession during the tact years. We have broader implications for ETA's job training efforts than only those that we are finding at community colleges. Just for example, employer engagement and career pathways are key in DOL training grants, but they weren't always, you know, so that comes out of the kind of evidence that's here. But to drill down on one specific example, you can take a look at the FOA or the funding opportunity announcement for strengthening community colleges training grants those that one closed in October. And awards will be announced before too long. But we, we grounded the SCC FOA in the learnings from these national evaluations, including those round four ones that will be published next week, which we had the opportunity to review and incorporate into the FOA even though they're not quite out there yet publicly. I'm fascinated and delighted that so much of what we worked hard to include in SCC is aligned with what you've talked about in in these chapters and what you've been talking about here now. This isn't necessarily related to what Ivy is saying because I just saw that this morning or this afternoon but but the other things that you said so you said include career pathways frameworks. That is the core of SCC. And collaboration as a core element of an initiative help support community colleges capacity building efforts. So we structured CCC SCC around collaboration by requiring that colleges partner with employers and the public workforce system, and by setting an intention that was stated in the FOA to award at least 75% of the funds to those consortium those statewide consortium. So instead, grantees benefited from logic models. Well this time we required that as part of the SCC application, not just the evaluation and I am really excited about where that may take us I can tell you. We also pivoted to focus more on the non participant capacity building and system building performance outcomes, which is a lesson that we learned from tact and we tied them to the logic model so there's a little experiment going on there we'll see how that one works. You had an interesting section in the book about the value of developmental evaluation, and that strongly encouraged as an aspect of our third, the third party evaluations that the legislation required for SCC. You said conduct rigorous evaluations. Now, we did not require that in SCC given that the size of those grants allowed under the SCC legislation is so much, you know is considerably different than under the tact legislation. So, and as you know better than anyone rigorous evaluation can be expensive so tact funding allowed both developmental or implementation evaluation and more rigorous impact evaluation and demonstrated the value of doing both where that's possible. More examples a couple more you said create a formal role for state agencies and we did that with consortium grants because we saw the value of involving a state entity for sustainability purposes. Sustainability focus is key and a plan is required in SCC. As intact a big part of that is, from my perspective, aligning grant funded activities with other campus or community or statewide efforts because that is a key way to ensure that the seed funding from a grant supports and helps build that ecosystem that in turn is helps ensure the sustainability. If you're off doing something in a silo, it's just not likely to last so we will continue to encourage that and other aspects of sustainability in this grant and in all of our other grants. Like I said they apply to to to Community College grants they specifically applied a capacity building grants but they also apply to other do well discretionary grants where they're applicable so I hope that it. I hope that it feels good to know that the things that you have identified and helped us learn are being used. It feels very good show. I, and I do see a lot of alignment between your comments and and some of IV sort of predictions or hopes for the future. I'm going to turn us then to to look at some of those strategies that that were so important to the tech grants and I'm interested Derek and having you comment on strategies that you saw as an evaluator that that you really believe did have a big impact on students and and and others. Thank you Deb. What a great opportunity to be here with all these excellent panelists and with the larger group of folks that have already spoken today. Many references have been made to what a lot of people are experiencing right now, in terms of job loss and what opportunities are going to be available to them as we get out of the COVID pandemic, particularly once the vaccines become more widely available to Americans. This sort of leads me to one of the first things I wanted to comment on, which is how important and increasingly important short term credentials are going to be non degree credentials are going to be especially for adult learners. And something that we learned in our Northeast resiliency consortium grant about how to structure those types of programs as everyone on this panel knows and certainly probably most if not all of the attendees. The short term credentials non degree credentials are widespread and colleges offer them all the time. They're almost always or at least frequently offered in non credit continuing ed workforce development kinds of programs. These are flexible places and you can be very responsive to labor market needs. One of the challenges with that however is it doesn't necessarily lead to a longer educational pathway for adults and others that are participating in those shorter term training programs. One of the things we were able to learn in the Northeast resiliency consortium evaluation is that by packaging a transparent pathway between non credit continuing education programs and credit programs. You can really yield positive outcomes for students and positive benefits to the community colleges as well. One example in our chapter that is forthcoming in the directions that's been mentioned. We document how students that were in these transparent non credit to credit pathways were more likely to actually earn credits that can be applied to credit based degree programs for example which seems sort of cognitively dissonant with the idea of non credit but they literally structured programs for this to happen. I found that if you were enrolled in one of those types of articulated pathways that you were more likely to transition into a credit based program and continue. And this is above and beyond the other positive outcomes which was simply to get employment, which was one of the key outcomes for the tack work. I'm really excited about that evidence and how it might inform future work around short term credentials and ensuring that those short term credentials, even if offered in a non credit type of manner, have clear articulations to credit based programming and community and technical colleges which we think is an essential reform model that the tack work can can contribute to. The second thing that I want to take a minute to speak about was already mentioned in fact by Ivy love in her summary and I just want to underscore it. And, and this is something that we saw really demonstrated across two of the three consortium grants, because these are the two that we looked at that. And, and well as many other evaluators on the on the panel today and that is the important, maybe even essential role of comprehensive support services for students in these types of programs, particularly our most recent round for evaluation in Wisconsin the act for healthcare work which, as you've already mentioned I think earlier Brandon Roberts I think here as an attendee was a participant on we partnered with equal measure and the 16 Wisconsin technical colleges. And we're able to learn that if, if you provide comprehensive support services to students across many different programmatic areas right this was the act work in Wisconsin was about healthcare but, but we have evidence and other sectors as well. So in terms of training, it really can make a big difference on program completion and eventual employment for students in there in those types of pathways and those types of programming. And, and when I use the term comprehensive student supports and I hope people attending will also hear things like holistic student supports or wrap around services I want people to understand what we're talking about here these are not just academic or basic skills that were widespread intact, although those were part of it and very important. These were also coaching career coaching types of supports to help people understand the employment opportunities and the pathways to jobs and mobility that were available to them, based on certain training programs. And thirdly these types of supports also address what I think is increasingly known as students basic needs, things like childcare and transportation healthcare, things to really help them make it through tough times. These are in critical lessons as we emerge out of a global pandemic and need to get people back to training and school and back to work in an important way and I hope the panelists and the attendees here will look into the chapter and other resources on how to understand how non credit to credit pathways that are clearly articulated and how comprehensive support services are both essential practices that can yield opportunities for Americans looking for better training, upskilling and a new job opportunities. So thank you Deb for including us and I'm glad they had a few minutes to share the findings of our evaluation work. Thank you, Derek and I wonder our funder for this project is the Lumina Foundation as you know I know that Lumina funded the earlier work Derek that meta analysis work that you did. If there's any chance you could put in any of those links in the chat. So folks can find a couple of those reports that would be super helpful to do that that they really demonstrate the value of short term credentials of support services for employment why not I'll get those in the chat. Thank you Deb. That would be great. I'd like to turn to share a gale to have her share a few thoughts about strategies that that she has seen as an evaluator that seem particularly effective. Absolutely. Thank you so much. So one of the major lessons learned was that programs need to center a paradigm of equity, which forces the accessibility of opportunity and realization of impact for all. So I'll address this issue first from the perspective of strategy as implemented by the National STEM Consortium, and then offer considerations for accountability for community colleges system wide. So the National STEM Consortium Consortium, which I'll refer to as NSC they developed a unique curriculum called stem bridge. This was a model of contextualized then workplace learning. STEM bridge was designed to ensure that students and technical stem programs had baseline academic knowledge. So let's say mathematical understanding, and that they were exposed to workforce skills training, which would include things like soft skills in a form of communication. And so this innovative educational strategy sought to improve completion and success through contextualizing academic reinforcements and providing supports for student development of foundational skills. This type of application was pertinent for learners who might not have otherwise exhibited their readiness to enroll into and persist within STEM certificate programs. Reflecting on some of the student outcomes that we've observed contextualized them learning was better able to support learners make connections between these often abstract stem concepts, and their relevancy relevancy to an application within industry context. And just in case everyone's not familiar with, with stem that science technology engineering and mathematics my apologies for that. Unfortunately, without a doubt, we all recognize that community colleges recognize the importance of soft skill inclusion within programming to position students for future career success. But we must go a step further and as evidenced by the NSC stem bridge motto, if we are going to progress toward transforming the community college landscape, and particularly within stem, it's imperative that there's increased focus on integrating these professional technical program curriculum, and not in manners wherein the supports are designed external components thought to be separate from core programming. So within the NSC design of such curriculum was made possible through educator industry partnerships where interaction cycles and feedback loops proved to be pertinent to the design of high quality contextualized them workplace instruction, and thereby students academic and professional success within stem. Similarly, we identified that there is not that it's not enough for programs to generically serve students is developmental needs, particularly for those looking to quickly into the workforce, or earn short term credentials curriculum instead needs to be heavily contextualized and informed by authentic industry events to effectively influence student academic and career readiness. So moving toward more of a policy consideration and program and implication, and my chapter and as that shared earlier, I include a call to action for community colleges, and I also would like to extend this charge to all of us here today. Specifically, as Deb shared earlier, it's imperative for community colleges to continually hold themselves accountable to and responsible for advancing sustaining equitable educational structures. This priority becomes even more crucial when we consider the numerous communities off that community college systems are intended to serve who overwhelmingly encompass populations left on the margins. This entails removing structural barriers that might prohibit access to higher education, and so examine the biases and limitations inherent to administer admission requirements for instance. It also means embedding supports that allow for equitable participation. This could look like diversifying the ways candidates can demonstrate their readiness to participate in higher education. It could also mean offering developmental coursework if learners don't initially meet eligibility requirements and implementing diverse instructional models which might include accelerated learning models or virtual formats which have become the norm in today's society. Embedded structure supports a for a more diverse candidate pool to leverage educational and workforce opportunity within STEM. These fields are high skill high demand and high paying the state position students to experience a better quality of life, where attainment of social and economic prosperity becomes feasible. Related we understand the significance of community colleges recruiting populations presently and historically underrepresented in STEM to enroll in certificate and degree programs, but it cannot stop there. We must also regularly and intentionally does examine our programs with consideration of student demographics to assure that outcomes are equitably realized. If it's not enough to merely admit our students into these programs, educational and workforce outcomes must also be equitably equitably achieved. Thank you to share. Thank you for reminding us of the moral purpose that Michael Fallin talked about. I'd like to move to John to ask John to share a talk quite a bit about the importance of the employer relationship and your chapter is about that. Can you, I'm mindful of the time to just to give folks we only have about 15 minutes left. Can you share with us a little bit of your observations of the importance of those employer partnerships and how they help support change. And again, it's a privilege to be here, and I want to thank the folks you know who have put this all together. The employer engagement piece I found to be interesting, because you know the majority of the colleges who were part of our tech experience. And then at the beginning of the project and said, you know I need you to do some additional kind of employer engagement, almost always it was about, Oh, program advisory committees. And, and so there was a little bit of truth, you know to that, you know, but I think an important piece is that the attack funding, because of all of the strategies. I think it really gave a catalyst to the colleges to be able to try things that they never really been able to do to do. And, and so, you know, it kind of ties to the employment engagements piece, because as a couple of colleges, you know actually in a very kind of straightforward way. You know, our employer engagement, it got better, because our response was better. You know, colleges had more tools in their bags so to speak, to be able to go to an employer and say, Hey, I don't want you to just come to a meeting. Sometimes a year and have some kind of rubber chicken meal. I want you to be engaged across an entire kind of spectrum of activities from, you know, I want you to recruit students that I want you to get engaged with this on the design of the content and on the design of the curriculum. I want you to do things in the program like come to our classrooms do that kind of institutional kind of support. And I want you to hire our students, and then I want you to give that kind of feedback, you know, to the college and the students about how's it going. So, because the colleges had more tools, the colleges were able to be, I think, you know, like better partners, or at least, you know, they were more responsive to the employers. So the employers then were much more, you know, kind of, the more likely they'll be to say, Hey, I'll engage in the full kind of spectrum of employer engagement. And as, and as colleges and employers kind of continued that as a dance, they got more comfortable more confident in each other. And for the colleges who took it from a grant activity to a relationship kind of building activity. You know, it's been a godsend for and employers also. Yeah, I think john you just gave us a very good example of what happens when when we're learning together. You know, all those parties were learning how to how to work together how to partner. Thank you for that. Jennifer, I'm going to turn to you and ask you to share some observations from the big picture that you have the tax grants, working with ETA and technical assistance and then we're going to round out the panel with Kate and we've had several questions Kate about what is developmental evaluation so but we're going to start with Jennifer. Thanks Deborah. So, much of, I'm going to sort of draw on some of the themes that other people have raised because it these are, everyone has mentioned great examples of what I view as one of the most significant impacts of tact, and sometimes often under recognized impacts of how we show up in evaluations. And that is the systemic changes to to the way the community, the way community colleges did business. And, and sometimes at the Community College level sometimes at the state level in the case of state consortia. A lot of people mentioned a lot of great innovations like john just mentioned changes in how colleges engage with employers. Other people talked about recruiting student recruiting adults working more with the workforce system so that unemployed adults could be recruited into programs. The students supports navigators and whatnot that were talked about. There were colleges who did those things for the program particular programs that were funded by tact, and everyone on the panel and many in the audience realize that that's kind of how tact was structured at funded particular programs, the building or redesign of particular programs. So there were program people who did all did many of what I just said, for those programs, but other colleges really took what they took what they did and applied it to the college overall. And that's where there was dramatic impact. So that's where, like the whole college might have learned from tact and really changed how they did that kind of employer outreach, or the whole college has now working has redesigned multiple things to break down the structural barriers that Tashira just talked about, not structural barriers, only for the people there are attempting to enroll in particular programs, but for all adults in many many programs all programs across the college. And so anyway my our chapter, we wrote chapter number two that goes into some of these examples but I think that those are some of the most exciting and broad impact, larger impacts of tact and then you and I'll just mention one other thing I had a little email chat about these issues. And, and I want to, you know, kind of certainly point out that what led to that these kind of broader systemic changes are overarching long term changes to policies and practices, really had to do leadership really had to do with leaders at these colleges are at the state level saying, Okay, we have this goal to do whatever it is, align with industry more effectively, build data systems to track our outcomes. We have this goal, we're going to, we are going to use tech dollars to build our capacity to do that stuff. So sometimes leaders really came in at the grant writing phase. And then other times leaders like stuck in there and, and really ensured that the lessons that the, that the products and the processes developed through these programs were then applied to the whole college. So it was really kind of leadership leaders really focus on focusing on transformative change that made the difference and certainly you know as you mentioned not with all colleges, but with many. So I think an important point there to is, we're talking about leadership as a process and a function, not necessarily someone's position, although that matters to leadership is is part of what faculty, it's part of what the navigators did it was part of, you know what the employers did so that you leadership is so critical. Absolutely right and those leaders, not just positional as you say but leaders are those leaders needed to have a strategy around how to, how to make permanent change in these practices and policies. That's a great point thank you Jennifer. So Kate I'm going to turn to you tell us what developmental evaluation is our five six minutes here tell us a little bit about your experience. Sure. So thank you, Deb and boy, developmental evaluation I'd say is probably one of my favorite approach to evaluation, even though I work for a company that does a lot of impact evaluations using sort of fancy quasi experimental designs and random control trials. I like it. And I'll talk about what it means partly because it involves the evaluator working really closely with the folks here evaluate really a partnership. So, a quick answer and I hopefully Michael Quinn pattern if he's, he's on the call probably not but he might, you know, criticize me for doing this too quickly. He pioneered this approach. It's a lot like a formative evaluation approach, except that it's even, you know you because you're working with the folks who are implementing the program that you're evaluating very closely but even more intensive involvement. And essentially is what Patton would say is you know you're really the evaluators embedded in the implementation team for the intervention, and you're contributing, you know, evaluative thinking as he calls it which is kind of data infused, thinking evaluators we tend to be in love with data, and hopefully not just for data sake, but we can really speak to that, you know, to the team who may be thinking about other kind of topics so developmental evaluation, he came up with it was something that works very well when you're innovating, because there what you're looking at is you're not able to know maybe in advance exactly the outcomes that you're going to get to or even exactly where you're going. So just sort of figuring that out as you go along, but the evaluator can provide that real time data, and that kind of data driven approach to the team to kind of help the folks move forward with that effort. And this brings to mind actually, I was going to talk about some key things about evaluation I think folks should keep in mind, you know, especially the feds when they're thinking about future investments. And part of that is ownership. And this I think gets to the heart of developmental, which is that you know at SPR we often do a lot of evaluations for the feds. But we work for the federal government and our key role is often to tell the feds how did this work. Now, developmental and tact and this is the wisdom of the Department of Labor when they when they came up with tech change that in that evaluators work directly for the colleges. And that made it much clear that although our role was to report on how successful these efforts had been that we were also working with the colleges the colleges in some sense, own the evaluation. And that allowed us, you know, I think to build the trust so that the evaluations could really work inside with the colleges, and they didn't just see us and some of them did we we work with six colleges and some of them saw the evaluation as check the box. We got to do this for DOL, and I'm not really that interested, but others, after we were able to, you know, and there was enough budget and so forth and they invited us in we set in on meetings of the colleges discussing things. And we built up this trust to the point where I can remember once we were sitting with one of my consortia, and they were talking about a real challenge with the kind of simulation technology. And all of a sudden there was a moment of silence as they realized that I the evaluator was in the room, and it was hearing that they were struggling. And they're like, Are you going to tell on us to the Department of Labor. And I said, Wait a minute, wait a minute. Yes, of course, we're going to report on this department for the use of the funds, but I'm also here to help you learn.  And that was a real turning point because then they started you know again. If you're not going to be honest with your evaluator and share things, you know, we don't have the data to inform the discussion. And that was just a big approach and I think it gets to in this edition that we're putting out some of the changes to the colleges to be a truly, you know, and not just institutional, you know, research parts of the colleges or the highest leadership, but folks running programs to really believe in data, and not see data as a four letter word but rather something that which often a very bad word, but something that can help them do better. And that's a key sort of, you know, that really gets a culture change and a long term change. Thank you. Stop there because I know we're running out of time, Jeff. Well, that was really wonderful. Okay, you made it sound much easier than it actually is. The partnership is really key. I want to give you a chance Cheryl to, if you have any links or advice for when we can expect more resources coming out that would be great. I'm so glad you asked. There are like a dozen pieces of the national evaluation out there already I've lost track of them all there are so many it's like everything with tact it's big right, but I am really delighted to announce that the last six pieces of the round four national evaluation are going to be published by next week, and I'll put a link into the chat, Janet Javar already put one in there but I'll put another one in there so you're going to see topical information on rural colleges. Sorry that's not going to go away. The local economic context. The, sorry, what does that happen when you're online right. Work based learning, you're going to see an outcomes assessment for a subset of grantees and participants and more learnings about capacity building and systems change it was really powerful. We were able to fold it into SCC so I'll put a link in the chat about that but we're really excited about those as well. More evaluation more always more right more more more. Yeah, well thank you thanks to everybody who made this webinar possible I really really want to thank the panel I want to thank all those speakers. I think New America and, and especially Pam Eddie for joining us and giving us this opportunity to bring all of this wisdom together. Michelle van noi and Heather McKay have just been incredible leaders to to pull these manuscripts together into a document that we believe will be really helpful. This webinar will be recorded. As you heard, if you're on at the beginning so you will be able to download this as more resources become available we will add those links to the new America website and web page for for tax so thanks everybody for joining us today thank you audience we really appreciate your time and attention. And have a wonderful day.