So, welcome everybody, we're going to get started here and I wanted to just quickly introduce myself and thank you for coming. My name is Lisa Guernsey, I'm the Director of the Early Education Initiative here at the New America Foundation. And we are thrilled to have you all here this morning. It really couldn't be a better moment to have this conversation and it's great to see that there's going to be a lot of interest in this. We have it live streamed on the internet and we also have, if any of you are on Twitter, we have a hashtag which is WatchTeach for those who are following along online. I know that we have many people who are watching online right now. So, today what we're going to do is we will start with the presentation and I'm going to take you through the findings of our research policy paper and invite my co-writer, Susan Oxhorn, to come to the podium as well and she'll be doing some of the presenting. And then we're going to shift to a panel discussion. We're going to be talking about the recommendations in the paper but we're also going to be really broadening the conversation. Elena Silva from Ed Sector, I'm so thrilled that she's here to help with this. She will be kind of facilitating a conversation with the teachers that have taken some time out from teaching to be with us today. So I'm really looking forward to introducing you to everyone who will be on the panel and I'll do that when we get to the panel part of the presentation and the event today. So as I don't have to tell most of you, teaching has become a red-hot topic. Just two days ago, in his State of the Union address, President Obama singled out teachers, pressing for more focus on ensuring that they're rewarded and recognized for their contributions. This will be the year that a large number of states revamp their teacher evaluation systems. That's spurred in part by race to the top but it's also because of the growing recognition that the status quo isn't working. And in pre-K settings, such as Head Start, some fairly radical reforms are pushing providers to improve the way that adults interact with children. We've got plenty of evidence to show a need for change. Only 71% of high school students graduate on time. Only one third of fourth graders are reading proficiently. There are pervasive achievement gaps in cities and towns across this country related to income and ethnicity. And when you stack our students' performance against those of other developed countries, we don't exactly stand out. Teaching has a lot to do with this. We've got only 7% of students in elementary school receiving consistently good experiences in their classrooms when you look across the grade's first, third, and fifth grade according to nationwide research. So finally, the conversation is focused on teaching. As I said, states are revamping teacher evaluation systems. Everyone is grappling with what students' test score data tell us about a teacher, the value-added debates and conversation. Experts are calling for overhaul of teacher preparation programs. And we here at New America have been working on that as well. Our early education initiative put out a paper earlier this year, sorry, late, it was in 2011, I guess I should say, several months back, called Getting in Sync. And it was about really overhauling teacher preparation for teachers in the pre-K through third grades. So I urge you to take a look at that. And the Office of Head Start has made funding contingent on the quality of teaching at the program level. And we'll talk a little bit more about that today. And yet, having this conversation, and maybe with the exception of Head Start, objective measurements of teaching and classroom quality are rarely part of larger discussions of public education or teacher effectiveness. This is one of our kind of key points that we're making in the paper. And I hope you all got a copy of it. If you didn't, there are several copies that are on the tables outside. And this is the paper that I'll be taking you through this morning. We, in this paper, are focused on the early years and the early grades. And it's important for us to be focusing on those because there are some real challenges when it comes to looking at teacher effectiveness in these grades. We have a dearth of standardized test data. We have questions about the reliability of one-time snapshots of children's responses to questions or performance in one-on-one games, teacher and student games, etc. There's difficulty in assessing the social development of children. And yet at these young ages, that social development is so important and so connected to their ability to learn cognitive, and do some cognitive tasks as well. And we have inequitable access to pre-K and full day kindergarten. So this paper really is trying to kind of take us through what steps need to be taken to change the equation. And look at the early grades, but also put in the context of pre-K 12 education more generally. We are looking at identifying effective teaching and how to identify what is good teaching, promoting effective teaching, and then finally rewarding effective teaching. And the observation tools that I want to talk about today are really getting at this first piece that in many cases has not been well defined in a lot of other debates is how do we really identify effective teaching. So to wet our whistle and really just think about what good teaching looks like, I'm gonna take you inside some classrooms. We have some video clips that I'm gonna be showing you. And the first one is inside an early learning setting for infants and toddlers. And I'm just gonna run this for a couple of minutes just to give you a flavor of what effective teaching, I mean we don't call it teaching with infants and toddlers but essentially it's adults interacting with children in a way that helps them learn. And I'm gonna show you this clip and this comes from early messages. It's a video program used by the program for infant toddler care in California. So we'll run the clip. Knowing how to modify our communication comes naturally if we attend to the child's cues. Well, you're ready to wake up. When infants look away, become fussy, or don't show any interest in us, we adjust our messages. Here are the various ways to make language understandable for infants. Now this one, this one. Do you have a baby you can put to sleep? Do you have a baby? Do you have a baby you can put to sleep, Casey? Rich baby. Nine, nine. Nine, nine. That one? Does she need a blanket? Yeah, a blanket for the baby. A blanket? Can she cover the baby?  Night, night, baby. Night, night. You ready over there? Okay. Night, night. Night, night. So now let's look at the elementary school years. Here's a clip of what effective interaction looks like in second grade. And this one doesn't have the voiceover or the text underneath it, it's kind of describing what effective interaction is. But it's essentially a clip that comes from the class video library that's showing a teacher taking a moment not to say correct a student who's wrong, but to help the student come to a correct answer on her own. It's from the clip. You can play down the grass. Okay, that's a different gut. So look again. Is there anything that's similar to a different gut? That's how that is. Okay, that's something that's alive, or is that something that's different? Different, okay. She gave us two things that are different. Can you think of one thing that's alive? Do you have anything in common, anything the same? We have a both grade. Okay, what grade are you in? The A grade. Good job. So we have tools actually out there, and many of these are already being used and some of them are still in development. Some of them are only being used for professional development. Some of them are being used for more informal evaluations. And these tools can really help us get at this question of effective teaching, and they need to be part of the conversation. To use these tools, professionals are trained, often over multiple days, in what to look for in a teacher, and how to accurately code minute by minute what is going on in his or her classroom. The observers are tested to determine if they are reliable. Are the numbers that they record the same numbers that trained observers record in similar circumstances? And if they aren't, then those folks are not allowed into the classroom to observe. You essentially want to make sure that the people who are doing the observing are reliable observers. These tools are not about providing checklists with yes or no answers. They're based on scales, say one to seven or one to four, on how well teachers are performing on a variety of tasks, and how well they are responding and interacting with students. This has big implications because you can actually hone in on exactly what a teacher needs to work on. And you can get teachers to talk to each other, across grades even, as really needs to be the case in the pre-K third reforms that we talk about here at the New America Foundation a lot. This allows teachers to have discussions across grades about what effective teaching looks like because they're measured using the same tools and they have the same kind of language about what an effective teacher needs to be doing. Here's a chart that shows how many different age ranges these tools cover from infancy through age 18. And I know this is really pretty hard to see. I just wanted to call it out for you. It's on page six of the report. So six months ago, very few people were talking about this. But several new reports and recorded conversations and webinars have burst onto the scene lately. There's a one pager that we put out that I hope you picked up on your way in that describes many of the conversations that have already been happening around this issue. It includes information about this paper that just came out from the Gates Foundation, Gathering Feedback for Teaching. And I should note that we actually, we don't get any funding from Gates, but this report really does help to bring some more depth to this conversation. And Elena Silva has been moderating discussions online around observation of teachers in school districts as well. Technology can be harnessed to make this process more efficient and cost-effective. And I really think we need to be thinking about the new technology and new tools to really get our conversation past the tired debates about teacher effectiveness. For example, observations conducted in the Gates project came from video recordings that captured a 360 degree view of what was going on in the classroom. And the My Teaching program, which we'll hear more about from one of our panelists, involves the use of online tools. There are video libraries. And I'm giving you a clip of just a screenshot here from one of the video libraries that class puts out that help teachers see what teaching really can look like in a classroom. So let's start talking about a few examples in schools. We're going to start with the youngest years. And I'd like to welcome Susan Oxhorn, Susan, there you are, to the podium to help us understand why this matters so much in the earliest years. I want to thank Susan for being a co-writer and a wonderful partner on this paper. And I welcome you. Thanks, Susan. And I want to thank Lisa. The feelings are mutual. It's been an extraordinary journey. And also thank Lisa for pushing Early Ed into the conversation and the early childhood workforce. As most of you know, the right kind of professional development for those working with our youngest children is the cornerstone of quality improvement. And it's here that the earliest relationships form a sort of a laboratory in which cognitive and social emotional development are kind of joined at the hip. The agreement of child and the engagement of child and adults actually creates the architecture of the brain, which is, to me, an eternally amazing, amazing fact. And it also builds the children's capacity to learn and develop. So in order to establish this strong foundation for school readiness, for better academic outcomes in the long term and in the short term, it's critically important that teachers and caregivers working with youngest children be active participants in this loop, this feedback loop of observation, of coaching, mentoring, and then observation. And this is a feedback that increasingly they are getting into. We're going to look in the next video. Oh, yeah, sorry, I can do that. It's just the, yeah. Here we go. In the next video, Amy Dombro, co-author of Powerful Interactions talks about the promise of coaching. The minute that you pause to observe a teacher, you're kind of quiet down inside and you really are paying attention to what she's doing. It's over time you start getting a sense of what a teacher is doing. I can remember when I was a teacher at the family center and people would come and observe me. People that were only there once or twice, they sometimes had something important to share with me, but they really had very little idea of who I was, who the children were, the kinds of decisions that I was making, the challenges maybe that I was facing that day. So I think to be effective observing needs to take place over time. Yes, we're using video in a professional development model we're developing around powerful interactions. Taking short video clips and then discussing them with individual teachers and with groups of teachers. Always careful to use a term that my colleagues Judy Javelin and Charlotte Stetson came up with, use I-notice statements so that it doesn't become a judgmental kind of situation, but really a true observation, staying objective and looking for strengths of what teachers are doing. Also always kind of keeping open the question of what were you thinking and experiencing? What was a child? What do you think a child was thinking and experiencing? And what you might do either the same or differently next time. If we can think about observing as a strategy to help teachers think about their practice and change practice from the inside out, then we would be making a major contribution to the field. Amen. And I think that Amy Dombro really highlights the promise of professional development and improving practice through the use of observation. However, it's one thing to talk about professional development, it's another to talk about formal evaluations. And they make particularly the workforce in the early childhood world rather nervous. While evaluations at the program level in this birth to three realm are becoming more common, observation-based assessments of individual caregivers and teachers are still rare. Across the country in Hawaii, we have an interesting example of an evaluation based on observation of interactions. It's called Tutu and Me, that's the program. It's an early learning program that was created mostly because Hawaii's children were not ready for school. They had a very low school readiness rate. And it was, Tutu means grandparent in Hawaiian. I'm not sure what dialect, but whatever. And as some of you may know, grandparents are the ones who care for the majority of infants who are in family, friend, and neighbor care. And this constitutes a fairly large percentage of young children in the birth to three realm. So while high stakes evaluations are not an integral part right now of the early learning years, the very earliest years, an evaluation of Tutu and Me provides a very nice template for the quality rating and improvement systems that are springing up around the country to assess and improve early learning. The Tutu evaluation used the childcare assessment tool for relatives, CCATR. And you'll see that on page six of the report. It was designed by Bank Street's Institute of the Child Care Continuum. And it measures grandparents and parents' interactions with children, mostly ages three and under. The staff members receive training, not keeping up with myself here. Oh, I am, that's it. I have a little more, sorry. The staff members receive training on the CCATR using three videotape observations. And then they designed a study in collaboration with Tony Porter of Bank Street to measure the changes in the quality of caregiver-child interactions before and after. So the findings bode well for children's development, I think. The scores increase for engagement. And you can watch those brain synapses sparking. And the ratings also increase for what's called back and forth or bi-directional communication between children and caregivers, which means that adults were talking to children and engaged in activities with them and or holding them, which is very important. And children were more engaged in the materials that sparked their development and learning. So now I'm going to put the ball back to Lisa for an example from the elementary school years. So thank you, Susan, very much. So I wanna tell you about First School. It's one of several examples we give in the report. We also looked at some changes that were happening at the Boston Public Schools. We looked at Chicago at the New Schools Project. There are also several pre-K third demonstration sites in Hawaii and Honolulu. But right now I just wanna take you to First School, which is a school-wide professional development program that was developed and is based at FPG in North Carolina. And what they do with this program is come to schools and from all in all pre-K classrooms and all kindergarten classrooms and all first, second, third grade classrooms, they observe what the teachers are doing and in some cases they're looking at the entire day. They're mapping out how time is spent by teachers throughout an entire day. And then they take that data to the teachers and to the principals and talk about what that data means and how they should change what they're doing. They have been using both the class tool and First School snapshot to do this. So I'm gonna run a clip here from a teacher who I talked to over Skype. And so that explains the kind of low quality of the video but I think you'll find it's very high quality of content that this teacher is able to tell us. We'll go ahead and run this clip.  And then they don't, I've been in education for a long time and usually a lot of people come in telling you what you did wrong. And these, they took that data and they let you look at it from a different window or a different viewpoint. They showed you what was actually happening in your room and celebrated lots of great stuff but then just sort of ask you questions so that you as a professional could make the decisions of okay, if I know that small group instruction is a great thing for student learning but yet when I look at my data I see a huge part of my day is spent in whole group. What could I do as a teacher to make changes in what I do so that my children are spending more time not listening to me but having those conversations with each other so that they're reflecting on their learning using their vocabulary and just growing. So I think what was great to me is that it was a true picture of what I do. And I think part of what's great about education is that you have to make those intentional choices once you know better, if you do better as one of my favorite quotes. When you have that data looking in the face you've got to use it to grow. And then also we spent our time looking at, you also got to see your individual data personally and they gave you the opportunity to look at your data and then they just gave us a sheet that said what are the things that the data calls to your attention? What does it make you feel for you? And when you're, for me, I found out that I was a bit of a control freak. My children spent a great deal of time listening to me and I know philosophically that that was not the best for children. So I had to go, hey, why am I, if I know this is what's best? You know, you're doing great things during that whole time of years but at the same time I wasn't letting go so my children could blossom and do the things that they needed to do and speak their own words and have their own conversations. So for me I took that as, okay, I've got to change what my classroom looks like and what was comfortable for me. Get out of my comfort zone and get into the years and how they were gonna grow the best. I think in the past our evaluation has just, well, I'll just tell you, I haven't had a single evaluation where they have the section in there that says strengths and then the areas for growth and I haven't had one since probably early on in my career where I had anything that was areas for growth and that's a compliment but at the same time inside you're going, that's not right because we always have areas for growth. And so I think this type of data gives you a huge amount of information for areas for growth. I think it should be a part of our evaluation. I think it's a solid picture of who you are. So at the very end of Andy's comments you heard her mention and describe the use of observation and performance reviews and that's an area that still needs a lot of discussion and research but it really is bringing us to the burning question of high stakes evaluation. So here's where we are so far. We already are using observation at the program level in birth to five settings. So through the QRIS, the Quality Rating Improvement Systems that are being set up in myriad states now and that are also being fostered by the Race to the Top Early Learning Challenge, we're finding more and more programs that are kind of going under the observation watch so to speak and are then using those results to try to figure out what they should be doing differently and how to improve. Certainly the new rules in Head Start bring this issue to a crescendo for many of the teachers in Head Start because now under these new re-competition rules that were released in December, Head Start providers will have their grants contingent on how they score in their class observations and how their teachers are really doing in terms of their interaction with students. We have standardized observation results being used in some school districts and the District of Columbia Public Schools is a great example of that. So DCPS has been leading the way with its impact system and I think Cheryl Olson may be here. I don't know if there she is, yes. So thank you Cheryl for coming. She's already been speaking. So Master Educator with DCPS who understands the early childhood piece of impact and we may have some questions around that later today in our discussion. But what about at the state level? What, if anything, are people talking about when it comes to use of observation in state evaluation systems? And are they really thinking about it in the early grades? Are they really thinking about kind of the pre-K-12 system that is now really kind of becoming the norm in so many school districts? Not saying that observation tools are the silver bullet here by any stretch in the imagination. So I wanna just pause with the caution that using these we found in a lot of our research that making sure that these kinds of tools are used smartly is critically important and that they are validated. We wanna make sure that the tools that are used are able to answer these questions, right? So do students do worse when they're taught by teachers with low observation scores and are students doing better when they're taught by teachers with high observation scores? Kind of seems like an obvious question to be asking, but in many cases we're finding that that may not be asked and that needs to be right there front and center on the conversation. We have some of that from the class scores already. There's more research to be done. So I'm just gonna quickly take you through our recommendations and then we'll move on to the panel here and I think that this is what, hopefully we'll stir up some more conversation discussion. We found that it's really important to be using the results from valid and reliable observation tools in identification of effective teachers. Tools should be aligned across all years and grade levels. So we talk a lot here about the pre-K third spectrum, there are other areas of transition so that the teachers speak the same language of effective teaching. Professional development and high stakes evaluation should go hand in hand so that teachers' trainings on the one side, right, and their performance evaluations are based on shared definitions of what good teaching looks like. Educators and policy makers should be trained on the significance of these tools. We have a lot of work to do to raise awareness about what these tools are, what they can do, what validation means, what reliability means. Research and development of new and current observation tools should continue. We need more research. We need to be looking at them in the context of English language learners. We need to be looking at this in the context of special education, in the context of multiple different kind of subject areas that are being taught. So that's it for our presentation. Thank you for listening today and I am now going to have shift us into our panel discussion and I wanna welcome our panelists to the table now and take your seats and I'll introduce you and kind of talk about who these wonderful folks are as you guys are coming up, so welcome. So we are really, really fortunate today to have Elena Silva with us. As many of you may know, Elena is a senior policy analyst at education sector and has been writing very eloquently about teacher quality for several years and has, I've seen her moderating panels where we're really gonna get at the nut of questions because of the way she's able to kind of draw out what the most important kind of aspects of the conversation should be focused on. So thank you Elena for being here and we'll soon kind of shift over and Elena will kind of take over the reins and she'll be facilitating a lot of the panel conversation that we have here. We also are very fortunate to have two teachers with us today and this means a lot because these are two folks who would normally be in a classroom in front of students right now but we're very fortunate to find schools where principals and directors were quite excited about having their teachers be able to talk about their experience and have given them the ability to be here. Danielle Wilson is from Hampton, Virginia and I thank her for driving up I-95. I'm from that area so I know what that battle can mean from driving up I-95 to be with us today and she is a high school teacher who teaches 11th grade English and Noreen Allen-Hompson is from the Early Learning Institute for Appletree, Appletree's Early Learning Institute which is a charter preschool in the District of Columbia and has multiple campuses and we're thrilled that you could take time I know that your kids would be involved in a million different projects right now and I'm glad that you were able to kind of take some time away I'm sure that they kind of wish that you were there but we could kind of take you away for a couple of hours today. So I want to start by getting into kind of the broadening of the conversation and understanding the full point of pre-K-12 spectrum that we're talking about here so that means I really want to kind of shift our gears out of early ed for a moment and ask Danielle, our high school teacher, to talk a little bit about what observation has meant to her and so Danielle, can you first just tell us about the program that you're involved in and how it's set up? Okay, I am participating in my teaching partner's secondary program from UVA. They approached me about a year and a half ago and asked if I would be interested in participating in this research study and they told me all these wonderful things and then they said and we're going to give you a video camera and every two weeks you're going to videotape your class and send it to us and then we'll talk about it. I was very put off by that. I don't mind people coming into my classroom and when you had videotaped evidence of it it's a little unnerving but I had just taken on the 11th grade curriculum and it's an SOL test a year so I was very nervous so I thought any help would be great so I signed on for it and videotaped my class the first time in October of last year and I was really nervous when I put that little memory card in the envelope and melded it off and then a couple of weeks later my consultant she contacted me she said okay I've pulled some great video clips they're up online I need you to go watch I'm gonna respond to the questions and then we'll talk about it and we do telephone conferences and then some of the other teachers who participate in the program do Skype. I prefer talking to Sharon on the phone and I was really blown away with that first consultation because she had pulled three clips that were about a minute, a minute and a half each and she had talked about the various parts of the class rubric that they talk, that they hit on and we got to really hash it out for an hour, an hour and a half that first consultation about everything that I was doing right and things that I could do to make the class go better and work better because I was really nervous the class I had chosen to videotape wasn't my best class which is why I chose that class because I needed all the help I could get and I was watching the video clip and I'm seeing it from my point of view going oh this lesson's going horribly this kid is doing this over here and this kid is doing this and Sharon helped me see that while they may not be doing what I would have thought would be perfect for that class they were still engaged in the lesson they weren't daydreaming they weren't putting their heads down and going to sleep if I asked a question of a kid who I thought wasn't paying attention they were on point the whole time so it really helped me kind of step back and see okay just because they're not all sitting there perfectly still and staring at me with these bright expressions doesn't mean they're not paying attention and getting the lesson. Excellent and so you're in the second year of this My Teaching Partner Secondary project is that right? And as I understand it there are some teachers that are part of the program but are not getting the same kind of coaching that you're getting. We have some teachers who videotape every couple of weeks and they send their videotapes off but there's no consult they just tape the class and send it off and then UVA uses that for their program and then there's those of us who they videotape and then we have the consultant and we get a lot of that hands on and my consultant is great anytime I need anything and sometimes she just tells me how that class was bad. She's honest. Thank you Danielle so Noreen tell us a little bit about how observations working at Appletree and I should pause to give a little bit of context for those who don't know Appletree was a winner of one of the I3 Investing in Innovation grants and through that has been developing a tool that they'd already kind of started crafting from what I understand called Quality Indicators which is an observation tool that's looking at preschool and pre-kindergarten teaching correct but you have also been observed under some of the class tool and another tool is the Elko which we describe in our report which is looking at an English language acquisition and learning concepts as well so tell me a little bit about what the observation has meant to you at Appletree? Well I am most familiar I think we started out my first year when I was at DC Prep using class in Elko that would make it three years since I've been using class in Elko I have had more experience with class in Elko and then which is my first year here my second year using class in Elko we started using Quality Indicators and what that meant for me was more observations I had my coach coming in she observes me with a Quality Indicator three times per year formally and she referred to them on a continuous basis based on what my needs are to be honest with you I never liked the idea of being watched especially it feels intimidating because you're doing what you do every day and someone comes in and they're at the back and they're typing and you're like what on earth am I saying no is it good or is it bad? But how it has really helped me is that when I ask how was the lesson which I mean I've been teaching for 10 years I usually get the lesson was good and I'm like but or how or what does that mean or what could be improved because I am eager about knowing how something could be improved how could it be better what next could I do how could I bump up that learning experience for my kids and she will look at me and say okay based on the different indicators on the Quality Indicator you did well on let's say student engagement but your concept to your development could have been better for last year because when in preparing for this discussion I thought back on my experience especially last year when I had great coaching my coach has been so great she comes in and she watched me again and she came back to our coaching or team meeting or our coaching session and she said to me the lesson was great but I think that you could have bumped up what was it supporting and checking understanding and of course it wasn't something that I had thought about before but it made me put into context what am I doing how could I have checked for understanding and that brought up other issues that I hadn't thought about before even though I've had great lessons if you get what I mean so it leads to that level of specificity that I like there are areas that I'm very strong in I'd be very strong in for example pro-social behavior but she comes and she says to me okay how could you check for understanding how about you use a graphic organizer what is a graphic organizer and she sits with me she will even come in and model for example I'm not too comfortable I'm not too sure how to use it or how to implement it she will come and she'll model for me in the next week and show me how I could effectively use a graphic organizer for a lesson and that has helped me so well and so do all teachers in Apple Tree get that kind of coaching? Absolutely and is your principal involved in what way or is that more of a performance evaluation that she's doing or he? My principal comes in but especially with last year because last year looking more at last year because we have a new system now but last year I had my principal come in once a month I definitely saw my coach more my coach comes in this year two days per week she spends about two hours in my classroom and we have a coaching meeting and of course if she is not telling me how I could improve I'd be like so how was the lesson and what next could I do? So I feel really confident about that continuous discussion and this year we have created goals for the year and conceptual understanding is my target for this year and throughout the year she will check that even though for a particular lesson I might maybe pro-social behavior wasn't good she'll talk about that but she'll always link it back to my goal for the year so do you think that you addressed conceptual understanding well in this lesson? What else could you have done? And that gives me that kind of long-term support in those discrete areas that you might think you're okay and you do good lessons but this particular area could maximize the whole outcome of students' learning experience on any given situation. Well let me turn it over to you then Alaynaz I'd love to hear just certainly your comments on the paper and some of the recommendations there but also dive in in terms of questions to the teachers as well as us on the panel as well. Thank you so I have a million things to say and I had a hard time even organizing my thoughts coming here today because this is an area that doesn't get enough attention not just early childhood but also the link between early childhood and the rest of the educational spectrum. So thank you I appreciate this. The first thing I want to say just as an observation to get me into some of the comments is that class which I think you both were talking about right and of course it's referenced in the paper and most of us would be familiar with as a tool it started as early childhood right so the fact that we're seeing it move now into K-12 is notable it's interesting but also is very, very significant because I think it tells us a little bit about the story or it can help us understand the story of where we are in K-12 and what K-12 might learn from early childhood education. That's not normally a power relationship you see where K-12 is looking to early childhood but certainly in this case and probably in others it's an important one. So a quick tale and then I have a related question for you all about how we got to this point in K-12 teacher evaluation we didn't come to it through observation we recognized a problem and the problem was observation. This is a really important point. What we realized was observation which was the typical traditional way that teachers in K-12 were evaluated wasn't working. It was a perfunctory, principle-based, drive-by checklist and you're done, right? That's a problem. Everybody recognized it. There was general consensus about it. This is some years back. We have to do something about that. What does that do? That makes observation the problem. I don't think anyone meant for that to happen but that's what happened and so the question became well what else could we do and it wasn't let's figure out how to make observation great. There were some conversations around that and there were some efforts to do that, still are. But there was a really big push to what else can we look at? What else can we look for? And we went student test scores. We went growth models and value added and there's a role for those but we went there and we went there strong and I've sat with superintendents who've said yeah, I mean some of the mistakes I made both practically and politically was stepping out first with that value added and saying this is how we're gonna do it. So we're learning from that. We're learning from the K-12 field is learning that in fact, maybe we put observation on the back burner and we put these other things up front and maybe that's not the way to do it. Maybe we should go back and try to figure out what should we do with observation? Maybe observation should be a part of this. I think everybody recognizes that there are or believes in the multiple measures in teacher evaluation that it takes, you need to measure teacher effectiveness and using multiple measures but how to use observation as that is still a really big question and K-12 doesn't have it down. Early childhood I think is probably much further ahead with that so the larger point here is that I do think there's a lot that this community can share with the rest of the teacher evaluation systems and I'll say more about that and your point about what states are doing a little bit later but I did want to bring up another point which is the relationship between high stakes and improvement. One of the recommendations that Lisa mentioned was the integration or I'm not sure what the word you used was but the connection between professional development and high stakes evaluation, I'm not entirely sold on that and there's different reasons for that and part of that is the way it's playing out in a lot of these teacher evaluation battles I'd say in districts and states across the country and so I'm gonna ask the question of the teachers here about whether you think if there were high stakes attached, let's assume it was Noreen, you call it a continuous discussion, you like your coach, you work well together, you like your consultant, you work well together, you seem to have a respectful, mutually respectful relationship and a lot of time is being spent on a process that seems to be working well in both cases if we added a component which was and now we're going to pay you or not pay you, I mean give you a bonus or not give you a bonus, attach your pay, attach potentially your security at a time by the way, which we know is very insecure for teachers and for jobs, what would that do, how would that change things? Of course, that's a very touchy topic. But I feel it's really something to think about. I have mentioned that continuous observation that I go through, it's not, I feel more threatened when it's one of those random pictures that is taken like someone drops in once a month or once per term and they get a particular look at which on any given day could be anything, it might be the worst class or it might be the best class and no teacher would want to know that they're gonna be evaluated or their job would be at stake or their pay would be based on this one particular lesson that could be what depending on how the kids feel at that time, but if it's something that is continuous and I feel that if it's continuous and there's support given with it, I would feel more comfortable with it, with the system that is in place at my school now. Personally, I wouldn't have a problem with it because I do my job genuinely. My coach is there, she supports me, she gives me ideas and I do the things that she suggests, I do them to the best of my ability, she will come and support me if she thinks it's not done in a way that she thinks is most effective. So I feel comfortable with that kind of relationship happening and not that one of those randomized situations and that's my thought on it. It makes me really nervous to think about it. I teach 11th grade English, anyone familiar with Virginia? That's the SOL year, especially the last few years we've been hitting them with reading and writing SOLs and we spend weeks in March doing the writing, preparing for it, doing the writing and then we spend almost all of May doing SOL testing in all the other core subject areas. So for someone to come in and say, okay, you've been doing all of this, these observations, you've been meeting with your consultant but we're also gonna look at your students' test scores on this day and this is what we're gonna base your evaluation, that would scare me. Case in point, last year I had this student bright, brilliant but he doesn't wanna do what anyone tells him to do. He's 17 years old, he decided he would rather fail on his terms than succeed on mine. So he walked into the SOL, slept through the first two hours of it, 30 minutes later he turned it in. He had a horribly low score, bright kid. Now when they retested him, like weeks later he was in a better mood that day and so he scored advanced proficient on it. So for me for you to look at that and say, okay, I'm gonna base a part of your bonus or your salary and we're gonna link it to this, that scares me because then it's not in my hands, it's not showing a true measure of what I'm doing because I have 17 year old kids. You're leaving my salary in the hands of 76, 17 year olds. Who are typical adolescents? Who are typical adolescents and on any given day will tell you they hate me or like me depending on what I made them do in class. I mean so what both, if you trusted the system though, if it were a good one and if it were over time and if it had this continuous improvement then it sounds like it would be something you could buy into. When I parallel this with the K-12, not many people have systems that a lot of people trust and so apple tree would be the exception and well class in general I think used well would be the exception. If we had systems like that it might work so it's not to say, I would agree if we had systems like that but I do think that there is a sequence or I guess over time a change that needs to occur where the high stakes starts to rise as the quality of the system improves and right now in K-12 we are still really messy. And if I could just add a clarifying point there because you've really touched on something that I've grappled with when it comes to that connection between professional development and evaluation. A couple of times in our interviews the folks that we were talking to would say oh you gotta make sure though that the person doing the formal evaluation is not the quote coach. You need that person to be, and this happens in QRIS situations a lot right at the early learning level. You, a coach will kind of already come with a bias that they really want their teacher to do well and they wanna see that improvement has happened over time and that's fabulous for the professional development moments and it's highly important. But the times when someone is coming in and hopefully it's not just a snapshot of a bad day that doesn't really tell you anything hopefully it's a couple of times a year or whatever but that person needs to be a little bit more removed from the coaching and professional development piece and really just being there as an objective observer who was trained to be looking for very specific interactions. And so then there's another just kind of more contextual point to make which is that one of the things that we're maybe getting out in that recommendation is that right now we have a situation where teachers have all sorts of professional development about how to improve that is not connected in any way to what they're actually judged on when it comes to their performance. And so I don't think we've got an answer yet as to how to make those connections but the connection has to be there. Susan, did you wanna add something? Yeah, I just wanted to add that I think what all of this highlights as well is really the social, emotional, relational components both of professional development and actually how it's responded to by the systems which you Elena rightly so are saying are really broken and are not ones that people that the workforce trusts. And Pianta who is the creator of the class himself penned a paper that looked at the fact that K through 12 teachers and actually high school teachers don't have really grounding in the developmental sciences. So you're dealing with adolescents who are going through their typical adolescent rebellion and this is not really quantifiable and that it has to be captured in the way in the current systems by the measures that exist is problematic to me and certainly your comments really drive that home. The student piece, I can't help but say this because I'm married to a high school English teacher. But, and I have little kids who are on the other end of the spectrum who are also crazy so I don't feel any worse for the high school teachers and the second grade teachers but the point with this is that part of what we're learning from the research I think Lisa mentioned with the gates has funded. It's actually a very comprehensive look that they're taking at how you would measure the effectiveness of teaching. So it's worth looking at it if you haven't. One of the things we're learning which probably doesn't come as a surprise and feeds back into the multiple measures point is that it's no single measure that will make the difference. So was K 12 right to point out that this perfunctory observation that was principal based at what was wrong? Yes, you don't get rid of it. You improve it and you find the other measures and you put them together and that's the best way that you'll be able to not just measure a teacher's effectiveness but predict which teachers are really going to be effective in the future and we do need to know that information so that we can have a really high quality teacher workforce for all educators from the little ones all the way to through higher ed I'd say. But one of the three measures that are being triangulated with med are in fact observations and they're looking at many, many models including class and value added student achievement or student test scores and student feedback. And so that is surveys that students and it's very well done. I mean the tripod survey has great questions. I would trust it as a survey but surveys are only good as the people that answer them and the intent of the people that answer them. So one question that's come up with some teachers that I've talked to including the one I live with is I really I teach 15 through 17 year olds and half of them hate me, right? I'm not sure if I want them doing that but the findings are that it's an effective means of evaluating teachers. So this is something I don't think lots of people are gonna be open to. It's a big question for the field. It'd be really interesting to try to survey a lot of teachers and find out how they feel about this. It's not something I can imagine quickly or easily ending up in policy but I do think it's something we have to pay attention to that we have observations from probably two sources, an external evaluator but then also to the point that was made in one of the videos, Andy, maybe was there an Andy? The person that knows you. There's something about that too, right? There's something about the principal that's worked with you that's been around assuming we have a good leader and then also this external person who comes in totally clean objective. I don't know you, I don't know anything about you. I'm just looking at your performance. Both of those things are important. We do have the outcomes, test scores and but we do, but test scores are the easiest and by the way, cheapest way right now for us to measure anything. And cost is a big, big issue that we haven't entirely addressed in this conversation that maybe we should. And then the kids, right?  Are you also working well with your kids? Are you able to simultaneously challenge and engage your students? That's not an easy thing to do. Not to mention differentiating instruction on all of this. Just to challenge and engage an audience, even if you know your audience, I'm trying to do it right now. It's hard. And teachers do it every day in front of these classes. So I say that because it's a pretty clear piece thing that we're trying to measure or that we're trying to help teachers be good at but it's not very easy to do. And so we do need a lot of coaches and we do need a lot of attention to it. It is expensive. Everybody that's a professional person in any capacity that works with people should probably have coaches. But the fact is the teachers are the most important, right? And so if they're the most important people in our children's lives professionally then why wouldn't we invest in that? Why wouldn't we do that? Will we, on the policy level? I don't know. Right, and it's sort of a tautology because yes, your assumption is they are the most important people but there is a real lack of, I think there's a real devaluing and a lack of trust in teachers. You know, it's kind of, it's a cycle. I think that's what has to be- And yet the reaction to the policy community will say teachers matter most, right? What do you hear? You go anywhere. First thing they say, teachers matter most. We've established this with research. And so we have general consensus that teachers matter. That's why the spotlight's on teachers. The question is what are we gonna do with that information? And so I think the larger point about you have to rigorously evaluate them to see if they're good and also support them. They do have to go together. They do. But coming up systematically with a way to do that remains a real challenge. Again, I'll say the early childhood world is approaching it in a developmental way that K-12 is not doing. Right. They are not approaching the evaluation of teachers in terms of the whole development of child and whether or not you know anything about child development and whether they're adolescents or not adolescents. That's not really part of most of the evaluation conversations in K-12. And I think that there is a big missing link there. So I wanted to ask you all, because I don't know exactly the populations you work with but the differentiated instruction piece and specifically the note about ELL students and special ed kids. There are big questions in teacher evaluation in K-12 about how we would measure not only teachers that are in untested subjects but also in untested grades but also how you would measure the performance of teachers that are teaching populations like English language learners, like special ed kids. And so I'm just curious to know if you work with those populations or if you have thoughts about how the evaluation might be different for different teachers teaching different populations. So I have personally had, well I've had one but he has grown in America so he uses English language a lot so he didn't, even though he is ELL on paper he had a very good grasp of English language so I haven't had any personally in my classroom but we still use similar systems for them. My experience with ELL is very limited. For one year I was teaching in middle school and we had a lot of students come from Burma and they knew absolutely no English. So my experience has been very limited. We did have ESL support helping with us but I've also taught inclusion classes for seven years. And I think one of the best ways to measure that or to check on the observation is I've had some really great inclusion experiences and I've had some really bad inclusion experiences and I think one of the best ways is you can walk into any classroom where you have co-teachers and tell right away if it works. Because you can tell by the way the teachers are with each other, you can tell by the way the teachers are in the classroom, you can tell by the way the students respond to the teachers. My last inclusion experience was wonderful. It got to the point that I could stop in the middle of a sentence if the phone rang or if someone came to my door and my co-teacher just picked up where I was because we had sat down together, we had planned the lesson, we had taught it before. So she knew my expectations as the regular ed teacher. I understood her expectations as a special ed teacher and for the last semester of that year if there wasn't a distinction, oh, you're the English teacher, you're the sped teacher. We were both the teachers. We both handled discipline. We both handled the different curriculums, the accommodations for the special ed students. It was just kind of rolled into one. Our class was lucky enough to just have two teachers. Right, so part of the reason I asked the question is from the teacher's perspective, but also to make the point that these complexities have to be owned also by the evaluators and the coaches. So you need, this is to the point of how well-trained the people have to be that are coming in and observing this because if you aren't familiar with how this works when you have a co-teacher, who are you evaluating exactly? The co-teacher, the teacher, the teacher's ability to work with the co-teacher? There are a lot of layers in this. And so the processes and the systems that we have set up to train these raters and evaluators, make sure that they really know what they're doing and they know all of the situations that they'll face. They can't just be a rubric. The rubrics can be good, but ultimately it's gonna come down to a lot of human judgment. Even with video observations, there's an incredible amount of human judgment. And even with the objective external person, there's a lot of human judgment because you're watching a human experience on an exchange. And on top of that, all of the complexities that go with teaching different grades, different ages, oftentimes in the same space, and kids that have very different learning needs and learning styles. Go ahead, Sir. No, I'm just, I think that that makes so much sense. And again, it's looking at the human relational element as, you know, which is hard. Yeah, the other- That's stuck out, both from the report, but I think it's also worth bringing up is the front end of this, which is teacher preparation. You had a couple of recommendations in there about how teacher preparation is changing and perhaps how it should change. I agree it is changing. I agree it should change. There's a move, a foot now, teacher performance assessment, where you're not only assessing the performance of a teacher in service, but pre-service. So we're not letting you pass the door, whatever, into the classroom, unless we know that you can not just, that you don't just know math, but you know how to teach math, which is different. And that's an important distinction as well, which is oftentimes, I think, minimized when we're trying to, when we talk about finding the top talent which we need. Do I wanna have a nuclear physicist teaching my seven-year-old? Well, maybe not when he's seven, but teaching physics to my kid? Yeah, bring it on. I want the NASA people. I live in an area where there's all these amazing people. Bring them on. But I also wanna make sure that the person can actually work with children and can actually engage and teach at the developmental level that's appropriate for my child. Both are important. And so keeping a handle on both, I think, is important. That starts a teacher preparation. So I'm gonna ask you guys about your experiences if you felt, not only if you feel prepared now, that's less of the question. It's more about thinking back to your teacher preparation. But with laughing or without laughing. Are there pieces of it that you think were best prepared to also focus on the positive? And what are your thoughts about the idea that you would have to go through? In this case, one example is a five day or a week long practice, exercise, where a series of exercises, where you have to have a portfolio that shows what you do, you have to actually perform, you have to, in front of videos, in front of people, and you have to pass this, the way you would pass the praxis, to get into teaching. I'll just say, when I first walked into the classroom, after the end of my first day of teaching, I went home and I told my husband I don't ever wanna go back and do that again. I felt horribly unprepared for it. Horribly unprepared. I had these wonderful lessons. I had my curriculum. I had spent months, because I was hired early. I spent months getting everything ready and I was so excited. And no, I was horribly unprepared for everything. I had to go in the next day and sit down with the head of my department and just say, okay, this is what happened in my class. This is how I handled it. And this is what didn't work. And so it was a process. And my first year I was exhausted all the time. I slept probably more than I've ever slept in my life because I would work all day and I would put all of my energy into trying to make everything work. And it took me about eight months to learn that, okay, I don't have to spend 12 hours a day or 12 hours a night planning lessons. I just need to get a few key pieces under control. And that would have been really great had someone told me that before. It would have been great if someone had said or if I had been able to see this going into it. Is there a way to teach your preparation? Could have possibly done that for you though. I don't know.  Did you have an advisor, mentor, coach kind of, you know, equivalent? You did not. And probably no induction process. No, I went through the student teaching in a couple of schools because I did it through my master's, which was in special education.  You got to work in the different populations, but they put me in populations that I wasn't teaching. Most of my experiences were in elementary and middle school, but I was going to be teaching high school. So it's just complete different monsters. And that happens a lot in early childhood too, where you said someone who's really been trained to teach like third through fifth graders, but suddenly they're put in a first or kindergarten classroom. So for my experience, it takes me back because I got my teaching, I got my teacher training in Jamaica, which is a whole new different sphere from here. And I remember when I was thinking about, you know, whether I wanted to do this thing, teaching. I had a discussion with one of my church sisters and she mentioned that the difference with early childhood is that you have to learn how to teach. It's not so much the content, but there is a lot of emphasis on the methodology. And it took, my first year was, it was devastating. It was difficult. There were fours and you know, they were crazy. It grew with experience for me. Personally teaching was never something that I'd have a second thought about for some reason. I'm not even too sure why, but I just feel like personally teaching has found me and I have found teaching. It's kind of like we're meant for each other. And which song, you know, yeah, I understand. But coming here in the U.S., I've had my first day here. I mean, I came here with eight years experience and then I was put in a classroom and it was like, okay, do this. And the accent was totally different. It was like, I'm in this world and you're in that world and it was really a steep learning curve for me. But my first day, I had a model, Miss Bryan, called Bryan, and she came and she showed me, you know, how things are done here, how you do things here. And that first day resonated with me so much. Just seeing her, and it wasn't the whole day that she spent. It was just that first section of the day, I think up to lunchtime and just seeing her do things and having that model to go by, you know, seeing her use, her modeling, modeling wasn't something that we used a lot in Jamaica. We used a lot of instructions, direct instructions, and I saw her model how to model, model how to use pre-corrections, model how to explain unexpected behavior, those simple things. I saw them done before my eyes and after a while, with my continued coaching too, which I got more of at Apple Tree, I had more discussion with my profession and my coach. I tried this, this didn't work. Why? And I'm always, you know, that aggressive about it. So how can I possibly get this to happen in my first shot? And it's something I experiment with every day. So, you know, being here, if we're talking about my two years here, my three years here, it has been, I have been really happy to have a coach there with me who can, you know, support me in those areas that I'm weak on. Yeah. So the coaching part is so important. This is resonating with me. And it's reminding me too that the, while that's true and you need that coaching, that help, it's the rubric too, right? I mean, it's the fact that it gives you clear standards and expectations for what we're all trying to do here. And then you can have that kind of continuous conversation in that developmental piece. The rubric too, I will say, having been in discussions with policy makers and various decision makers and districts and states, it's tough. There's great rubrics out there, but you know, it's every long conversation is to figure out exactly what it is. And yet that's really important because you do have to get that right. So you are asking the right questions about what you're looking for and what you're expecting of teachers. Your Jamaica point reminded me, I just wanna make this point, it's slightly tangential, but I'm gonna take the privilege of sitting right here to make it. I was at a seminar, an international seminar. It's called the Salzburg Seminar, and it's wonderful. In Austria, with all of these amazing educators and ministers of education and leaders in all these nations, and it was an amazing experience for lots of reasons. And many things came out about, it was K-12 was the topic, K-12 education and achievement gaps in particular. And one of the pieces that came out that was very clear that everyone agreed on amidst, you know, we're developing and developed nations, a lot of difference, incredible variation, was the importance of early childhood education. And a lot of these folks were from universities, and so it was difficult for them not to talk about higher education because that was their thing, that was where they sat in the world. But an important fact that I wanna put out there is that international, globally, we spend more on tertiary education than we do on early childhood education. Globally, so I mean that is, and in individual nations as well, that is astounding, we could have a big debate about whether or not that makes sense because higher ed's longer or developed, whatever. But the idea that we would spend more on tertiary, higher education than we would on the very first years of the development of children's brains and lives, to me is astounding. And with this community of people there, we were all sort of sharing in this, wow, that's a problem, and that we should invest more in early childhood education. So I put that out there just because I think it's a great point, that's a great gift. And of course, James Heckman has been pushing that with his lovely graph, and those of us in this community know that the investment, that wonderful chart with the, it's really insane, but it's wonderful to have you articulate it so eloquently. That's not to say I don't think you should invest in tertiary education. We need to do all, but we do need to have more investment here. Well, thanks. It's interesting actually that you've been bringing up a lot of points that I wanted to raise too. There are really more questions than anything. When you were talking about special ed, ELLs, when you were kind of describing the co-teaching that happens, it raised the question in my mind about how we evaluate that collaboration between teachers and what do you do in a classroom in which maybe the paraprofessional, particularly in kindergarten classrooms and pre-K, certainly, they really can make a huge impact on how smoothly the day goes, on how well the children are paying attention, on how engaged they are and what's going on, and are we, have we gotten to any place with these observation tools or any other kinds of assessments of really looking at what those other folks in the classroom are doing and what that means to the kids? And I would be really interested in hearing any of your thoughts on that. And then the second piece is this look, going back to the Gates Foundation's Methods of Effective Teaching project and this new paper that's come out that really does look at these three and a multiple measures as Elaine described, right? It's looking at much more closely at observation tools, which is very welcome. It's looking still at an important kind of connection between the value added scores and teachers, I'm sorry, and students improvement. And then it's also looking at student surveys. And I have to say as an early ed person, looking at that, I just started laughing because of course you can't ask a four-year-old, what do you think of your teacher? Because my four-year-olds, when they were four, as long as the teacher gave them chocolate cupcakes for snack, that was a good teacher rocking and rolling, you know? Absolutely no way at that stage in a child's development for them to have any real kind of assessment necessarily of how good their teachers are. And I actually will also say what makes it so hard in early ed is that kids really love their teachers. I mean, for the most part, even in situations where you find a teacher who may not be scoring all that well on a lot of these measures that we're talking about who really cannot be developing, is not doing a good job of developing concepts, it doesn't really have a grasp of what instruction should even look like. I mean, the kids still wanna go and give them a hug and embrace them and feel very happy to be there in most cases. And so that makes this highly challenging and we figure out then what are we really gonna look at to make some differentiation and to identify what good teaching looks like. But I just wanna say that that attachment, that kind of relationship is really the basis for looking at the interactions and using these observation tools. So it's really rich in that way. And yes, I think that if teachers are able to look at that and the relationship is there and it's strong, then they're in a better position to really critique what it is they're doing and understand what is possibly resulting in better outcomes for the children. Yeah, absolutely. The connection has gotta be there from the beginning. I'm just curious if any of you had other thoughts related to this. I'm gonna open it up for questions here after this. Okay, so let's go to some questions. I'm sure there are many out there. And also I wanted to offer Cheryl from DCPS if you have any comments or actually, do you mind if I put you on the spot in fact? And we can maybe bring the microphone over to Cheryl because I was wondering if you might be able to share just a moment, some of your thoughts related to this discussion and how impact is working in DCPS. And that clears behind you with the microphone there. Thank you. A lot of what's been said on the panel and beforehand has really resonated because of these are a lot of the issues that we've been grappling off with over the past several years. This year is the first year that DCPS has a new early childhood rubric, which is based on the same rubric that is used really first grade through 12th grade. Kindergarten is considered a component of early childhood. So we're looking at preschool, pre-K and kindergartners, but the standards that are in the rubric that's used first through 12th are the same ones, but the language is a little different. So I think one of the things that we've achieved in doing this is creating that common language so that we can now have conversations not only with teachers, but also in some cases with principals and school leaders so that we are talking about the same kinds of things. And we have also tried to take the language that's in the other rubric and put it in more young children friendly terms. So it's a lot less teacher directed. And it's a lot, we've really tried to capture the kind of instruction and teaching and learning that we like to see with much younger children. So it's not, it's capturing the teacher-child interactions. It's capturing what's happening when the students are playing. So it's not just that teacher-directive, objective-driven lesson that we're looking at, but what's going on when kids are in centers? What kind of interactions are the teachers having? What are the kids doing even when the teacher's not present? The teacher might be on another part of the room and what's happening with the kids who are in the block area. So it's trying to capture all those types of things that tend to happen in a preschool pre-KK classroom that you may not see in a third, fourth, fifth, or certainly in middle school or high school classroom. We've grappled. Can I ask you a question about that though? But in the later grades, this goes back to the point about the later grade evaluations learning from the earlier ones. I don't know what to do about this, but I'm gonna put it out there because it's a big issue in the public. What about that other, the free time or the play time or whatever it should be called in those later grades? So the idea that the centers that we may be, specials or centers that the littlest kids go to, in the older grades, you still have, let's say before you get to high school where it's not subject specific, but you have let's say a second or third grade teacher that's responsible for the kids all day. They still may have the centers, but my question would be whether or not that rubric should look more like the early childhood one. Yes. Yes. Yeah, I think so. I mean, what we tend to see in the upper grades and I don't observe those classrooms personally, but I think what we tend to see is centers that are objective driven. So they're much more, when you go to this station, I want you to work on rhyming word puzzle. I want you to work on something like that where it is more objective driven.  We were trying to move away from that in the younger grades where the centers were set up in a way that really support the kind of play where kids are learning socially, they're learning in terms of language development, cognitive development, all of those things are happening, but the teacher's not saying when you go to this area, this is exactly what I want you to do. So I guess your question may be more, should there be more of that in those grades? Yes. Probably yes, we don't see that. But that's somewhat of an irony, right? Because the impact was in part designed based on some of these early childhood rubric models. So for K-12, and then was like, oh, we also need the early childhood and then developed an early childhood. It's interesting because impact is a head of the game in terms of being an evaluation system. But it's curious as the nation watches impact to see if its early childhood approach will affect the rest of its approach. I don't know, I'm saying. There's a lot of people who want to respond to this. So let me just go to the back. No, that's great. The woman in the polka dots, I'm sorry. Yeah, there's the microphone. And introduce yourself, please. Holly Blum. Hi, Holly. I think I'm very happy I worked with some DCPS school. I'm happy to see that what we advocated, when they started impact for an early childhood impact, rubric, we had advocated for that at the beginning when impact was being developed. And so we went back and we did that. The disconnect between, as you mentioned, what our expectations now are in preschool and kindergarten in terms of continuity of instruction, I think there's a disconnect. And I think that one of the ways to make that connection is not only quality practices, but I think it's something that you all were talking about before and that is differentiated instruction. So I'm not sure that there's an un... I shouldn't say that there isn't an understanding, but a possibility that one way to differentiate instruction would be at higher grades if you know how to differentiate instruction would be to use centers and stations in a different way other than as another way to do direct instruction. And so to connect the dots here, that sort of thing is that captured in impact. Is there a way, say, second or third grade levels, to capture that differentiation is going on? Maybe Cheryl, if you could just respond briefly and then we'll go on to some other questions here. What we found at the early childhood level was the teachers and in many cases the administrators were thinking very different things when we were talking about centers. And so we felt like I might walk into a classroom and I'm talking about centers, but that's very different with the principal. What the principal means. And so we found that by creating this rubric that is specific to early childhood, we were able to kind of put everything on the same page in terms of what those centers looked like. So it's not that the rubric that's used for older students discounts those things in any way. I don't think, but it's that I think the early childhood rubric has put it more in younger kid-friendly language. We're talking about child-initiated activity, child-directed activity, that kind of thing, learning from materials. And so I think the language looks a little different, but we were trying to sort of create that sense in the common language so that when we walk into a principal free gang, read in kindergarten classroom, when we're saying centers, we're pretty much speaking the same language. Let's take another question in the back here, the purple sweater. Hi, I'm Lauren Hogan. I'm with the National Black Child Development Institute. I was just wondering, one of the things that struck me was a lot of conversation about the need for really excellent coaches. And I was just wondering if you could talk a little bit about where you think those are going to come from as part of the system. If we're gonna rely on that very much, and I think they're clearly an incredibly important part of the work, but this is sort of about leadership development and how we create a pipeline for teachers as a place to go. And I was just wondering about your opinion about how we can make that part of the system, coaches that need to be incredibly well versed in everything from developmentally appropriate practice to culturally appropriate practice, to different grades and different ages and what teaching looks like in so many different places and how we sort of foster that. So I actually would love to hear what you think of this too, but let me just mention that this coaching question I think is really essential. It's also essential to cost questions. And it's also, I think, needs to be a bigger part of the conversation about does teacher preparation or what ed schools are providing. But I think that coaching and building more of a pipeline of people who can do coaching well is absolutely something we're gonna see in the future. There's that New Yorker article from October that a school got one day wrote on coaching. I hope that a lot of you read that. It looked at everything from in the classroom to kind of at the surgeons table and really kind of shine a spotlight on how much we as a society and as people are starting to feel the need much more for kind of coaches who can really help us evaluate ourselves as adult learners. But I'd love to also hear what Elena or anyone else on the panel has to say about coaching. It was just last week or week before last when I sat with my director of education and she made a statement and she said, our coaches were never this great. We made them great. And what it meant was that teachers who are great, teachers who became coaches weren't great coaches but they went through a system. They got support too. They got some kind of training on what great coaching looks like. And I feel that that kind of systematic support even for coaches will help coaches be good coaches for teachers. Yeah, I think it's unlikely that we would have a whole nother industry of coaches, the way you'd go to school and become a teacher coach. I think more likely we, and my vision would be that in education we would see the teacher workforce or the educator work possibilities expand so that when you go into teaching and you become a teacher, you have a lot more options than just teaching the same thing for 20 years or leaving, which is essentially what you have right now. If there were different career pathways and coaching were one of them and there are places where this is the case, then you see more opportunity because you see more people funneling into the system, being teachers, then being coaches, then some go on to be leaders, that's great. It's also great to have teachers go in and then go off and do other things in education. The space of education, the field of education can be a lot more dynamic and richer than it is right now. I think part of this conversation would lead us there, but one of the issues in policy I'll say, research as well, but specifically in policy around coaching and I'll add mentoring to that too because those are terms that just sort of get thrown around together is that they feel fuzzy and vague and touchy-feely and a little bit like the reason early childhood has been put aside or is often put aside because it's like the stuff you do with little kids, I don't know, be nice, give them cupcakes. Coaching feels that way to a lot of people that are like, trade-offs, I need to make choices. Right now, I have this much money, what am I gonna spend it on? If you don't know what it is, it's really difficult to get it put through. This is the same issue with observation. What we're doing here is defining it and we're making it clear. This is what observation is and this is how it works and this is why it matters. If you can do that for coaching, I think we'd see a lot more possibility for the future. States are beginning to do career lattices and professional development systems, as you probably know in early childhood, that's really burgeoning and I think that that's critical too is really defining what the core competencies are and I think that that will go a ways toward addressing the issues that you bring up which are incredibly important. And yet expensive. Yes, always. Even more heightens the need to be very clear about what we're talking about and be able to show how it works and what it's for. Then you can say this is why we're paying so much for it because it's very expensive to have, as you know, to have coaches that can do real coaching and not sort of, again, these perfunctory observations that led us to here in the first place. Another related question that I won't try to answer but I'll put out is just how long is the observation supposed to be? This is a big question out there in the field right now. Is it enough to go in for 10 minutes? What if you go in 10 minutes 10 times? Is that better than going sitting for two hours? Those are really big questions and we do another recommendation is that we study this more. I think that there is big room for that and I know that's what you say all the time when you don't have the answers to just research it more but I do think it's very important because we do need to answer these questions if we're gonna have systems that work and are sustainable. Let me just quickly address the cost point because I think it's a really big one that we haven't touched on enough and I don't have an answer except to say that I would really like to see some research and analysis on professional development programs and their costs kind of the old school or the kind of typical professional development and then in their effectiveness contrasted with what we might be seeing from programs like my teaching partner or others they still have flaws and I think there are a lot of things that in the coaching realm can get too fuzzy. I mean I think there's a real risk of that honestly but really try to map out well how much money are our school districts already spending on something that we don't have any data on and we don't know is actually have any impact on ultimately on student outcomes and then what do we know that's actually we can trace some impact there. So let's go to right behind the man in the black sweater there. Yes, yes, thank you. Yeah, you. Hi, we want to thank you for this conversation because this was really very helpful. Can you introduce yourself real quick? Oh, my name is Bonnie Malanga. I'm the executive director of the 1199 Child Care Corporation from New York. Both Erica and myself drove, come over here to really listen to this very wonderful conversation and what drove us here is actually one of the observations that we made from many learning experiences that children go through in the classroom. So we have a learning center, it's a preschool and observations that we have made and I'd just like to share with you a little anecdote that brought us here. There was a group of children working on like it's a free choice that we're drawing and one of the children just said to a teacher, she said, Michelle, I'm going to give you pizza, slice of pizza. So what this child did was to put three points. And so the teacher was saying, oh, you made three points and she said, yes, I will connect the three points and I will have a pizza for you. So because our center is really very strong on the foundation of the theory of multiple intelligence and we're also really rolling on right now the bloom taxonomy. So in the mind of the teacher, she was thinking, what should I use? Should I expand the conversation to food, to visual, to let her go on with the drawing or to go to math or to go to language, language and linguistic intelligence. So those kind of thoughts was spinning in her mind in a very, very split second. So then she decided to go on with the conversation and she said, okay, so we'll have a pizza, you'll give me a pizza and maybe we will, what will we put in the pizza? So the child was saying cheese and chicken and all of these things.  So maybe I should expand the conversation to math. So she did. So she said, okay, there's three, do you know how many sides there are in your pizza? And the child said, yes, there's one, two, three. There's three sides.  And she even said, there are three points too in this, in my pizza. So she said, okay, so the teacher decided to go to math. So she said, okay, I'll go ahead with the conversation on math. She said, do you know that this triangle has a very, very special name? You know, there are three sides, that's true. And there's its triangle and three meaning three. So she said that. And the child said, so where's the angle? You know, so she kind of pointed out the angle. And she said, and what about these two sides? These two sides are the same. So that's what the child said. And she said, yeah, yes, it's true. There's two sides. And she said, you know, all triangles that have two sides have a very, very special name. And they're called esosceles.   Esosceles, esosceles. So at that point, you know, when I was looking at it and watching the teachers, she was thinking in her mind that, okay, this is like on the level of application in the Bloom taxonomy, okay, the child is evaluating the child is synthesizing, where do I go from here? So at that point in my conversation with the children, we were saying, what rubrics do we have now to capture this kind of experience? We're watching the teacher do this, right? I couldn't find any rubric that would capture that. You know, the New York right now is rolling on the common core learning standards. And, you know, going through all the performance evaluations that I was, you know, that I really went through just to capture that type of experiences, which is going on in our classrooms, in the three of our classrooms, I couldn't find. That's why when we saw the title of watching Teacher Work, we said, this is it. We'll drive to Washington DC and see and listen, you know. Listen to see what else we can do and how really us in the leadership positions can really help the teachers and help our coaches because we do have peer coaching in our system too. I mean, we have coaches, but we need our coaches to be trained, you know, so. That's another big question, it's the peer piece of this. Let's go to another one, thank you very much.  I'm right here, yes. I'm also from New York, and one of my roles right now is an early childhood consultant for Head Start Center. I came from a meeting yesterday in which some of those questions you just raised were addressed more specifically at a major teacher training institute. What they wanted to know from my director that I'm working with was how she could integrate all the different standards that she's accountable for with the common core standards now that being asked. And her question was about the performance tasks. You talked about one shot, one to one tasks not being reliable. We're talking about children assessment, right? Right. But right now she rolled out a piece of roll of paper that went three times the length of this room with the standards that her teachers have to coordinate, teaching standards gold, Head Start framework, all of those things, and now common core. If you take a performance task, a one shot task, like that, that may not be seen, and yet the DOEs collecting that for their evaluation of these Head Start Centers, the director was trying to indicate how difficult that is because of everything else they have to do with the diverse populations with the ELL kids. She has, one teacher has eight IEPs in one room of 16 children. Last year she had three homeless children in her class. So in terms of Head Start with all the diverse settings, how is this gonna be worked out in terms of teacher accountability? Not for class, that's not the issue. But in terms of what the DOE collects to determine what Head Start Centers are effective. One thing I'm hearing what you're saying, which I think is really interesting and worth pointing out is the common core conversation is in Head Start. And maybe it's not, I'm seeing some shaking heads, but I think that there are teachers who are teaching four and five year olds who are looking at the schools that their children will be moving into in the next year and trying to figure out, wait, what is the common core going to mean to me as a teacher, a four year old? I'm not gonna say I have an answer for you other than that alignment is going to be really important and recognition of the multiple standards that teachers are being asked to put themselves up against is gonna be incredibly important as well. Yeah, I would add not to be negative, but because I think the common core is a good thing, but something's gotta go. I mean, what's gonna happen is that things aren't gonna get done, because not everything right now that's being asked of teachers and of schools is going to get done. And so the extent to which common core can help focus and narrow or simplify things, that could be a good thing. But common core right now is on top of other standards and other objectives and other requirements. And it's all happening at once and it's happening very quickly. It's a very fast moving train. So I just take your point. I think we're hearing that all over the country and it is one of the biggest issues right now is how we will move forward with so many different reform strands and not break the backs of our schools and teachers. What we don't want to happen, and this is the point where I have a big caution, is for teachers just to say, oh well, I'm closing my door, I'm gonna teach my kids, because I don't know what you're talking about. I'm gonna just like not do anything different until my principal comes to my door and says I have to because I can't do it all and I'm overwhelmed by it. And I think we are seeing some of that. We need to find ways to avoid that because that will certainly put the brakes on some really important reforms. Really interesting point. I think we have time for just one or two more questions. Let's see, where are you Claire? Okay, right here in the front, please. Hi, I'm Wendy Eptain. I'm a former kindergarten teacher and I'm now working at a nonprofit called Hope Street Group, where I'm working to connect teachers with policymakers and specifically working with some states as they're kind of revamping their teacher evaluation systems. And so I think you guys kind of made a mention of it at one point, this idea of multiple components being part of a teacher's whole evaluation. And observations are just a small part of that, or in some cases a large part depending on what the state has decided. And then another part of that is student test scores. And obviously this is difficult in some of these non-tested grades and subjects, especially early childhood grades, but even for some high school grades as well, foreign language, PE, all these things. So I'm just curious kind of from each of you guys what you propose or what you think should be in place of student test scores. Or maybe another way to think about it is what components should be part of a teacher's evaluation in addition to observations? That's a great question. I think that actually a lot of us should probably answer that. And I think we've come from different perspectives on it all. Maybe just hear from the teachers quickly. And I'm happy you mentioned that because I was thinking what I heard most of you mentioned. And it sounded like you guys were thinking of scores, test scores being the source of what the salaries or the great decisions would be made based on. And I'm thinking of my quality indicators that we use and even class and alcohol. And my thought process is that it's not so much the scores of the kid, but what the teacher does that is taken greatly into account. Did the teacher do certain things that could make, that is an effective scaffold or instructional support if you get what I mean. So I'm thinking that I'm really defending quality indicators here because it goes, as you said, putting it in the hands of your 17-year-olds, even the four-year-olds, what do they care about what your salary would be? They don't care about that. But what can you do and what have you done is what I think is the greatest consideration for a teacher's evaluation. So Danielle, yeah, in a K-12 setting, there's already a conversation about this. I wanna get an SOL-tested subject, 11th grade English. My problem is by using that as part of my evaluation, it's not fair to me and it's not fair to that particular, to the ninth and 10th grade teachers who also taught that student because as we know, our SOLs are spiral. It builds on the skills from ninth, 10th and 11th. I've actually had a chance to look at some of the SOL questions on these tests and these questions go back to something they learned in the ninth grade, things that they've learned in the 10th grade. It's spiral. So you can't just look at an SOL score as a part of my evaluation because now you're cutting out teachers in ninth and 10th who don't have SOL scores. How are you gonna evaluate them? So if you're gonna look at scores, you're gonna have to look at something more than just the standardized testing score. Maybe you're looking at an evaluation from the beginning of the year of a score that the kids did on a diagnostic from the beginning to the end. But you have to do it fairly and across all levels. Because as you said, the foreign languages, they don't have SOL scores. They don't have the standardized testing. Even with some of the English courses or with some of the math courses, you have SOL test at math courses and non-SOL test at math courses. So you need to find some type of, if you're going to look at a test, needs to be something that's fairly distributed throughout the entire school. Or you need to look at something other than the standardized test score. Right, well, so I don't think there's any question that test scores are not enough or that they're not the full measure, but they are a measure and we have them and to not use them in my opinion would be irresponsible. So they exist. Could they be better? Yes, and there's actually just to add to the wave of things happening. We have assessment consortia right now, two of them trying to develop these amazing new assessments that will change assessment and they will make the tests that aren't good better. Because we'll have like common core standards, we'll have sort of common assessments that states can choose to adopt or not adopt. Hopefully it'll work, but again, we've got two years, they're supposed to be done or something. So it's a really big challenge and a big climb. But if we had better assessments, and I'm the first one to say, a lot of tests out there aren't good. Some are, some are okay, but a lot of them aren't and they aren't enough, but they are there. And we do have research that shows there are correlations between quality indicators and these tests. So if in fact the teachers that do well on the quality indicators are also the teachers that have kids that have high test scores, well, there's something there worth looking at. So what I would say to your question is you need observations and again, I would say you need a combination of the external and the person in the building that knows them. There needs to be some combination there. You need to look at some standardized measure because the other piece is that we wanna be able to look across, right? We wanna be able to look across school, across district, across state. And right now that's test scores. Make them better, yes, but right now that's test scores. And then in the MET project, they're looking at student feedback. I would imagine that there's a student-parent feedback piece. I don't know how the parents are coming into this, but they should because they're the ones that are gonna start looking more and more at grade schools and all these others and making decisions based on, by the way, test scores, primarily. If there was more good, rich information in there, you might have parents be able to make more informed decisions and be more a part of this schooling experience that right now they're not as much a part of. In the end, most of the research will say, and not surprisingly, that the most predictable measures you'll have are multiple. So you have multiple raters, multiple observations, and then essentially what you see is this building of a lot and a bigger and bigger system. And so that's another complication with this, is we can't afford to, in time or in money, to have 18 observations with six different people. But the larger point is that a single observation or a single measure isn't enough. You wanted to make it, because I'm the radical here. So, and Lisa knows this, and this is, with all due respect, I am not a great fan of standardized tests.  And it's just, I just think that they're very unpredictable and I think they're all kinds of cultural biases. So, I do think that just because they're here and they're objective does not mean that they are the best way to, you know. No, so they don't use it all, that's a question. They're not the best, but so they be used it all. As they are now, no. I would say so. And then I guess I can take the prerogative of having the last word on this, in which, because Susan and I actually kind of disagree on this point a bit, because I see that the tests are telling us something. They're not telling us much, but they are telling us something in the tested grades about where kids are and we can't ignore that information. What I think is really hard and complicated about this value added conversation is the practicality of doing it really, really well. So when you look at the researchers who talk about, you know, how to filter out all the different variables that need to be there and getting the data over multiple years for the same teacher and making sure that the kids have attended for the kind of the same amount of days in the school year and gets incredibly complicated. And I think that that complication is something we can't ignore. I think in the early years and the early grades, it's even more fraught with complication and some practical questions. So what I think that is important, and that's why I think this is a really interesting question and I'm just gonna close here, is that it absolutely has to be in multiple measures. Observation cannot be the silver bullet on this, but the way we do observation has to change and become more standardized if it's going to be useful at all in evaluating teachers. And so being able to use tools that can give you a score that you can then compare to teachers in other districts to districts or headstart scores or school scores to other schools, grades to other grades, that is a highly important piece of information when it comes to kind of quality of teaching. And I wanna just end with just reminding us of just the achievement gaps that we are dealing with here and how important getting quality teachers, highly effective teachers with the kids that need them most will be so important. So I'm gonna say a huge thank you to everyone here. Very much, thank you for coming. And this will be, this will have been recorded. It will be available online tomorrow and I'm looking forward to hearing more comments. Thanks again for coming.