Please welcome Brian Jacob. Hello, good morning. It's great to see so many folks here. Again, I'd like to thank all of our doctoral students. Okay, I thought for you I was going to have to bring it down a little bit. Down, I'm the middle. You can see how much in charge of anything I am. So, yeah, just again a thanks to all of the very hardworking students and postdocs and staff who have helped make this event possible. So, let's see. I'm going to jump right in. I think try to give some discussion of this. Our initial finding is the Michigan Merit Curriculum. I'm going to say this a few times as I go through the presentation, but one thing that's important to keep in mind is when we say the impacts or the effects, what we're really talking about here is just on the very first cohort of students that experienced the mandate of the curriculum. So, as you know, the first group for whom the curriculum was binding was those who were graduating in 2010-2011. And so because we're looking at things like high school graduation and not just four-year graduation but five-year graduation and trying to look at college enrollment, not enough time has passed for us to look at these outcomes for subsequent cohorts. So, I think this should be useful information, but we shouldn't take this as the final word on anything. Okay, so just a little context. Where did this come from? What is the context in which this Michigan Merit Curriculum was proposed and adopted here in Michigan? It was really in the context of stagnant high school achievement across the U.S. for the past 40 years. So, some statistics here. Achievement has barely moved, and you don't have to take my word for this. Wow, this is slow. So, these are NAIT, National Assessment of Education and Progress, Achievement Trends in Math. Look at the line for 17-year-olds on the top. Compared to 1973, those students in 2008 were virtually the same. And reading, this looks at reading, and this shows you for 17-year-olds and reading, doesn't just give you the average or the median, which is the 50th percentile line in the middle, but looks at the very lowest achieving students and even the very highest achieving students at the 90th percentile. And you see, these numbers have barely budged at all. And so, I think this is people, this data is made possible by Jack Buckley and all the folks at NCES. And this helps us understand the lack of improvement. I think that was one of the motivating factors in the adoption of the MMC. High school graduation rates have not risen much nationally. So, this is something, first of all, they've been very hard to measure, surprisingly. But in the last five or so years, researchers have kind of looked more carefully and found ways to properly account for various kind of mobility and so forth and found figures that look like this. This is based on high school graduating classes. If you go back, I mean, for males, comparing the high school graduates of 1966 to 2006, that is 40 years. And we are at the same place. Female's been some improvement, but that's been mostly in the last, you know, five or so years. So, people look at these trends and I think are motivated to change. BA attainment is leveling off despite the large economic returns to college. So, some figures at SUE and others have put together before.  The blue line is BA's, completed BA's. And that certainly was growing up over time. The x-axis here, the horizontal axis is actually year of birth. So, it's a little bit different. But if you look at the last, you know, five or six years on the blue line, you'll see there's not been much change in BA attainment rates across the US. So, okay. And then, as many of you have heard in the news, kind of the US performs about average internationally and that has some people worried as kind of economics becomes more of a global marketplace. What are some of the common policies that people, states, districts have tried to improve high school performance? When you go back and look at this, I was actually a bit surprised at how many different types of reforms there have been at the elementary level. When you look at the high school level and what kind of reforms have been tried, it's much more limited. There's been some organizational structural changes. There's some work, especially in earlier decades on career technical education, some dropout prevention programs. But by far, the most kind of common reform to stagnating high school achievement has been to raise standards in some form or another, most prominently high school graduation requirements. So, and this is not new. And one thing that I think it's important to understand the context of the MMC is that we've had policies like this for a long time, starting in the 1970s, minimum competency exams of basic skills that were starting to be required for graduation. The 1980s, for those of you who are kind of active in education at that time, you probably remember the new basics, which is the push for all states to raise the number of required courses to four years of English and three years each of math, science and social studies. And many states and districts did that. High school exit exams. And then no child left behind. I guess the last decade, our country's been mostly focused on no child left behind from the biggest perspective, which I think most people believe is focused largely on elementary schools. And there hasn't been as much national attention on high schools. That is changing, obviously. So what is the rationale for higher standards? One, and these are things that come up not only in national debates, but if you go back and look at the record in Michigan in 2005, 2006, when the MMC was passed, you'll find kind of these arguments being made among advocates. One is that it ensures students are prepared for college and are jobs in the 21st century economy. It kind of adds more coherence to the education system. One of the, I think, primary intended benefits for many is to reduce some of that variation Sue showed us before. We know there are big gaps in high school graduation and college enrollment even within districts and certainly even within high schools to some extent that might be due to variation in course taking and if we can kind of formalize and improve the course taking of some students that weren't taking rigorous courses in the past, maybe this will help reduce some of the gaps later on. The thoughts that kind of implementing these new requirements would spur other changes in curriculum, pedagogy, teacher or professional development. Now, what are some of the concerns with higher standards? There certainly was a concern voiced at the time and continually that they may lead to increased dropout rates particularly among disadvantaged students. People have cited the reduced opportunity to take electives and CTE courses. On the other hand, I know talking with some folks at the State Department and various districts that there really are still kind of substantial opportunities for some elective taking, but that's certainly been mentioned as one issue. There's a concern about unintended consequences. Certainly some mention of things like cheating. One thing that has come up anecdotally is kind of relably of courses now that everyone has to take and pass Algebra 2. Well, maybe they are, but Algebra 2 might not be exactly like we think Algebra 2 would be and this is something that we don't have much evidence on now but as part of kind of the second phase of our research project we are embarking on a large study of high school transcripts and we have and will be contacting probably many people in the room to try to encourage your high schools to work with us and helping us look at student high school transcripts and to shed better light on course taking changes as a result of the MMC. One concern is that this requires substantial capacity on the part of teachers in schools that sharply increase the demand for math and science teachers and that's a difficult area to fill even in the best of times. Now we're going to have even higher a higher bar and to what extent is the state prepared for that. And then there might be some concern that this reform really won't have much of an impact perhaps positively or negatively on students that were already taking these courses and we still need something for those students. So can we learn anything from past efforts by other states and districts that have adopted similar policies? Well there's been a good bit of research on high school exit exams these exams students take in 10th or 11th grade that are required for high school graduation. I mean I've done some of this work, some other folks in the room have done some. I think it's fair to say that there's been little promise of these policies they have increased drop out rate slightly among low achieving groups and don't show much measurable improvement in student achievement among other groups. There is somewhat more evidence to support raising graduation requirements there has been some evidence that of economic payoffs for taking higher level math and science courses. So that certainly was I think one of the thoughts behind the Michigan AmeriCurriculum. There have been very few policies thus far that have attempted something exactly the same as the Michigan AmeriCurriculum the only the policies that we have evidence on are two policies one in Chicago and one in Charlotte Mecklenburg, North Carolina where they were really requiring all students to take Algebra I and even Mecklenburg requiring them to take it earlier in 8th grade as opposed to 9th or 10th grade. These the results of these studies have not been too positive they led to reductions in student achievement. The authors of these kind of evaluations have cited two potential hypotheses which I think are important for us to keep in mind when thinking about the Michigan AmeriCurriculum. One is these students that were kind of quote unquote forced into the classes under these policies may have been over placed not sufficiently prepared for the courses and that teachers may not have been prepared to teach these courses or these students in these courses and so this is something that we might think about when we're interpreting some of the results for the AmeriCurriculum. Okay, so this is a chart showing other states that have adopted policies similar to Michigan. So these are states that have required students high school graduation classes to pass at least Algebra I in geometry. You can see Kansas starting with the high school graduating class of 2009. Oklahoma in 2010. Michigan is among the first of these states but not the very first. One of the next steps in some of our research here is to do some comparisons comparing Michigan's experience to the experience of these other states but again when we use national data sets to look at this there just hasn't been enough time elapsed to get college enrollment, high school graduation from many cohorts for these various states. So we'll be certainly using data in coming years to try to examine variation across states and hopefully learn from that. So what Michigan high school reform? I think many of you, most people in the room are familiar with this but just so we're all on the same page. Michigan passed a set of reforms in spring 2006 that were based on a set of new K-12 education standards. One was the introduction of the new high school exam, the Michigan Merit exam, part of which was the ACT. And that's nice for research purposes because it gives us another kind of common metric and one that's used across the country to track high school student achievement. And as opposed to the past, we're looking at ACT scores, one would always be concerned, well not every student takes the ACT and of course when you're looking at ACT scores but that's only the more college inclined students. The set of data that we're looking at starting with the ninth graders in 0405 all of these students were required to take the MME and thus the ACT so when you look at the test taking rates they're really quite high and don't change much over time so we don't have that issue of selective test taking that we might have had in the past. Michigan Merit curriculum, this is what we all know of, we'll be talking a bit more about that in a slide or two. And then the Promise Scholarship, as Barbara mentioned this was discontinued, certainly was a very important reform. Initially we'll be doing a little bit of research on that in the coming years as well, this was part of the package of reforms that was passed. So the Michigan Merit curriculum, a set of more rigorous college prep courses that were required for graduation. Okay, so the ones that you mostly hear about are the math credits, so I think the innovation of the Michigan Merit curriculum was not the fact that four math classes were required but which classes specifically were required. So students were required to take and complete algebra one, algebra two, geometry, plus one other course. Science, biology, and then chemistry or physics plus one additional course and some of the other requirements that folks know of are by understanding is that the math and science, kind of what anecdotally and based on the data we have were really the biggest changes relative to what had been happening in the past. To give us a sense of how big those changes were, there was a survey done by the superintendent in 2005 of districts to understand what districts were requiring on their own prior to the passage of the MMC. And you'll see that algebra one, only 32% of responding districts were requiring even algebra one. Once we get down to algebra two, only 12% of responding districts were requiring algebra two. And we have some other data that I'll show at the end. Among those districts that weren't requiring it, a very small fraction of students were taking these kind of upper level math and science courses. So this is just to underscore that this was a big change. This is a big change that students and teachers throughout the state have been grappling with over the past few years.  As a gentleman mentioned earlier, there is a kind of personal curriculum that allows students to graduate in ways that are not exactly like those outlined before. My understanding is it's relatively limited. There were initial plans for statewide and of course exams, those I think for funding reasons, perhaps for other reasons were never progressed to fully to completion. And then, so I think one of the other interesting things about this policy, which is distinct from some in other states, is the end of course exams that students are required to pass to indicate that they've met the Michigan AmeriCurriculum can be developed by schools and districts themselves, and the schools and districts themselves can set their own standards. This is very different than other states where there is a centralized state algebra one exam where they centralized passing cutoff and so that's another kind of unique feature of the AmeriCurriculum. Okay, so finally getting to some of the current study and some fun figures like Sue has. So what are we doing here? The MSR, our consortium is engaged and have two different pieces to this research project. One is a statistical analysis of student achievement, high school completion and college going. This is the work that I'll be presenting to you today. The second though is this Michigan high school transcript study which I think is going to be at least as, if not more interesting and informative than the statistical analysis where we're doing some more intensive data collection in 150 randomly selected high schools throughout the state. And the goal there is to measure the implementation of the Michigan AmeriCurriculum and to try to help uncover some of the underlying mechanisms, how students and teachers were responding, what some of the challenges were, what some of the successes have been. But today we are going to stick to the statistical analysis for someone like me that feels very comfortable and natural so I'm happy about that. As Sue mentioned, we are focusing, this analysis focuses on ninth grade cohorts. This is students starting ninth grade in 0405, up through students starting ninth grade in 0809. One thing that we're going to be doing throughout this that is not so much shown in the figures but is underlying all the estimates and results that we show you and everything that we do, we've been controlling for students eighth grade math scores and demographics and other school characteristics. So one thing that I think everyone is aware of is that Michigan, like the rest of the country, has experienced an economic downturn over the past five, 10 years. And you can see that very easily in the data, I think as Ken will show later on, the fraction of students eligible for free and reduced-priced lunch in the state has jumped considerably. And so we're trying to account for those changes using some of the information we have in this rich administrative data set. We know that otherwise there had been some improvement in student meep scores over time, perhaps better preparation among elementary school students in Michigan. Again, by looking at students eighth grade math scores and controlling for that, we're trying to take into account those changes. The research design we're using is something about an interrupted time series. I will illustrate that in one of our slides coming up. The outcomes are the ones we've mentioned before, high school completion, college enrollment, high school achievement. And I think the one big caveat is that all of the estimated effects that you're going to be seeing here are based solely on this very first cohort of students to experience the Michigan Merit Curriculum. For these ninth graders that were falling over time, we track them and out of all the different outcomes that students could experience, we group them into four categories for simplicity. Graduated, that is the official graduation, so it is not a... But it could include folks with personal curriculum, but it is not some other high school completion measure. It is kind of the official graduation indicator. Whether they're still enrolled in the Michigan Public High School, that's the other outcome. Whether they dropped out. And again, this is kind of the official, somewhat narrow definition of dropping out that is used kind of in some, some, but not all Seppi reports. And others, the category of folks that either transferred to a private school, left the state, or had an unknown status, meaning kind of their district and school didn't quite know what happened to them. And note this kind of other category is relatively small. Probably one reason for that is we're starting with students who are in ninth grade in Michigan Public High Schools, so it's not as if we're not capturing a lot of the turning and exit that you see up through ninth grade. So what do we see here? What do these pluses and minuses and zeros mean? These are meant to show you the effect. So this is kind of the effect of the Michigan AmeriCurriculum. So if you first look at all students, four-year graduation. This is whether they graduate high school within four years of their ninth grade year. That negative sign meant there was a reduction. So we'd say the Michigan AmeriCurriculum reduced the likelihood of four-year kind of on-time high school graduation overall. This was a small, very small effect to kind of give you a rough guideline. It kind of went from 72% to about 70%. So it was kind of small change for that group. Still enrolled, you'll see that that actually increased. And over four years, in the four-year time period, no change, zeros mean no change, no effect on dropping out or kind of being in this other category. I think one thing that's going to be, I think the most interesting thing, and hopefully the thing that jumps out when looking at this table, is if you look at top quartile students, for most of our analysis, what we've done is we group students into four groups based on their pre-high school performance. This is kind of taking into account their eighth grade standardized achievement scores, their poverty level, a whole host of factors, and using that information kind of have quartiles of students, we are mostly in this presentation going to be focusing on the top and the bottom quartile, because that really illustrates the spectrum of effects. If you look at the two middle quartiles, they kind of roughly mirror. The third quartile roughly is similar to the fourth, and the second is similar to the first. So to clarify, we just look at the top and the bottom.  So really, that's telling you that the Michigan Merrick curriculum did not have any impact, positive or negative, for the highest performing students entering Michigan high schools. This is in terms of their graduation enrollment dropout. We'll see, we'll later, I'm going to show you the achievement effects. But for bottom quartile students, we see there were some impacts. We see reductions in high school graduation, and then we see an increase kind of in all of the other categories. You know, they were less likely to graduate, and what were they doing instead of graduating? Well, some of them were still enrolled. Some of them had dropped out, and some were in this kind of other category, which can include a bunch of different things. So let's go to the figures, and hopefully this will kind of clarify some things. This is a four-year graduation by initial achievement quartile. You'll see the bottom line is kind of for the lowest quartile students, and the top line for the top quartile. So even in 2005, I should say for the ninth graders in 2005, who were six, seven, eight, the high school class in theory of 0708, the four-year graduation rate for this group was only about 40%, up to about 90% for the top quartile students. And so what we're doing in this analysis is tracking the changes over time for these different cohorts of students. So for the top quartile students, you see pretty much even trends, 90%, 90%, 90%. And it doesn't really change much. This red line here distinguishes the class for whom the Michigan AmeriCurriculum was binding. So you'll see this, the group here in 2008. These are freshmen in 2008. So if you count forward, that means they're sophomores in 09, juniors in 10, seniors in 2011. So this was really the group, the first class for whom the requirement was binding. Not much change in this group. Going down to the bottom quartile, you'll see that from 05 to 0607, there was slight increases in the high school graduation rate of this group of students. And then in 08 for this group, it declines noticeably. So one thing we're going to be, just to kind of give you an intuition, what should we have expected this 2008 high school graduation rate to be? Well, you might think if the trends in the state, this kind of slight, modest improvement among the bottom quartile students had continued, you might expect the line to be there. And then what we are doing, in the essence of what researchers refer to as the interrupted time series, you say, well, we might expect if our trends continued as before, the 08 cohort to be here, we see them down here where the dark triangle is. This little square bracket here shows you the difference between where we expected them to be and where they ended up. And this illustrates kind of a negative impact on high school completion. They are below, the black triangle is below where we expected them to be. So throughout the analysis, when I present the results, you can kind of think about this intuition here. One thing we also did is in some cases, if we weren't sure about the prior trends, we just compared the 2007 versus 2008. So this shows you kind of the four-year graduation reductions. So where are these students? What happened to them? Well, this is an indication of whether they were still enrolled in a Michigan public high school after four years. So this is in the fall of what would have been their fifth year in high school. And so we see here increases. You know, it's going down here and then it kind of bumps back up again. And so as illustrated by this graph, after four years, what largely was happening is students were extending their stay in high school. We didn't see any large changes in dropouts right after the fourth year. We didn't see many changes in kind of this other category. Basically fewer people were officially graduating, but they were back in high school in that fifth year, presumably trying to make it through a lot of the additional requirements. So that was, I think, if we want to take away something for the four-year high school completion results, that's the main takeaway. Five year. This is after five years. And so you notice we only have this last cohort, 2008 is the last group year. That's because five-year graduation for them was June 2012. And that's kind of the latest data that's available. We see reductions in this group among the bottom quartile. Again, notice for the top quartile or even the top two quartiles, not much changed at all. We see still enrolled. Some increases among students that are still enrolled. And then four-year dropout rates. I'm going to skip through that. Fifty-year dropout rates. So this is at the end of five years. What do we see? Again, now you're going to have to switch your eyes. For dropout rates, the bottom lines are the highest performing students because they have the lowest dropout rates. This top line here is this bottom quartile of students. Right here, we see, you know, for the 2007 cohort of ninth graders, it was about 11%. And then for the 2008, it went up to about 12 or 13%. So it was an increase of about two percentage points relative to an 11% baseline that's kind of the increase. Again, among this bottom quartile of students. So you'll see throughout this, one of our main takeaways is very big differences based on the prior student preparation, which is something we probably shouldn't be surprised about based on earlier policies like this. Are there any questions on this? Like, I know I've been talking just nonstop for a while here. So there's two things. One is to break these students up into quartiles, it's largely using their eighth grade math test score. Now, it does take into account some things like whether they're free and reduced price lunch. It's got a general measure of achievement and socioeconomic status kind of, you know, combined. But it might be easiest to think of as just quartiles of eighth grade math achievement. So, any questions on, yep? Let's call them the Michigan Community College Association again. I'm just a quick question. So can you talk, in terms of the... I may be answering my own question here. I'm trying to get a sense of the number of students overall. How many students are we talking about in cohorts? Let's see, each cohort of ninth graders is about, let's say, 120,000, 125,000. And so 1% of that would be 1,300. Is that right, Sue? You have to check my math. So 1% point would be 1,300. And so if we're saying a 2% point increase in dropout would be 2,600. Thank you. Yes, sir. I was interested to know if you track that amongst IEP students or students with learning disabilities. We certainly could. Again, like Sue, we have that data. My guess is sometime in the last two months among the 150 tables we did, I have that somewhere I don't remember it off the top of my head. I just know from personal experience and amongst my peers, I think that gap from the four to five-year kids, that's primarily that group. At least that's been our experience. So this could, a lot of this, that could be one of the reasons why this bottom quartile is showing, if you're saying that the largest of facts might be for students with IEPs, there's probably, I think there is a large overlap between students with an IEP and students with kind of lower eighth grade math scores. So I think that's, a lot of them are probably contained in this upper line here. Although you're right, it's not focusing explicitly on them. Interestingly enough, we can keep them until they're 26, but it counts against our graduation rate where they don't finish before. Right. Okay, good. Ishmael from Macomb Community College. I was wondering whether you have found any regional variations in terms of districts? Yeah, this is, that's a great question. We have not done that yet. So this is what we did as we had done kind of results overall, and then we started by looking separately based on some kind of student subgroups. And we did this a few different ways, but I think we decided the most informative way to show this was based on their prior preparation. I think the next step is to be looking separately by districts, regions along the line that kind of Sue mentioned, to understand whether there are particular districts that are, you know, appearing to be more successful or districts that are having kind of more challenges. I mean that's, we haven't done that yet, but we certainly will. Julian Wilson, Portage Public Schools Kalamazoo Resa. So let's go back to 1A. 1A, yep. So we've got approximately 125,000 ninth grade students in this cohort in, let's say, 2005. Call us. I don't get my kitchen math and swag in here. So pretty, pretty constant in the state of Michigan we graduate. We have cohorts of about 125,000 students a year. So yeah, I mean it's been going down a little bit the last five years, but 125-30. Okay, so the cohort in 2005 is at ninth grade. And then is that, that they graduated in 08, or is that strictly a ninth grade cohort? No, so the 05 dot, the dot here that is the 2005, let's say the 2005 black diamond or black triangle, that is the fraction of students who entered ninth grade in 0405 who were in the bottom quartile of their entering high school class. So it's taking ninth graders in 0405, the bottom quartile of those ninth graders in 0405 and then looking, following them four years later, so that would be about 6 or 7 or 8. So looking at by June 08, what fraction had graduated high school? And so 40% by June 08, 40% of that group had graduated high school. Michigan Public High School. Extension of the MME. Thank you. Okay. So let me kind of push ahead now. I have some other results that are going to show on high school achievements. So maybe I'll show those and then we can come back to answer other questions as well. So now we're going to look at test scores by student achievement test scores. And so we're going to look separately by subjects. First of all, this figure is all the ACT scores. The figure on the left is for the top quartile students. The figure on the right is the bottom quartile students. The numbers have been standardized. So for those of you that are familiar with these, these are standard deviation units or kind of effect sizes. And so let's look at the top quartile. So for the top quartile students, you see there were steady increases prior to the MMC in all subjects. The top line is math. This is kind of the... And again, if you want to get your head around this, ninth graders in 05, if they're taking an 11th grade, this would be the 2007 test-taking group going up. And then not much, a slight improvement from 07 to 08. But if you look at this, it looks like it's kind of roughly following the same pattern as before. So actually when you do add the controls and do the statistical models, what you come out with, there's not much effect on math, a slight positive effect for the highest achieving students on their ACT score in math. But then look down the next two lines or science and reading. Here you see much more noticeable increases between the 07 cohort and the 08 cohort. And when you, again, run this through the statistical machinery, you get modest... I would say modest, meaning bigger than small, but smaller than huge, improvements in reading and science achievement. And so this suggests for the top quartile students, the Michigan Merit Curriculum was increasing science achievement, increasing reading achievement, as measured by the ACT, had maybe a tiny increase in math, but not so much in math. The big, one thing, the noticeable difference is writing. You see ACT writing scores kind of going along mostly, even a little bit upward and then dropping down dramatically after the MMC. So we see a substantial reduction in writing achievement associated with the MMC for the top quartile of students. For the bottom quartile of students, we see some similar pattern for writing. This top line is writing right here, it goes here, it goes way down. We see not much changes for any of the other subjects. So relatively even trends, so really not much positive or negative achievement effects for the bottom quartile. Okay, here's what we do looking at the MME test scores. One interesting pattern of results that we found that you might notice if you look carefully here is that in general the trends and the improvement among Michigan high school students, both prior to the MMC and during the MMC period was larger for the ACT scores than it was for the overall MME scores. Now that's a little bit tricky to think about because we know the MME, part of the MME is the ACT. So that is a bit confusing to think about, but I think I'd be curious to hear people's thoughts or hypotheses about differences between ACT and MME at some point. Here again we see, I think the pattern of results is for the top quartile student maybe some slight improvement, certainly a noticeable improvement in science. For the bottom quartile students not much, not many, no positive effects and we see some, this is kind of for reading, maybe math, maybe a negative effect there. And so maybe the easiest way to summarize this is here. For the top quartile students we see positive effects for the ACT in science, math and reading, but negative effects in writing. And for the bottom quartile students we see the zero, remember the zeroes are no effects, the negative is the minus signs are negative effects and the positive are, plus signs are positive effects. If you are kind of, if you are thinking yourself, okay what's the bottom line, tell me in one sentence what does this mean, why is this, there's all sorts of different things here, and I think after wrestling with the data for many months and talking it over, I think the bottom line is that it is a mixed pattern of results. There is not one clean, simple answer that is true for every type of student in the state. I think in general we see kind of more positive effects for the upper quartile students in terms of these achievement test scores, which is consistent with the, you know, kind of better outcomes for them in terms of the high school completion. We see some more positive effects for science than for math, which I think was, we certainly didn't predict that, and that's kind of interesting and worthy of further explanation. And we see across the board for all types of students, substantial decreases in writing ACT writing scores. So why don't I stop there now for some more questions about this? Let's compare to the interrupted time series. Is this positive gain in these subject areas above and beyond what you would expect as far as the trend line goes, or is it just increases? When you see the summaries here, these are plus or minus or zero relative to what you would expect. So a plus for the ACT science doesn't just mean the 2008 was higher than the 2007. It means it was significantly higher than you even would have predicted it to be based off of that. Mary Beth Dan, U of M. So I have a question. Did you look at the effects or try to control it all for, I would expect that there were any increased rigor in terms of classes yet from the change in terms of that. But I also wondered, what about educational financing in the state? What effect did that have and do you think it shows up? I understand the increased amount of test prep is interesting. I work in schools all over the place and many of them have been increasing the ACT test prep, especially at lower achieving schools. And it's fascinating to see that this might be an interesting result for that too. Those are both good questions. We definitely at this point have not looked into kind of any of those deeper mechanisms. And I think that the test prep is the type of thing that we're hoping to get from the high school transcript study when we go, which is going to include some interviews with principals and teachers, not simply the collection of transcripts. So I think we're going to, hopefully some of that will come out in this kind of second big piece of the project. But that's an excellent point. Was any data gathered for the top and bottom quartile with regard to radiation rates for those going to community colleges? You know, we, not yet, although that is definitely on the agenda. One of the, we have some college enrollment results. And I think the college enrollment results largely near the high school completion. See, not much change at all among the top group. The bottom quartile group, we see reductions in college completion, which is not surprising because we see reductions in high school graduation. But I think we've been a bit more leery to even kind of start looking at the college enrollment because it's been such a short period of time. We know that a lot of students even before weren't graduating in four years, there's a comment from the audience that usually for low income students you see college enrollment more two years after. So we're definitely going to be looking at all sorts of college outcomes including remediation. I think that will probably be the next the next set of analyses we do. What are the, what are the hypotheses behind the writing scores going down for all students? I mean, I think this is I think some of the hypotheses that we've thought about on our end. I'm sure people here who are actually working in schools every day will have an even better sense. But our initial thought had been largely a reallocation of time and effort and resources away from writing to some of the new requirements on the MMC. So I think it's What were the requirements because as well? Well, I mean the English requirements there were four years, it was four years of English but almost every district before was requiring students to take four years of some English. There were maybe small changes in English and kind of the social studies history, but if you go back and look at the new requirements relative to the old state but also relative to what districts were doing on their own, you see that the primary changes that were being required in the MMC were changes in the math and science. And so I mean that result of the writing is consistent. There's other studies that show reductions in achievement among kind of low stakes subjects. So even when you look at elementary schools under NCLB and you see improvements in reading and math, in some places in some cases you see slight reductions in science and social studies and the hypothesis there is teachers are kind of shifting the focus kind of toward the higher stakes subjects. Evan Hordeig from Looking at figure four would I be interpreting that correctly to say that in 2009 the reduction more than offset the gains from 2008? So yeah, so this is certainly true that's kind of the this is the raw data kind of the average achievement of this group. This doesn't have any of the kind of controls for eighth grade ability or the eighth grade achievement scores of the other factors. I think when we look at this in some of the models when we looked at this the 2009 group didn't look as looked a little bit worse than the 2008 group but not as much as this picture would make you think yeah, but this is I don't have the exact numbers off the top of my head but the 2009 group was not quite as positive as the 2008 group Brian Glenn Nelson from Ann Arbor Schoolport I just want to put a little more emphasis on Mary Beth's question about finances that I think the estimates you get on your control variable or variables with respect to resources available or funding might be of at least as much interest as those you get on the MMC that in Michigan we've had this general decline in funding but it's been different across districts because of the move towards equalization so you have variation among districts as well as the statewide trend and to me at least it would be very interesting even though it's probably going to be called a control variable what the parameter estimate on that is in six years from now maybe a lot more significant policy than the stuff around MMC yeah, that's right so this is one thing that and I think Glenn is right he's pointing out the importance of economic conditions and funding levels irrespective of what the particular high school graduation requirements are at the time which I think is a good point I think the point that Sue wanted me to remind people about is that in kind of the analysis, this interrupted time series it kind of understands and kind of takes into account any smooth or constant change over time so you looked at the trends and when we were looking at the change relative to the prior trend because we know that Michigan today is different than Michigan was five or seven or ten years ago I mean in some positive ways in some less than positive ways in terms of the economic climate and none of those kind of the general trending of the economy one way or another shouldn't substantially influence the estimate we get using this the research design we're using now so Dana Dyson University of Michigan Flint, I have a question perhaps that I don't know, maybe it's more of a statement I don't know, but it seems that we just are going to have students in the state that are going to continue to decline in terms of their academic achievement based on this particular reform when you have in the bottom quartile students who are either the MMC is either not having a fact or it's having a negative effect those predictions for the future seem very negative that said we know that there are unintended consequences for policy all the time and could you please speak on the fact that this suggests that in a few years we're going to have 21-year-old students who are going to be in the same facility with 14-year-olds one thing that one of the big take-aways for us among the research team was as I mentioned before the variation the way in which the policy seems to have differentially affected different subgroups of students and I think clearly the most from my perspective the most troubling aspect of the policy is the effects on the bottom quartile of students I think I think that's actually the benefit of some of this early analysis, early research that we can hopefully some of what we're doing here some of the more depth qualitative work that others are doing and continue analyses we can try to really pinpoint what Sue said the choke points and the pipeline or the problems with this particular policy itself and try to help schools and districts adjust and help their students even more the one last thing I want to show to highlight the importance of this kind of transcript study that we're embarked on is one of the things that we'd like to do in addition to looking at the high school completion and achievement outcomes is looking at course taking and obviously the best way to do this is to look at the high school transcripts what we have here while we are still working on the high school transcript component of the study we are able to look at the ACT data so as you know students that take the ACT which is now all test taking students in Michigan high schools are required to fill out a questionnaire where they answer lots of questions one set of questions they answer are about what courses they have taken or plan to take by the time they graduate and this data should be taken with a large grain of salt because A it is student self reports and B courses that may contain certain content might not be labeled the same thing across schools but kind of even even with a large grain of salt we thought it was sufficiently interesting to put these in the presentation the yellow highlighted rows are kind of the fraction of students the percent of students who report taking the courses that would have met taking or will have or would have taken they report that they have taken or will eventually take the courses required for meeting the Michigan Merit Curriculum and so we see in the top row for these classes of 0809 2010 about 30 percent of students were reporting taking algebra one, algebra two, geometry and one other math course this increased to 38 percent in 2011 so we do see some change even among these very you know rough and approximate student self reports you know and we see the same thing below for science kind of the fraction of students taking biology plus chemistry or physics goes from 60, 70 percent to 80 percent but so I think we do see some change which kind of I think tells me that there's some information in these figures but one thing that we see is you know the number 38 percent in 2011 for the fraction taking classes that would meet the MMC 38 percent is very different than 100 percent and even if we take into account the personal curriculum and even if we take into account the large grain of salt of students self reports I think this is suggestive that there's kind of been differential implementation across schools and districts in the state and I think this kind of illustrates the importance of the second component of the project we're doing which is the transcript study where we can actually get in and talk to teachers and principals look at transcripts and really try to understand what was going on in the schools so this is the final slide conclusions in ports of examining impacts separately by generation level potential slippage in implementation will schools grow in their ability to implement over time hopefully the transcript and related data work will shed some light on that there's some variation across subjects which I think is kind of interesting in puzzling the math versus science I think is particularly kind of interesting in puzzling to me and then the broader implications for other states as you saw before Michigan is just one of many states and more than half the states so far have passed policies very very similar to the Michigan American curriculum so many many states in the coming years will be implementing similar reforms and then the common core is going to present a roughly similar challenge probably many other challenges as well but kind of a similar challenge at the high school level in terms of more advanced course taking requirements so with that thank you very much


Chapter 13.5 of the 9-1-1 Commission Report. This is a LibriVox recording. All LibriVox recordings are in the public domain. For more information or to find out how you can volunteer, please visit LibriVox.org. Recording by Elana Jordan. The 9-1-1 Commission Report. Chapter 13.5. Recording America's defenses in the United States. The future role of the FBI. We have considered proposals for a new agency dedicated to an intelligence collection in the United States. Some call this a proposal for an American MI5. Although the analogy is weak, the actual British Security Service is a relatively small worldwide agency that combines duties assigned in the U.S. government to the terrorist threat integration center, the CIA, the FBI, and the Department of Homeland Security. The concern about the FBI is that it has long favored its criminal justice mission over its national security mission. Part of the reason for this is the demand around the country for FBI help on criminal matters. The FBI was criticized rightly for the overzealous domestic intelligence investigations disclosed during the 1970s. The pendulum swung away from those types of investigations during the 1980s and 1990s, although the FBI maintained an active counterintelligence function and was the lead agency for the investigation of foreign terrorist groups operating inside the United States. We do not recommend the creation of a new domestic intelligence agency. It is not needed if our other recommendations are adopted to establish a strong national intelligence center, part of the NCTC, that will oversee counterterrorism intelligence work, foreign and domestic, and to create a national intelligence director who can set and enforce standards for the collection, processing, and reporting of information. Under the structures we recommend, the FBI's role is focused but still vital. The FBI does need to be able to direct its thousands of agents and other employees to collect intelligence in America's cities and towns, interviewing informants, conducting surveillance and searches, tracking individuals, working collaboratively with local authorities, and doing so with meticulous attention to detail and compliance with the law. The FBI's job in the streets of the United States would thus be a domestic equivalent operating under the U.S. Constitution and quite different laws and rules to the job of the CIA's operations officers abroad. Creating a new domestic intelligence agency has other drawbacks. The FBI is accustomed to carrying out sensitive intelligence collection operations in compliance with the law. If a new domestic intelligence agency were outside of the Department of Justice, the process of legal oversight, never easy, could become even more difficult. Abuses of civil liberties could create a backlash that would impair the collection of needed intelligence. Creating a new domestic intelligence agency would divert attention of the officials most responsible for current counterterrorism efforts while the threat remains high. Putting a new player into the mix of federal agencies with counterterrorism responsibilities would exacerbate existing information sharing problems. A new domestic intelligence agency would need to acquire assets and personnel. The FBI already has 28,000 employees, 56 field offices, 400 satellite offices, and 47 legal attaché offices, a laboratory, operations center, and training facility, an existing network of informants, cooperating defendants, and other sources, and relationships with state and local law enforcement, the CIA, and foreign intelligence and law enforcement agencies. Counterterrorism investigations in the United States very quickly become matters that involve violations of criminal law and possible law enforcement action. Because the FBI can have agents working criminal matters and agents working intelligence investigations concerning the same international terrorism target, the full range of investigative tools against a suspected terrorist can be considered within one agency. The removal of the wall that existed before 9-1-1 between intelligence and law enforcement has opened up new opportunities for cooperative action within the FBI. Counterterrorism investigations often overlap or are queued by other criminal investigations such as money laundering or the smuggling of contraband. In the field, the close connection to criminal work has many benefits. Our recommendation to leave counterterrorism intelligence collection in the United States with the FBI still depends on an assessment that the FBI, if it makes an all-out effort to institutionalize change, can do the job. As we mentioned in Chapter 3, we have been impressed by the determination that agents display in tracking down details, patiently going the extra mile and working the extra month to put facts in the place of speculation. In our report, we have shown how agents in Phoenix, Minneapolis, and New York displayed initiative in pressing their investigations. FBI agents and analysts in the field need to have sustained support and dedicated resources to become stronger intelligence officers. They need to be rewarded for acquiring informants and for gathering and disseminating information differently and more broadly than usual in a traditional criminal investigation. FBI employees need to report and analyze what they have learned in ways the Bureau has never done before. Under Director Robert Mueller, the Bureau has made significant progress in improving its intelligence capabilities. It now has an Office of Intelligence overseen by the top tier of FBI management. Field intelligence groups have been created in all field offices to put FBI priorities and the emphasis on intelligence into practice. FBIs have been made in improving the Bureau's information technology systems and in increasing connectivity and information sharing with intelligence community agencies. Director Mueller has also recognized that the FBI's reforms are far from complete. He has outlined a number of areas where added measures may be necessary. Specifically, he has recognized that the FBI needs to recruit from a broader pool of candidates, that agents and analysts working on national security matters require specialized training, and that agents should specialize within programs after obtaining a generalist foundation. The FBI is developing career tracks for agents to specialize in counterterrorism, counterintelligence, cybercrimes, criminal investigations, or intelligence. It is establishing a program for certifying agents as intelligence officers, a certification that will be a prerequisite for promotion to the senior ranks of the Bureau. New training programs have been instituted for intelligence-related subjects. The Director of the FBI has proposed creating an Intelligence Directorate as a further refinement of the FBI Intelligence Program. This Directorate would include units for intelligence planning and policy, and for the direction of analysts and linguists. We want to ensure that the Bureau's shift to a preventive counterterrorism posture is more fully institutionalized so that it survives beyond Director Mueller's tenure. We have found that in the past the Bureau has announced its willingness to reform and restructure itself, to address transnational security threats, but has fallen short, failing to affect the necessary institutional and cultural changes organization-wide. We want to ensure that this does not happen again, despite having found acceptance of the Director's clear message that counterterrorism is now the FBI's top priority. Two years after 9-1-1, we also found gaps between some of the announced reforms and the reality in the field. We are concerned that management in the field offices still can allocate people and resources to local concerns that diverge from the national security mission. This system could revert to a focus on lower priority criminal justice cases over national security requirements. A specialized and integrated national security workforce should be established at the FBI, consisting of agents, analysts, linguists, and surveillance specialists who are recruited, trained, rewarded, and retained to ensure the development of an institutional culture imbued with a deep expertise in intelligence and national security. The President, by executive order or directive, should direct the FBI to develop this intelligence cadre. Recognizing that cross-fertilization between the criminal justice and national security disciplines is vital to the success of both missions, all new agents should receive basic training in both areas. Furthermore, new agents should begin their careers with meaningful assignments in both areas. Intelligence and analysts should then specialize in one of these disciplines and have the option to work such matters for their entire career with the Bureau. Certain advanced training courses and assignments to other intelligence agencies should be required to advance within the national security discipline. In the interest of cross-fertilization, all senior FBI managers, including those working on law enforcement matters, should be certified intelligence officers. The FBI should fully implement a recruiting, hiring, and selection process for agents and analysts that enhances its ability to target and attract individuals with educational and professional backgrounds in intelligence, international relations, language, technology, and other relevant skills. The FBI should institute the integration of analysts, agents, linguists, and surveillance personnel in the field so that a dedicated team approach is brought to bear on national security intelligence operations. Each field office should have an official at the field office's deputy level for national security matters. This individual would have management oversight and ensure that the national priorities are carried out in the field. The FBI should align its budget structure according to its foreign main programs, intelligence, counterterrorism, and counterintelligence, criminal and criminal justice services, to ensure better transparency on program costs, management of resources, and protection of the intelligence program. The FBI should report regularly to Congress in its semiannual program reviews, designed to identify whether each field office is appropriately addressing FBI and national program priorities. The FBI should report regularly to Congress in detail on the qualifications, status, and roles of analysts in the field and at headquarters. The FBI should ensure that analysts are afforded training and career opportunities on a par with those offered analysts in other intelligence community agencies. The Congress should make sure funding is available to accelerate the expansion of secure facilities in FBI field offices so as to increase their ability to use secure email systems and classified intelligent product exchanges. The Congress should monitor whether the FBI's information-sharing principles are implemented in practice. The FBI is just a small fraction of the national law enforcement community in the United States, a community comprised mainly of state and local agencies. The network designed for sharing information and the work of the FBI through local joint terrorism task forces should build a reciprocal relationship in which state and local agents understand what information they are looking for and, in return, receive some of the information being developed about what is happening or may happen in their communities. In this relationship, the Department of Homeland Security also will play an important part. The Homeland Security Act of 2002 gave the Undersecretary for Information Analysis and Infrastructure Protection Broad Responsibilities. In practice, this directorate has the job to map terrorist threats to the homeland against RSS vulnerabilities in order to drive our efforts to protect against terrorist threats. These capabilities are still embryonic. The directorate has not yet developed the capacity to perform one of its assigned jobs, which is to assimilate and analyze information from Homeland Security's own component agencies, such as the Coast Guard, Secret Service, Transportation Security Administration, Immigration and Customs Enforcement, and Customs and Border Protection. The Secretary of Homeland Security must ensure that these components work with the Information Analysis and Infrastructure Protection Directorate so that this office can perform its mission. Homeland Defense At several points in our inquiry, we asked who is responsible for defending us at home. Our national defense at home is the responsibility first of the Department of Defense and second of the Department of Homeland Security. They must have clear delineations of responsibility and authority. We found that NORAD, which had been given the responsibility for defending U.S. airspace, had construed that mission to focus on threats coming from outside America's borders. It did not adjust its focus even though the intelligence community had gathered intelligence on the possibility that terrorists might turn to hijacking and even use of planes as missiles. We have been assured that NORAD has now embraced the full mission. Northern Command has been established to assume responsibility for the defense of the domestic United States. Recommendation The Department of Defense and its oversight committees should regularly assess the adequacy of Northern Command strategies and planning to defend the United States against military threats to the homeland. The Department of Homeland Security was established to consolidate all of the domestic agencies responsible for securing America's borders and national infrastructure, most of which is in private hands. It should identify those elements of our transportation, energy, communications, financial, and other institutions that need to be protected, develop plans to protect that infrastructure, and exercise the mechanisms to enhance preparedness. This means going well beyond the pre-existing jobs of the agencies that have been brought together inside the Department. The Department of Homeland Security and its oversight committees should regularly assess the types of threats the country faces to determine, a, the adequacy of the government's plans and the progress against those plans to protect America's critical infrastructure, and b, the readiness of the government to respond to the threats that the United States might face. We look forward to a national debate on the merits of what we have recommended, and we will participate vigorously in that debate. End of Chapter 13.5 End of the 9-1-1 Commission Report


Everybody forced here up in twenty twenty one and twenty one twenty two optic basketball Hobby four box mixer number seven It's a random team style break. Check out the results late in sports cars calm anytime. There's a link in chat for you as well Yeah, we know Ryan is pitching. Yeah, we're all room for the marriage though. We're pretty cool Yeah, we're going just he's killing a man No Oh Manual quickly Yeah, so I started 2021 this time. I guess twenty one twenty two. We started look last time. We didn't hit anything huge. So Holler any threes JJ reddick pelicans JJ reddick Things look really promising, you know, they're actually caring about the team A green Andrew Wiggins Nice green one of five the dubs Nice Wiggins there. It's got the Warriors Scott F with the Warriors Why has been rated rookie as well as a hollow Jason Tatum? Who scored I think I may have started bears defense somewhere and that's looking pretty good All stars red nick on the yoke at 66 of 99 nuggets Nice Jeff. You have a 20 box high-end basketball mixer on sunday I love me some high-end basketball Yeah, man. It's got like NT noir first off the line Got a lime green Warriors rated rookie Nico manian 91 of 149 spectra's not too far on the corner too I feel like Should release wednesday wednesday Yeah, man. Wednesday. It's crazy day. Hollow rated rookie buck store noir. I think that's where it's at still Hopefully it stays that way. I don't think change quickly the schedule now And T rogues world cup wednesday, too.  Yeah Let's hope so I'll have to be maybe watching I probably want to be able to watch Hollow Celtic shale and brown I'll be gone for five days Matt's gonna do the two wednesday and friday daytime makes sam just breaks tuesday through saturday night Sixers auto tony bradley signature series That is not numbered Sam's gonna be tired man. Yep That's a long stretch Marcus smart 79 to 199 Although all stars well and being sixers All right, jeff with that lineup. What team would you want in the random? There's a lot of different stuff It's tough to say Blue yannis this year 16 to 59 bucks It's 21. It's 2021 and 21 22 just different different years of each Or the high-end products in those two years hollow rated rookie nicks will be topping I'll send a box wrapper with this. There's like a dimple on the right side of the for the topping T was probably pretty good probably have the biggest rookie to two years The pistons will be pretty good too because you got not only There's a Edwards base right there not only um Cade but you guys to the base while the other year so it's higher four Warriors probably a good team to get not the biggest rookies, but all that chance at high-end stuff and those rookies Isaac a coro dominators rookie auto calves Six of 99 I don't think so kyle, but what's up, man? I knew the bears would do something like this Leave it to the bears No, they're they're space von. They're space Or a kyle they're space sorry noora a lime green 22 of 149 bucks Von with his selfish abby That's sick. That's a sweet abby von.  I'm pretty sure we open up against the sixers next week Is that the opener next Tuesday? I think so You play the sixers hollow stargazing jazz out of a Mitchell and then we play the heat on friday It's been a magic on saturday at home.  Yeah, I get back in like at night I probably won't have enough money to go to a game after after next week anyway Pistons orange jerry grows 10 of 199 We can hit up our boy Hollow rocketeer at gordon hollow grizzlies jaren jackson jr Hey, one of our reggie be dope. You don't see reggie in products though. Oh, that's right.   I'll topple in steep bay. Oh, man. There we go Biggest optic card of the night so far. I would think that wiggins green was sick too. So this is definitely better than the Ones in the last break. So we got 21 22 now Josh christopher will go real world die one two three left four five six rights for the all-stars cards here There's anything gold or lower all random A four going right side Right side for the whole couple of boxes here orange kings jaren fox 180 of 199 Carl It's like watching a high school game Hollow grizzlies john randt guys. There's probably preseason basketball on you can be watching or the ucf game but even then A lot of points are being scored by this lopsided. So it's not super exciting fron's Wagner rated rookie 56 to 13 still the third good lord, man comminga Got a 75 years kade cunningham pistons There's a ding on that card right side. I'll send a box wrapper to the pistons Now amazon's doing just fine. Don't you worry magic johnson and larry bird If you ever catch me feeling bad for amazon, that's a mistake amazon needs nobody's sympathy. Hello nix mitchell robinson james booknight rated rookie 67 mil per game. That's like chump change to them kade cunningham. That's like that's like my grocery bill to them Got lime green nets blake riffin numbered 119 of 149 actually right now grocery bills way higher than normal Things are insanely expensive right now Pretty much anything but food is going way up right now, especially food right now bones highly got hollow wizards rude hutch moorth we're like gas went down to like Kind of where it was it's about and food is just outrageous Like you can't buy eggs We got spent like double what they used to cost heaven mobly rated rookie Yeah, I usually just don't eat it actually I can afford bread and peanut butter and jelly. So there you go Davey on mitchell got hollow rated rookie j.t. Thor for the hornets Who's is getting cheaper? Hollow stargazing jazz done in Mitchell. I believe it Dude, they just gonna have the overrunners on these games at like 22 I'm way low 75 years jamal murray no wave on I wouldn't count on it Red hornets gordon hayward 73 of 99 Alligator is one cent a pound in a pocket right now. No, it's not Illegal hornets hollow terrors here He loves apopio though, it's his favorite place um Outside of actually his favorite place ever hollow rated rookie auto charles bassy 76ers Now you can find it the apopka airport Which is which is honestly just uh 4 36 very late at night with no traffic Right by it That's the road I get to go home on I just take 4 36 all the way home I live off of 4 36 Yeah, alligators great. We have it everywhere on here every restaurant floor to sell the alligator. Yeah The mcdonalds here sell sell the alligator and the burger king Gator stocks Lime green pistons killian hay is 63 of 149 the best ones are always out like A fish camp. Yeah, where there's like no ac Yeah hollow rated rookie nets Cameron Thomas with his like they're giving out they have like, um Airboat rides and the captain's missing teeth.  Oh, yeah Yeah, true we need like apopka like tourism ads Yeah No, no legally don't Scotty Barnes rated rookie. I did say it like, you know, it I wouldn't put it past places in Florida to have the mick gator mick gator nuggets Dang it. How am I supposed to see my gator bites that honey mustard as they livers Hollow tray young reigning threes hawks, even though that's what you eat it with, right? Yeah Just the chicken nugget. So it is. Yeah, should we chicken nugget all the rated rookie hawks Shreve Cooper? So I think there's a there's like a Cajun Like New Orleans style place But I used to order from you know, the old shops doesn't out him on or longwood somewhere And I'm pretty sure I got like a gator bite po boy one time It might have been tibbies because I also used to get like I used to order like a crawfish pobo before I go live Sometimes because that's like the biggest sandwich you could get hollow thunder Derek pavers and it was so good Like if I'm gonna eat one meal today You get a crawfish po boy Tatum and paul george The rookies kade cunningham Yeah, sorry, man. Maybe there was something here josh skitty rated rookie a red charles bassie sixers 66 of 99 Herbert jones hollow blazers damien lillard Sixers charles bassie rated rookie auto Another bassie 25 years jaylon johnson Nice a bones highland orange 18 of 199 nuggets rated rookie hollow lakers russell westbrook And another auto so two autos in the last box wizards spencer din witty 67 of 99 Quick recap here. Yeah, wizards auto spencer din witty Sixers bassie bass and hollow rated rookie autos okoro calves rookie auto to 99 Signature series sixers tony bradley a green to five andrew wiggins for the warriors And a desmond vane bane black velocity Grizzlies will get these cards right out to everybody. Thanks you for joining. We are on to Football now


Hi, I'm Zor. Welcome to Unisor Education. Well, let's continue the course of advanced mathematics for teenagers and high school students. It's presented on Unisor.com. That's where I suggest you to watch this lecture from because it has notes and exams for registered students and other goodies, and it's free. Today we'll talk about one particular aspect of statistics. It's related to normal distribution, random variables which are distributed according to the normal distribution law. So, you know what normal distribution is. It's this bell curve usually represents this particular distribution. Now, it's very important in statistics to deal with a random variable which is distributed normally because then it's just sufficient to evaluate its mean value, mathematical expectation and standard deviation to basically know everything about the distribution. So, distribution is defined by only two parameters which we usually use mu for mean or mathematical expectation and sigma for standard deviation. Now, knowing these two from whatever our empirical data are, we basically can recreate the entire distribution and find the probability of our random variable to be within any kind of a boundaries from two which is basically what distribution is all about, right? So, it's very important for distribution to be normal in statistical research because it's easier to deal with. And then we always know all these nice rules like two sigma rules for normal distribution which means that the probability of random variable to be within two sigma boundaries from its mean it's 95%, 0.95. So, we know these nice things about normal distributions and that's why it's easier to deal with. Also, let's not forget the fact that in many, many statistical research we are dealing with random variables which actually do behave like variables with normal distribution. And let me explain you why. Let's talk about one simple natural phenomenon like temperature of the body, of the healthy person. Well, we know that this temperature is somewhere around, what, 98.6 Fahrenheit or about 37 degrees Celsius, right? We know that. That's kind of an average temperature. Well, mathematically speaking, temperature of a randomly chosen healthy person is a random variable which has mean, mathematical expectation, this particular value and some relatively narrow distribution, statistical standard deviation from this mean variable. Now, why do we consider this particular random variable, the temperature of the human body to be distributed normally? Well, and actually many other phenomenons in nature. Well, here's a very simple explanation. We all should recall central limit theorem in theory of probabilities which in its simplest form states that if you have random variables which are independent and identically distributed, their average behaves more and more normally as n goes to infinity. So this is the central limit theorem. The more random variables we mix together, the more result looks like the normally distributed random variables. I do encourage you actually to go to a lecture about central limit theorem in Unisor.com to basically feel that this is true. Now, it's not only for independent identically distributed random variables that this particular central limit theorem is true. Actually, it's true for much broader category of random variables which we are mixing together. They're not necessarily completely independent. They're not necessarily identically distributed. But again, the central limit theorem has certain very, very broad conditions under which the more we have these mixed together, the closer the distribution is to normal. And now, let's go back to temperature of the body. How this particular random variable, which is the temperature of the randomly chosen person, how is it actually formed? Well, let's just think about it. The human body is extremely complex organism. It has millions, billions, trillions, whatever different cells which each of them is doing something, probably emitting certain amount of heat. And all of this is basically added up together. So all these miniscule amounts of heat emitted by every cell of the body is added together. And so what we have right now is that the temperature is the result of a mixture of huge number of random variables, each of them associated with some little cell in our organism. And that's why it's reasonably to assume that the distribution of this random variable, the temperature, is normal. It's normally distributed with certain mean and certain standard deviation. And we do know actually the mean for the healthy person, these numbers, because that's the result of our statistical research throughout all the humanity. We know all the temperatures of healthy people and they're all somewhere around these numbers. Not exactly. They're still healthy if this is 98.7, for instance, or in Celsius, 36.6. But anyway, it's very normally distributed around these mean values. So if you don't know anything about how this particular random variable, which you are measuring, is distributed normally and not normally, and you would prefer it to be normal because then you can use all these nice properties with two sigma, etc. So it's very nice to make sure that your distribution, the distribution of the random variable which you are sampling, is really normal. And now I will just talk about very quickly. I mean, it's a very big introduction into a very short story about how you can basically make a judgment whether a particular random variable is normal or not normal based on sampling of the data. Okay, so I'm suggesting two different ways. There are actually much more precise and much more interesting but there are much more complex ways to prove or disprove normality. What I'm going to prove is something which is really, really very, very simple and most likely it will be sufficient for some kind of a rude experiments. So number one is related to histogram. Now, we all know that normally distributed random variable has bell-shaped distribution. It's the density of probabilities, if you wish. So what we can do is we have a certain amount of data. Based on this certain amount of data, we can have maximum and minimum, which we have obtained. Let's say we have a thousand different elements of data. So we can choose something like 25-30 intervals between minimum and maximum and then we just count how many values fall into this interval, this interval, this interval. And you see, I'm drawing my histogram more or less in accordance with this bell-shaped form. So if your histogram looks like this, well, it's a very good indication that your random variable, which you are measuring, does have normal distribution. If, however, using these samples, you get something like this, this doesn't look like normally distributed random variable. So what's the properties of a histogram which we can actually pay attention to? Well, the first property which indicates normality of the distribution is symmetry. It should be symmetrical relatively to the middle between the maximum and minimum. So symmetry is important. Another, what's important? We have to have a hump on the top here. So you probably will have one maximum by height rectangle in the histogram and then symmetrically down all others. So that would be a better choice for a histogram which has normal distribution. It would be very nice if you can identify concavity, if you wish, on the top. Concavity directed downwards, which is gradually converted into concave upwards as you go to periphery from this. So if you have these symmetry hump on the top and then concave directed upwards on the periphery, then it looks like a bell shape and then there is a very good chance that your distribution might be normal distribution. What's next? So here is my first methodology, if you wish. Just look at the histogram. It should look normal. Okay, now what's the next one? Next one is more related to calculations. Let's think about this way. If we know that a particular distribution is normal, then there is a rule, if you remember, the sigma rule, two sigma rule and three sigma rule. Now what is sigma rule? Well, the probability of our variable, random variable, to deviate from its standard deviation, the probability that it will be no more than sigma is 0.65, 65% or 68, I think 68. Yeah, it's 68%. For normal distribution with mean mu and standard deviation sigma. Now similarly, probability of this deviation of the random variable from its mean that it's less than two sigma is 0.95. That's the most frequently used evaluation. And the probability of our variable to deviate from its mean by no more than three sigma is 0.997. So we know these rules. So what does it mean? It means that if our sample from which we can calculate sample standard, the sample mathematical expectation and sample standard deviation after calculation of these, and then we can basically construct the interval from mu minus sigma and mu plus sigma, right? And see how many values fall into this interval and compare it with the total number of values. If we don't have something closer to 0.68 as a ratio, well, that means it's not probably as normal as we expected. I mean, if it's 0.70, it's probably okay. But if it's 0.1, that's definitely not okay. That's definitely different from whatever we expect to be from the normal distribution. So after we have calculated sample mean and sample standard deviation, we can actually evaluate. So here is my mu, here is my sigma, two sigma and three sigma interval. So this is from mu minus three sigma and two mu plus three sigma. This is my mu minus two sigma and mu plus two sigma interval. This is my mu. And finally, this smaller one is from mu minus sigma to mu plus sigma. So after we have calculated mu and sigma based on the sample which we have, we just count how many out of an entire population which we got fall into this smaller interval, how many into bigger interval, and how many to the biggest interval. If our ratios are not around these, then it's probably not normally distributed random variables. If it is, then the chances are it is. So if you have histogram which looks like a bell shape, and you have sample data which you can just calculate the frequencies of falling into these intervals based on your sample calculations, if these frequencies are around these, well then the chances are you can really think that this particular random variable is normally distributed. And for the future, you can actually predict certain probabilities based on this. Since you know the distribution is normal with these parameters, then you can actually predict because what's the purpose of the probabilities? The probabilities purpose is to predict what's the purpose of statistics. The purpose of statistics is to find the probabilities. So you found these two values, mean and standard deviation. Using this, you have proved that your distribution is normal. And now since you know the normality of this distribution and main parameters, new and sigma of this distribution, then you can basically say that the probability of the new experiment to have the data in this particular interval, whatever the interval is, it may be even from this to this, doesn't matter which one. Since you know new and sigma, you can evaluate any probability using the corresponding formulas in case of normal distribution, it's integral, but it doesn't really matter. You can do it, that's the problem. So you can predict what's the probability of your random variable to be within certain range. And that basically ends this problem of identifying normal distribution. Again, there are much more precise, much more scientific ways to determine the normality of the distribution. And don't forget that you cannot really determine normality with 100% certainty. As everything else in statistics, whatever you are stating about anybody or anything, or any random variable, etc., it has certain probability of being true. It has certain certainty level. So there are very important approaches, methodologies, which can tell that, okay, this particular distribution is normal and this is a certainty level of this particular statement. Now, I'm not going into these details, they are really very, very advanced and it's probably only for professionals who really specialize in this area. But this probably would be the most important and kind of the most precise characteristic, which you can give to a random variable observed through the sampling data, that this particular random variable is normal with such and such parameters and the certainty level of this statement is not 100%, significantly less. All right, so that's it for today. Thank you very much. Try to go to unizord.com again and go through the lecture, through the notes. It's always helpful. There are a couple of pictures of normal and not normal distribution. And that's it for today. Thank you very much and good luck.