OTAN, Outreach and Technical Assistance Network. Nobody knows your students and your program better than you do. So really, you're the experts when it comes to the how. I'll tell you about the why and the what, and then we'll get to the how. OK, before we do that, I would like to know a little bit about you. So I'm going to try to launch a poll here. So I just want to know about your familiarity with IELCE. So how familiar are you? Very moderately familiar, a little, or not at all? So please go ahead and answer that and submit it. OK, this is good, good to know. OK, so 10% are very familiar, 25% are moderately, 47, a little, and 18, not at all. OK, this is good for me to know. So well over half of you either only have a little bit of knowledge or are not at all. So that's good. What I'll say for those of you who have just a little bit or not at all, this is going to be a lot of information. And if it's your first time hearing it, it can be a little overwhelming. So just take a breath and just make your way through it. Absorb as much as you can and realize that there are lots of resources out there for you to get your questions answered. There are lots of resources on the web that I'll show you. There are lots of folks who would be happy to talk to you about it. So please don't feel overwhelmed if it's too much information at this point. OK, here's the definition of IELCE that comes from the Workforce Innovation and Opportunity Act. I'm not going to read this whole definition to you, but I'll highlight a few items here. So first of all, in that first line, it talks about English language learners. That's who this is developed for. I like the second line about professionals with degrees and credentials. That's an interesting addition they made there that is applicable to those folks. And so the fourth line about acquiring basic and more advanced skills, that's very important, function effectively as parents, workers, citizens in the United States, literacy and English language acquisition, citizenship, civic participation, and may include workforce training. So why does it say may? May include. Let's dive into that a little bit more. The reason it says may and not must or shall is that there are two ways to look at it. There's the activity. There's IELCE as an activity and as a program. As an activity in California, we call it English literacy and civics education. You may have heard of this as EL civics and it's been around for over 20 years. This is an activity under Section 231, the base grant. And when you're doing it as an activity, when you're doing EL civics as an activity under Section 231, it may include workforce training, but it doesn't need to. The other way it's delivered is as a program. And that would be the Section 243, integrated English literacy and civics education program. And when you're implementing this as a program, then it must include workforce training. And the rest of this presentation is going to be talking about that second one as a program, Section 243. And we'll get into that workforce training requirement. This may be the most important slide here because it really lays out sort of the goal and the philosophy behind this. So an ILC program is designed to prepare English language learners for and place them in unsubsidized employment in in-demand industries and occupations that lead to economic self-sufficiency. So right there, we see that this is a program that's getting people jobs. It's specifically mean to prepare them for a specific occupation and to put them in that occupation. So when we start talking about the difference between workforce training and workforce preparation, think about that, that workforce training really gets them a job or gets them the opportunity to have a job. And then also, number two, integrate with the local workforce development system and its functions. And then down at the bottom, it says that ILC program must be provided in combination with IET activities. OK, great, we have another acronym, IET. Let's talk about IET. IET stands for Integrated Education and Training. And it's a service delivery model that provides adult education activities concurrently and contextually with workforce preparation activities and workforce training. So you've got three components. You've got the adult education and literacy. That's what you all do. Hopefully you know that one very well. Specifically for this program for IELCE, that tends to be ESL. You've got workforce training that, in most cases, that might be what you would call career technical education, CTE. And you have workforce preparation, which could be soft skills or any intermediary skill that gets people ready for a workforce training program or gives them those soft skills that are useful. We'll talk more about workforce preparation when we get to co-apps. I know co-app was something that Jay mentioned this morning. And co-app is a California's way of addressing that workforce preparation. So we'll talk about that. And then in the heart of all that, if those three things are happening concurrently and contextually, then we have integrated education and training. And if you have integrated education and training, then you have a compliant IELCE program. I'll just note briefly that IET doesn't have to be in IELCE, in a solely in IELCE. IET could be in many different program areas throughout the nation. ASE students are involved with IET programs, ABE students. Of course, for this program that we're talking about, IELCE, it's focused on English language learners. So for the most part, we'll be talking about ESL students. OK, what does this currently and contextually mean? So it means that within the overall scope of the program, those three components, adult education, workforce prep, workforce training, must be of sufficient intensity and quality. There has to be enough of it, it has to be good. And based on the most rigorous research available, it has to use occupationally relevant instructional materials. And those three components have to occur simultaneously. Simultaneously, people are always asking, so what does that mean exactly? Well, it kind of means what it says, happening at the same time. One of the easiest ways to think about it is what it's not. It's not sequential. Sequential would be the old model, where perhaps a student would come to a school and they would be enrolled in an ESL class, and maybe they would be in that ESL class for a year or two years or three years. And then finally, once they were ready, they would be able to get into a workforce training program, a CTE program, that may have been their goal from day one. This model turns that on its head and says, these students should be able to begin that contextualized instruction sooner. And by contextualizing the instruction, we accelerate the learning in both the English realm and also the career technical. So to meet I&T requirements, a program must have a single set of learning objectives and activities organized to function cooperatively. This single set of learning objectives comes from the regulations. And it really just means that the ESL teacher, in this case, and the CTE teacher, and perhaps an administrator or coordinator, has come together, sat down, and developed a cohesive set of learning objectives. Well, it's not an ESL objectives and then CTE objectives, but they are integrated, truly integrated. And this is really just the first step. I mean, really, you should have a full syllabus, an integrated syllabus for the programmer of the course. The second bullet, obviously, talks about aligning with the state's content standards for adult education. And then the last one, to be part of a career pathway, very important, maybe the most important, to really start thinking about your program as part of a career pathway, or several career pathways, depending on how many programs you have. And not only that, but the students should know what pathway they're on. If we come out to a school and we're talking to students, we'd like to know that they know what pathway they're involved in in this program. This is a list of training services that come from the WIOA statute. We're not going to read all of these, but you can see that they happen in many different modalities or settings. They allow on-the-job training, incumbent worker training, the bottom there upgrading and retraining. It goes on, entrepreneurial training, transitional jobs. So again, a pretty wide menu of different types of training services that are allowed under WIOA. In California, it's helped us to think about this training more as what industry-recognized credential does it lead to? And this is just a list, not necessarily an exhaustive list, but some potential credential types that you might be using. And this might depend on the setting that you're in. So for example, with the second bullet, if you work closely with your workforce board with Title I, for example, maybe you know providers who are on the employment training provider list, ETPL, or maybe your school itself is on the ETPL and offers that, that would be fantastic. Maybe if you're in a community college setting, the third and fourth bullets might look familiar to you. If you're in a K-12 LEA, maybe the fifth and sixth, apprenticeships, pre-apprenticeship programs. So lots of different options. And really, this comes down to you telling us what credential your training program leads to. We sort of put it on you as the provider to prove to us that you've got a great industry-recognized credential that your training leads to. OK, co-enrollment, this is an important slide. So remember I mentioned at the beginning that IELCE, when you're implementing it as a program under Section 243, has to include workforce training. In the overall program design, it has to have that and it has to occur somewhere in there. However, when the Feds were setting this program up, they recognized that not every student in the IELCE program would necessarily be taking advantage of that training. They may not be ready for it. They may not want to do the training at this moment. And so what we say is that the English language learners in this IELCE program have to have the opportunity to be enrolled in a class or a program that offers workforce training in their specific career pathway. So not every student needs to be doing the career training, but a percentage greater than zero does. We need to see some students doing it. And certainly all students need to have the opportunity, need to have access to that training. I hope you understand that distinction, because that's an important one. Here are different types of co-enrollment. When we say co-enrollment, usually we're talking about ESL and CTE. So for example, in the first one, maybe you've got classes that are offered. I mentioned the ETPL earlier, the eligible training provider list. Maybe you've got students who are co-enrolled in one of those courses. The second might be about an in-house CTE program that you have at your school. Oftentimes that's the easiest. If you actually have CTE programs, you can implement it more easily. If you don't, the third bullet acknowledges that sometimes you need to cobble together a partnership with one or more partners, maybe through your consortium or through your title partners. And then the fourth one, acknowledging that you could also have this relationship directly with the employer who is providing pre-apprenticeship, apprenticeship on the job training, et cetera. When we talk about team teaching strategies, we break it down into basic two categories of co-teaching and alternating teaching. Co-teaching simply means that the teacher, the ESL teacher, the workforce training teacher are in the same classroom at the same time delivering instruction to a group of students. Alternating teaching means that those teachers are in different courses at different times, but it's coordinated. And even though it's at different times, it's still concurrent, meaning it's not one semester they do this and then the next semester they do this. It's happening at the same time. I see the question in the chat. I'll get to that in just a moment. OK, co-apps, get another acronym for you. Jay mentioned the co-apps earlier. And I'll just talk a little bit about them here. So co-apps stands for Civic Objective and Additional Assessment Plan. It's a description of a performance-based assessment that you will give learners after 30 hours of instruction based on the civic objective that they're working under. As I mentioned earlier, these co-apps in California are designed to meet the workforce preparation requirement of IET. That doesn't mean you can't have other workforce preparation activities happening there. But specifically, we believe that those co-apps fulfill that. In addition, when you're selecting a co-app, you can either pick one that's already been created or you could propose new ones. If we have time, I'll show you on the website where you can find these. And there are around 30 or so that are designated of these civic objectives that are designated for Section 243. And then that second bullet, just another reminder that the co-apps should be part of a workforce preparation curriculum targeting in-demand occupation or cluster of occupations. OK, just a couple more slides, and then we'll take a break for questions. So the program requirements, you do need to have a student needs assessment that you administer each year to find out what the students actually need. And this is a document or series of surveys or assessments that we will ask to see if we come out to visit your school for a federal program monitoring review. Of course, as I mentioned, you'll be selecting the co-apps, the civic objective, and additional assessment plans. You'll be creating the additional assessments. These additional assessments are created by the agency or borrowed from somebody else who's created them. That's OK, too. Of course, you'll plan and offer the instruction. We talk about 30 hours minimum of instruction here. You administer those additional assessments. Sometimes not every student passes the additional assessment, in which case it is allowable to provide some additional instruction and offer the assessment again for them to pass. Not necessarily in this order, but you'll see for six, you will need to administer pre and post-cossess testing. And then at the end of April, you'll be completing an IELCE report that report describes what your program did with IELCE. I'll talk a little bit more about that in just a moment. Something that everybody's always curious about, the payment points. How does the money work for this? So here's a little table that illustrates the relative payment points for each of the types. So if we start up at the top, the top is indicating Section 243 program, where you've got an IELCE student who takes and passes one of those additional assessments and is co-enrolled in workforce training. So if that student passes the additional assessment and is co-enrolled in training, then they get that highest payment point value indicated there by $3 signs. If you have students who pass 243 additional assessment, but they're not enrolled in workforce training, of course they have the access to it, but they chose not to be enrolled in workforce training, then they still get a payment point, but it's at the lower level. And in fact, that lower level is the same level as an IELCIVICS student in Section 231 would get by passing a 231 additional assessment. Currently in the current grant award, I believe the numbers for the $1 sign was $104, $104, and I believe the higher one, the $3 sign, was $355. That can fluctuate each year depending on priorities that we have at the office. Okay, I think this is my last slide before we'll take some questions. Again, I mentioned the IELC report that's due April 30th.  It's really just a way for you to tell us what you did with your IELC program that year. So you'll be describing all the components, describing how it's contextual and concurrent, describing who your training provider is, who's providing the training, talking about some of the hours that are involved. Of course, you'll tell us what co-app you're using. And there is an outline of this available on the website that I will hopefully have time to show you. Okay, I know that was a lot of information and I see that we have some questions. So let's cover those and if you have any others, I'd be happy to entertain those. Okay, so Cheryl says, please clarify a little more on the IELC workforce training being available. We're starting a medical assistant program that will be available to any of our ESL students of the appropriate NRS level, but only have one student interested. Okay, Cheryl, why don't you come off mute and tell me a little bit more about your question there, because I'm not exactly sure what you're looking for. Well, I just want to clarify if one student is enough. Oh, I see. Well, I mean, it's not ideal. I'll tell you that, you know, we're definitely moving toward having more students involved in this program. That said, you have to start somewhere. So, you know, technically we say a percentage greater than zero. So if you have one, that's greater than zero. What I would say is if that's all you have, start with that and try to build it. Really, you know, think about that needs assessment and really question why you only have one, you know? Is this program really feasible for your students or are there some support you can provide to students? Well, our ESL population is small and the majority of them are level three or below. So they aren't, most of them are ready for an IET. We have a few students that are higher. So we're trying to, we're a new ESL program to less than a year and a half. So, yeah, I just want to make sure that one is enough because that's all we've got at this point. Yeah, yeah. I mean, like I said, you've got to start somewhere. One recommendation I would give you is to really think about that pathway idea, maybe in your medical assistant pathway or your health pathway or whatever you're going to call it. There are earlier courses in that that students could be enrolled in that sort of get them to the actual training course. Sometimes these are referred to as bridge courses. A bridge course itself wouldn't be, would probably wouldn't rise to workforce training, but it's certainly helpful to get these type of students ready for that, the true workforce training. So you might just want to develop the pathway even further and think about, could students be involved? Could these English language learners be involved with some of the earlier courses as well? Oh, that's an excellent idea. We do have a bridge medical terminology course. So that would be maybe I could get them interested in that. Exactly. Okay, thank you very much, appreciate it. Yeah, no, thank you for the question. Michael is asking, do there have to be two teachers if the instructor is qualified to teach both? No, if that teacher actually has those necessary credentials, we've seen that happen. It's, I think that's fairly rare, but it's possible. And it's perhaps not ideal. Also, there's just a great benefit to co-teaching that we see in the research. And so I would encourage that, but on the face of it, if they're qualified, they're qualified. That might be something you want to talk to your consultant about and just talk about the details. Let's see, Molly's asking, thanks for the info. What are the timelines to apply for the program? Is it a yearly process? Oh, that's a good question. It's part of, it's built into the RFA that we just completed. So when you wrote the RFA, you completed, I believe it was consideration 12 in that that spoke to IELCE and based on those responses, you were led into the grant, led into this section of the grant. And so in terms of adding it again, it's probably not gonna be available again until the next grant cycle, which would be four years from now. And if you're in it, if you're already in it, which I'm assuming you are since you're here, then you will, as long as you stay in good standing and you keep up the program, you can stay in it for the duration of the grant cycle. Last year, we did not sign up for 243. When do we have another chance to opt in on the 243 options? Yeah, so I, as I just said, it is part of the grant application. So it'd be the next grant cycle. Did I miss Colleen? If we have a teacher with an ESL credential and a CT credential, can they be the sole teacher of an IET? Yes. You Sally, the bridge training could qualify then as a career prep course. Yeah, if you mean by the workforce preparation, it could definitely fall under that, for sure. In the earlier example, medical terminology would be a great example of something that doesn't necessarily get somebody a job. It's, you know, that's not, you're not gonna get an industry recognized credential for a medical terminology course, but certainly a useful course to prepare somebody for a workforce training course. So yeah, for sure. Leticia asks, can workforce training be integrated with the adult education literacy course that covers the IELCE or does it have to be a separate course? So integrated with the course that covers the IELCE or does it have to be a separate course? So let me try to answer that. And if I don't answer correctly, you can unmute and tell me what you meant. But it's certainly okay for them to be separate courses, you know, it could be a separate course that a CTE course that you offer at your school, it could be offered at a different location. But to the first part of your question, it does also have to be integrated. I mean, this is integrated English literacy and civics education. So it definitely has to be integrated, that separate course, the teacher there and the ESL teacher should collaborate, should get together, should sit down and work out an integrated syllabus. The objectives and the goals of the course should be aligned. So I would say yes, and both of those need to be, it could be correct. Definitely needs to be integrated and it could also be a separate course. Let me know if that doesn't not answer your question I'm making reference to the Venn diagram where you have a dollar education, I don't see it in workforce training and it's separate bubble. And so can workforce training be incorporated into the ESL class, what I'm referring to? Yeah, there are lots of different program designs that we see and we're not especially prescriptive when it comes to program design. So long as those three components are there and it's, as mentioned earlier, it's sufficient intensity and quality and it leads to an industry-recognized credential, that model could certainly work. What I would recommend is send an email to your regional consultant and tell them what you're thinking about and we can give you more details. I'll also put a plug in right now, I was gonna mention this later, but Lori Howard has monthly EL Civics network meetings which are amazing places to opportunities to get more information and to kind of bounce these ideas off other folks. So the quick answer to your question is yes, that's possible, it just needs to be designed with all the IET requirements in mind. Corey, will we talk about the IELCE plan if we have time? Yeah, I can definitely show that. It might come into play there, thank you. Okay, I see one more from Michael. How is a program blessed as a proper IET with all the necessary elements included? I can propose that particular program that I offer meets all the check marks, but at what point in the process does someone take a look at it and say, yes, that's an IET? I like this question. This speaks a little bit to what Diana just mentioned. We do have what you might call kind of an annual checkup which is called the IELCE report. And as I mentioned, that's due at the end of April. And there you will give us a rundown of exactly what you're offering. And that's reviewed by staff. They're mostly CASA staff at this point who review those. And if there are issues with that, they may ask for clarification. And if the program is deemed inadequate, then sometimes the consultants get involved and then we have to look at is it actually a compliant program? In some of those cases, if it's not a compliant program, we have to move payment points around and things like that. So when we first started this program six years ago, we did have a process where we said, okay, propose what you wanna do and we will approve it. And then you move forward. Now that we're six years into this, we're moving away from that a little bit in the same way that we don't ask for you to submit plans for all your other courses and we have to stamp approvals on them. We're trusting a little bit more that folks know how this works, but we do have at least an annual check-in which is the IALC report. And then of course there's the FPM process. We really hope it doesn't come to that, but certainly when we review your school through federal program monitoring, we'll be looking closely at this program. And again, if it ever comes into a compliance issue, we may have to take some steps there. Again, hopefully it never comes to that. The most important thing is communication. So please send an email to your consultant with the ideas that you're working with. Involve your CASIS program specialist in this. They are all experts in this program as well at this point. And even though it may not be like a rubber stamp, yes, this is blessed. At least you'll know you're on the right track and that you're in compliance. I hope that answers the question. Thank you, Corey. I really appreciate it. Thank you. Of course. Okay, let me glance at my clock here too. So we started at 2.10. Okay, so we're doing fine here. If you have other questions, please drop them in the chat. I'd be happy to answer those. And like I said, the rest of this presentation is much less the regulations and more about the how. And in all honesty, I don't have all the answers when it comes to the how. Because I don't know your schools. I don't know your students the way you do. So what I'm gonna do is share some of these kind of models and structures that Lori developed, some guiding questions, and then we're gonna have a discussion about them. And at one point, I'm gonna put you into a breakout room even. So be prepared for that. Just what you want at the end of the day in this breakout room. So here's the basic improvement cycle. You're gonna review and evaluate what's done before by nature of being in this grant. That means that all of you should have a compliant program. I'll just leave it at that. Plan for changes in the current or the future programs. You know, obviously you're gonna try to improve the program. You implement that program. You fill out the IOC report, turn that in by the end of April. You'll get feedback on that based on the feedback. You go back and iterate the process. Here are a few steps that I think are very helpful and some questions to think about as you're developing this. So needs assessment, as we mentioned before, thinking about the true needs of your students and the opportunities in your area. So what are the priority jobs and training needs that exist in your area? How do you interact with your workforce development board with the American Job Center of California with your other title, we have a title partners, advisory groups.  How do you interact with those folks to answer that question? The next bullet, what training opportunities exist at your agency or with your partners? So, you know, once you've sort of decided what the need is in the community, then you think about what your capabilities are as a school. If you don't have those capabilities, maybe you go to partners. And then very, very important, you know, not necessarily in order, you know, what do your English learners want? You know, what do they need? Hopefully these three things can align and you can find something that fits your resources and also the needs of your students and the needs of your community. But having an intentional process where you identify these needs and opportunities is really a crucial first step. Next step, I mentioned this earlier about support services. This is not an easy program for many of your students. I'm sure you know this already. And so the more support we can give to them, the better. So what kind of support services do your English language learners need so they can access and succeed in the training? Do you provide orientations, counseling, navigation? Are you doing everything you can to eliminate barriers like childcare and transportation, time constraints, et cetera? And then importantly, that last bullet, what mechanisms do you have in place to gather this data? Again, getting back to the needs assessment a little bit. So again, a very important step. And then the last step we're gonna look at in this section is actually facilitating that program coordination. How are you continuing to involve all of the stakeholders, the CTE teacher, the ESL teacher, all the counselors, the navigators, the support staff, other partners in your area to work together to improve the IELC program. That single set of learning objectives. Again, that's essential, that at the very minimum you've got this set of objectives that are integrated. And again, as I mentioned a couple of times, I would take that even a step further and have a syllabus that's created that students can read and that you go over with students. So they understand the benefits of this program and how it's integrated. Curriculum, including co-apps and additional assessments. Of course, you're always gonna think about those. Gonna talk about your instruction, support services, the schedule classes, the recruitment and others. Okay, so these are just guiding questions to kind of get you thinking about this. Again, I really view you as the expert in this area. And so we're gonna take just 10 minutes here to have a quick breakout where you can begin talking about these. Those 30% or 40% of you who said you that you were a little more advanced with IELCE, maybe you could help those of us who aren't as advanced in the implementation phase as you're meeting with the group. So we're gonna put you in a breakout group and what I want you to do is talk about these first three steps. And we'll give you the questions to use. But here's the basic rules. Introduce yourself very briefly because we don't have a lot of time. Perhaps identify somebody who's gonna help facilitate the conversation. Maybe you have somebody who's recording and reporting. We're gonna have a pretty short reporting session but I'd love for that. You're gonna get three questions. They'll be in the chat. They're already in the chat. And I'll also put them on the slide, I hope. And then when we come back, I'm gonna ask you to put some answers in the chat or unmute and talk about what you found. Before we move to the activities, does anybody have any questions about the activity? You know what I should do? I should show you the questions. So here are the three questions. These align directly with those steps. I just talked to you about what opportunities for training and employment exist in your area? How do those match with the needs of your students? What kind of support do your learners need? And how will you continue to involve everybody in the development and the implementation of your program? So three kind of essential questions that I'd love to give you some time to talk about. Okay, welcome back, everybody. Let's see, so we're running out of time a little bit but I would love to hear how your discussion went. So, you know, feel free to put stuff in the chat but I would love it even more if somebody would volunteer to unmute and just tell us a little bit about your discussion. Janice, are you waving to- I'm waving. Okay, go ahead. Hi, I'm Janice Vera. I'm new to my role at Solano as the new consortium manager and so that's why I'm enjoying the NAR training this week. I was joined in my breakout room by Denise from Porterville, Rosa from Riverside Community College and Mona from Oxnard. And what we talked about a little bit was the transition specialist role whether it's handled by a faculty member or by a specialized counselor and the importance of being able to reach out to the students. And some of them, one of the gals mentioned that they do it with surveys or maybe they actually do it one-on-one in like a face-to-face setting between the faculty and the ESL student to learn more about what their needs are. They looked at different ways to, let's see what else she talked about, a job developer working with the students to once they find the right person then they can commit with the students and figure out how they can pick industries so that they can offer co-apps that meet the students' needs. And that whole process was, well, the kind of interesting to share. We talked about the livable wage because it's good thing to train a student for a job but other consortium managers that I've talked to have said that they actually put a stake in the ground and said 70 grand. We want our students to come out of here on track for making that much money because there are lots of jobs out there but we want them to have that living wage that gives them the security and the confidence to go forward. So those are some of the latest things. Excellent, thank you so much. I love those. Yeah, the idea of getting data from our students in terms of what their needs are, so powerful. So thank you. Anybody else? We have maybe time for just one more if somebody else wants to weigh in on what you discussed. Elsa, I see you have my camera, go ahead. Yeah, sir. So at Moody Valley Community School last spring we participated with the IET program and with Pharmacy and so this year we've implemented a co-teacher that goes in the last two hours of the course and so she provides support not only to the English ESL students or Yale students, the whole class really. And then we provide tutoring for the students in math because they do have a pharmacy technician that's require a higher math and sometimes the students struggle with that. Another thing that we do is we do give them a pre-assessment to see what level of math they're at because if they don't have the basic skills in math then we provide tutoring for them to get their basic skills too so that they can be successful in the pharmacy. Also with the IELCE program this year to improve it, to make it a little bit, well improvement, we added in the past because I'm fairly new to principal at this school and so in the past they would come three days for the ESL and then two of those days they would come back for the computer portion of it. And so this year they come to the class it's like two hours ESL and then an hour of computer so it's all integrated they don't have to go and come back and it's not separated. So the students are really liking it because they don't, it's like, okay we're gonna do two hours of English and then, okay, grab your computers let's do the computer portion of it. And then we partner with AUMT to bring in EKG, phlebotomy and medical assisting classes. And if those students need assistance they can come and participate with the tutoring with us also they're not enrolled with us they're enrolled with AUMT but they're welcome to come and participate with our tutoring. That's fantastic, thank you, thank you for sharing that. Your earlier comments about the support the support teacher reminded me of Suzanne's comment in the chat that I should remember to address. So Suzanne was asking about having a partnership with the local community college and providing career pathways for students we have support teachers for each pathway to attend class with students provide support classes each week to help students access the curriculum is this IELCE even if not all students are ESL. So it definitely can be. I think the as Elsa was saying it's definitely a model that can work it really comes down to the details and it comes down to again the intensity and quality. So I get a little worried when I hear support teacher like what does that mean? Are they ESL students? Are they getting ESL instruction? If that, you know, if you can sort of make the case for the fact that these are really that there's enough intensity and quality in the instruction that they're getting and that they are ESL students then something like that can absolutely work. Elsa, did you have a follow up? Oh, okay, sorry. So I hope that answers your question Suzanne and then Matt also said IET is really the I best model developed in Washington years back. That is true. However, this model is difficult to enact in California community colleges because of funding issues. I'd love to hear from community college folks on how they are doing it doing IT. Absolutely. And again, I put another pitch in in addition to what Diane was saying about the about the resources from CalPro. I'll put a pitch in for Laurie Howard's monthly networking meetings where you can hear from folks directly who are making it work. But great comment there. Okay, well I wish we could just keep the conversation going but we do need to wrap up. So let me just go to the last section here very quickly and got a lot of things on my screen here. So step four is evaluating the program. How do you get feedback about the success of your program? How use the ILC report review? How do you define success? What went well? What would you change? How do you plan for continuous improvement? How will you document progress? Plan next steps? Share success and future plans? So we only have a couple of minutes here. So what I'd love you to do is in the chat with regard to this last idea of evaluating the program, just pick one of these. Just pick one of these questions that kind of speaks to you. And when you put in the chat put the number by your answer. So if you're gonna answer one, put a one. If you're gonna answer four, put a four, et cetera. So for example, how will you get feedback about your ILC program? How will you document progress? How will you define success? How will you share the story? Thank you, Maria. Start the ball rolling here. Surveys, good. I'm assuming that's for number one. Get feedback through surveys, thank you. Paul's saying he'll define success by defining the number of students completing the program and gaining employment. Love it. Absolutely. I'm glad you added the gaining employment. That's really crucial there. Elsa says the number of students making learning gains. Yes. Brian adds any chance to hear from the student is great. Yes. Learning gains, yes. Okay, well, in the interest of time thank you for that feedback but let's just jump ahead to our last slide. So again, this is just reminding you about that cycle. Remember to keep it iterative. We talk about continuous improvement and the same thing here. We want to continuously improve. Okay, exit poll. I do want to know how you feel after the session if the needle has moved at all. So just very quickly. Not going to have this open for very long but how do you feel, how ready do you feel to work on this cycle? More than ready, ready. Getting there and thinking about it, need to learn more. This is helpful information for us. So thank you for completing this. Getting there is good. I'm gonna share the results. I think it's most everybody's in there. All right, so getting there is 55% ready. Wow, we've got ready and more than ready. Almost a quarter of you, excellent. And then 20% thinking about it, need to learn more. That is absolutely okay. The last slide I have for you is talking about how you can learn more. Lots of resources. There's a whole document in the canvas around OAR that will give you links to all of these. If I had more time, I would take you to the CASUS website because there's just a wealth of information there. So I encourage you to go to the ILCE page on the CASUS website where you'll find webinars and documents and a YouTube channel. There's a whole YouTube channel dedicated to this with trainings. So lots and lots of information. As Diana mentioned, there's training for OTAN and CalPro as well, so please take advantage of those. And then here's my information and I included Lori's too, just because she's so integral to this work. She's the specialist at CASUS on this program. So if you have questions, feel free to reach out to me or to her. And as we've said multiple times, reach out to your CDE consultant and your CASUS specialist because they always wanna know what you're working on and would love to talk it through.


Live from Las Vegas, Nevada, it's theCUBE covering EMC World 2015, brought to you by EMC, Brocade and VCE. Welcome back everyone, Jeff Frick here with theCUBE. We are live from EMC World 2015. This is our third day of continuous coverage. We have two cubes running back to back. We have a ton of great guests, a ton of great content. Hopefully you're enjoying it out there. Getting a sense of what's going on here at the show as we get our tech athletes, as we like to say, to come on inside theCUBE and share some of their knowledge and insight. So we're excited for this next segment. We've got John Cardente, Frequent Cube alumni. Good to have you back, John. Distinguished engineer, Corporate CTO Office at EMC. We always like distinguished engineers and fellows too. We get those too sometimes. Craig Steele, GM and Senior Director, Global Strategic Alliance from Pivotal. Awesome. So welcome. First time? Yes, it's my first time. Excellent. So Strategic Alliance, I've got two companies here. Talk a little bit about the relationship between Pivotal and EMC. Certainly. So Pivotal is the majority ownership is held by EMC. Part of the Federation. Part of the Federation. So we try and stay within that Federation framework and our go-to-market structure and some of the engineering campaigns that we have in building out technology and solutions together. And so far it's been working pretty well. Customers seem to gravitate towards this sort of ease of use. Right. And, you know, at the end of the day, a one-throat to choke model is always helpful as well. So yeah, we're well on the path from a strategic alliance perspective. Yeah.  Has the Federation worked in terms of differences than kind of the traditional alliance that you might have with other companies that aren't part of the Federation? Yeah. So having Pivotal as part of the Federation really enables us to work very closely on building and delivering some complete solutions to customers, right? So for the Federated Business Data Lake, we very much take the view that it's about data analytics and applications. You need to be able to do all three. Right. And so EMCII has great infrastructure products. With VMware, we can build flexible and virtualized environments. And with Pivotal, we have the analytics. And most importantly, the application development capability. Right. And talk about the difference really from, you talk about data-driven applications. How has that a different approach to maybe traditional application development and kind of use in the past? Yeah. So, you know, if you look at traditional business intelligence, having applications to do, but dashboards and reporting and things like that are, you know, an important part of the things. But with big data, what you're really trying to do is deliver value on insights, right? So it's all about looking at the data in new ways, finding new relationships, and then quickly monetizing that. And so in order to do that, you need the applications to deliver that information, either to customers or folks in the field. And so you really need a way to develop applications quickly so that you can take that insight and deliver it in a way where it can be consumed. Right. I would just dovetail on that and say that, you know, that's the entire business model for Pivotal, and it really relates well to the Federation Business Data Lake model. Our go-to market is really focused on building new agile applications. Those applications eventually have data associated with them. That data can be analyzed, and then you leverage that analytics to recreate or iterate on the application and then build more use cases into the environment. The data lake solves a big problem for that, for customers there. And the way that the applications are developed, leveraging technologies like Pivotal Cloud Foundry really speeds up that process in new and interesting ways. If you think about it, you know, the process for developing applications prior to this has really been, you need hardware, you need middleware, you need operating system, you need messaging and load balancing. Pivotal Cloud Foundry really takes all of that sort of time-consuming issues out of play and makes it really easy to build the application in the first place, but also scale and manage it in that process. Right. Now you said the interesting thing, that as the data comes in and you start to get insight from the application, you're actually iterating on the application as really thinking about it instead of building a new application. So that is a very different type of approach. Yeah, we believe it is. So if you look at the, you know, sort of traditional application development, it's very, very slow. Now if you look at the new applications that are kind of following the message of software eating the world, take an example like Uber, it's the biggest transportation company in the world, and they don't own a single car. Right, right. Airbnb, you know, number one hospitality company in the world, they don't own any property. It's really about the end-user experience. So once you develop an application that's around that end-user experience, and then you analyze that data, you're then able to come up with new use cases, new value that you can drive, and you have to very quickly be able to iterate that application to go out and monetize those capabilities. Right. So that's what makes it interesting. Yeah, and another important facet of this too is that the ability to create mobile applications quickly and easily enables gathering more data from the field, from customers or from people out there making observations about how people are using products and things like that. So it's not only delivering insights to the edge, but it's collecting data from the edge as well. That's really important. Right, and then two, the other thing that's so different, right, is now integrating other data besides your data and really being able to bring a much broader, more disparate data set into play. How are you seeing that kind of working out in the field with real customers? Well, we're seeing it in a lot of different places, right, because customers are realizing that not only do they have hidden treasures in their own data, but when they want to do things like sentiment analysis or lead generation or things like that, they need to pull in some external resources, right? A lot of it's messy and unstructured, and so it needs these new tools in order to be able to capture it. Right. So we're seeing a lot of potential capability that we're building inside the Federated Business Data Lake is to be able to take both the data you have and the data that you have access to and use it and analyze it to get those insights. Right. Yeah, I would say that's part of the structure of a pre-engineered capability when you look at something like a Federation Business Data Lake. All that capability is built into the solution day one. And it's built in such a way that you can, again, iterate on the application, but you also might be finding, you know, different information that you can bring in from outside your traditional organization that you can leverage for business value and marry the two. You leverage technologies like, let's say Gem XD as an example, as something that can also be sort of a modern day messaging, right? So you can take sentiment information from Twitter and marry that with weather analysis, if you will, in real time and come up with ways that affect outcomes. Right. So the other funny thing I think that's interesting here is the whole concept of agile and software development and agile and iterating your applications. But now it's really driven to being an agile business in terms of the business models, the business process. You know, using that exhaust, using some of that new stuff that you didn't even know that you had or you're seeing it in a different way to now actually really bring an agile kind of methodology process mindset into your actual way you run your business. Yeah. Yeah. You know, I view big data in a lot of ways as the long tail brought to analytics and that, you know, the relationships and the behaviors that are easy to monetize, we've done that.  Because they might go away. Or, you know, there might be three of them. And so what you really need is an infrastructure that you can do, you can capture many of those, get the volume of the tail with the right economics so that we actually make money. And that's really what big data is changing, right? It's now it's possible to do that. We couldn't do that before. You couldn't make money going after those, you know, transient or very rare insights, right? Right, right. It's like the old, you know, early days of the electronic training where the guys that, you know, could get ahead of the curve and see those little opportunities that were so transient and jump on them were way ahead of everybody else. Absolutely. It's really true. And the other thing that we sort of looked at when we built out this model was understanding that if you have an application development process and a platform on top of that. And so as an example, Pivotal Cloud Foundry runs on top of the sphere, runs on top of Amazon Web Services. It runs on top of EHC and other, you know, vendor supplied infrastructure. So the good news is it's not like the old days where you're kind of locked in. I mean there are several customers that are, you know, paying a tax to IBM to continuously run their cold ball. And right now we're seeing customers that are writing application in the Amazon Cloud that, you know, down the line because they're so intertwined in that Amazon that they're going to be kind of locked into that. To that infrastructure. So Pivotal Cloud Foundry allows that application to move to different, you know, different infrastructures, public, private, and then leverage the best technology for the use case, which we really believe we've cracked the code with in terms of the Federation business data. Yeah, because it's really about, at the end of the day, it's about the workload and the application of that workload at that moment of time. Whether it's test dev early days or PLC trial or massive scale in production and being able to move that wherever it needs to be for whatever the appropriate level of security and access and speed, et cetera, et cetera. I want to shift gears a little bit and talk about it, you know, real time. It's like about real time. What does real time mean to you in this kind of fast paced agile development and business model world in which we live? Yeah, I would follow, you know, I follow our CEO. So the message that we hear there is really about capturing things or people in the act of doing something and be able to affect the outcome in real time. So if you start looking at it that way, you know, the talk track is everything from your socks to jet engines will have some sort of sensor data or information on it. And, you know, they're really smart people, data scientists included that can figure out how to monetize those different real time capabilities. And then again, it goes back to the application. How do you very successfully go and attach that application and user experience to the 1.5 billion cell phones that are out there to then monetize that and bring value back to your business. So real time is really about capturing that information on whether it's a machine or a person and then affecting the outcome in actual real time. It's very exciting stuff. The example I like to use is that, you know, in the dawn of time, you know, you knew that men bought razor blades. It was a pretty easy thing to figure out, right? And then you did a little bit more work and you figured out, you know, maybe men between 30 and 40 like triple blades, right? So a little bit of market segmentation, a little bit of digging, kind of your traditional BIDW. For me, real time in big data is Craig tweeted this morning that he forgot his razor blade at home. So let's send him a coupon so he can go buy a razor blade, you know, one of ours, right? And so now maybe that's the beginning of a relationship, right? If you think about what's necessary to monetize that event, you've got to eat the Twitter fire hose. You've got to find that tweet that's relevant to you. And you've got to be able to quickly send a coupon, right? And perhaps go through a lift model to figure out whether or not that's even worth sending to someone like him. That's real time. That's acting on an insight before he goes down to the lobby and buys a razor blade. Right, right, right. And even then, that's still kind of, that's still kind of after the fact, right? It's event driven because you, because you tweet it. Because the other thing that people love to talk about kind of the dream of big data is moving from that model to really predictive, right? That's right. And then beyond that, prescriptive, which I guess in your instance would be, your phone indicated that you rushed out of the house, you know, from the time you picked it up when the alarm clock went out to the time you were in the Uber was a very short period. Good chance maybe you forgot the Dopp kit in the, in the bathroom. So maybe I should send you that razor blade coupon before you even land because I could put those things together. Amazing how much data, I'm always fascinated by people's concern of, you know, kind of big brothery types of stuff when we're all packing this little GPS with an accelerometer. That's right. That knows exactly where we are, how fast we're moving all the time. So it's really an interesting feature and how that starts to be delivered in terms of new products and services on the positive side versus some of the things people worry about. Yeah. There are new types of communities being developed around different things that many people don't even think of. I'll give you an example. So Rip Curl is a company that has a legacy business of building wetsuits for surfers. Okay. They came out with a watch. That's a GPS watch that has geospatial sensors in it that tells you how many miles you paddled, how many waves you caught, what your longest ride was. Right, right. All these different statistics and data in real time and there are communities being built up for others that use this technology to compete in real time in the water for those different statistics and then share with their friends globally. So it's really amazing that some of the things that's going on with in terms of data-driven outcomes and people leveraging that for their personal life as well as business. Right. Well, it's funny you mentioned the socks because down the street in downtown Vegas this week is the collision comp. I don't know if you've been paying attention. We're very busy here in the sea world, obviously. But I'm getting a lot of stuff from people reaching out to see if we're going to be there. And one of the companies is an athletic tech sock company where it's got sensors in the socks. It's got receivers on the sensors. I'm sure it's got an application for your iPhone. They sent me like the whole pricing package and get replacement socks, replacement sensors. So it's happening today and it's happening now and it'll be interesting to see from kind of a psychological impact when the feedback starts to come. Now people know that I walked 10,000 steps or not. No one ever knew that. It was an arbitrary thing, but now suddenly, I'm only at 8,000. I got to go walk down to the block a couple of times. So it is interesting how now these machines and the feedback loop can really start to influence behavior. Yeah. And I think to draw that forward, the internet of people is kind of an interesting thing because we all have these devices, right? So a good example of going from the general to the specific is we could take a lot of telemetry and say, well, people who walk 10,000 steps a day are generally more healthy, right? So it's a general recommendation. But more specific is I looked at your calendar and I noticed that you have to be from this building and this building.  That's the kind of prescriptive kinds of things that you can do with analytics that is only enabled though if you can process that stuff really quickly. One last thing, one last point I'd like to make. So we're seeing this sort of influx of new capabilities, new requirements from different customers on a daily basis. Folks that wouldn't normally, you would see like in a Pivotal Labs environment, CEOs of major corporations coming in to see how Pivotal Labs is building these applications that can be leveraged in a data lake environment. It's interesting to watch because the market is still nascent. Hadoop is kind of, I think people have figured out what Hadoop is and what you can leverage it for but it's really the analytics and the real-time analytics now and going forward that are going to drive real business value for many customers. It's great to see you. Our much shorter answer to the real-time question is in time to do something about it. Yes. Can you get in time to do something about it?  Great.  What are you excited about in the next six months? What are you working on? What should we be looking for? So I think it's a continuation of sort of the same progress that we've had over the last six months where we're seeing a significant amount of interest and adoption in the technologies that we're putting forward as a federation. I'm really excited about some of the additional use cases that we'll be able to add to the Federation Business Data Lake environment and I think the partnership with EMC, VMware, and Pivotal is stronger than ever and will continue to be that way as long as customers still have the demand for what it is we're creating. So really excited about the future. Excellent. Well, thanks for stopping by, Craig. John, always good to see you. I'm sure we'll see you again. I'm Jeff Frick. You're watching The Cube. We're at EMC World 2015. Three days of wall-to-wall coverage. We're in day three. We're on the backside of the last day but we're excited. We're standing at the end and we'll be right back with our next guest after this short break.


Welcome back to Kids Fun Science. My name is Ken. Today's experiment is the deflated balloon. As always, adult supervision is required. What you need for this experiment is a two-liter bottle, water, and a balloon. To set up this experiment, you start off filling a two-liter bottle of water all the way to the brim so all the way to the top and then place a balloon over the top of it not filled just just like that deflated and then you're going to take a nail and you're going to poke a hole in the bottom of the two-liter bottle some water will come out so you have a dish underneath it and then start giving a pull up and down on the balloon a number of times and it'll create a little bit of pressure and then suddenly it will invert into the bottle so just keep pulling it up and down and there it goes just like that. When the deflated balloon is placed over the bottle the air pressure acting on it pushes into the bottle. This is because any water that leads through the hole creates a vacuum in the space above the water. Air can thrust enter the space that the water formerly occupied. Once the air pressure is greater on the outside of the bottle than the inside of the bottle the air is pushed into the bottle not sucked. However, there was always a little bit of air still left in the bottle because you cannot remove all the air. Initially, very little water will flow out of the hole since it's only a small amount of air in the space in the water. By pulling up on the balloon the water will flow immediately increased since the air in the bottle will expand to fill the greater volume causing the excerpt of less pressure. When the balloon is allowed to snap back into the bottle it momentarily compresses the air in the bottle causing it to exert more pressure and force the water out of the hole. When air is blown into the inverted balloon it serves to compress the air already in the bottle causing it to exert greater pressure and force the water out of the hole. The same effect can be observed by poking at the balloon with your finger. If the bottle itself is squeezed the balloon will inflate and be forced out of the bottle. The water also will be forced out of the hole. This is another way to compress the air in the bottle by reducing its volume. The air exerts greater pressure causing the balloon to inflate the water is also forced out of the hole. This experiment serves as a good validation of Boyle's Law which states as the volume of gas decreases if pressure increases and vice versa. I hope you enjoyed this experiment remember to click thumbs up and to subscribe. Thanks for watching!


Hamilton and I'm a K-12 art instructor. I'm going to talk to you about how I use rain sticks across grade levels kindergarten through seniors. I started using rain sticks originally as a classroom management tool. This was my first rain stick and I use it as my quiet signal. So whenever I'm transitioning between steps in projects with elementary art students, I use my rain stick rather than flipping my lids, clapping my hands, doing a hand stand, whatever it is you have to do to get your kids to be quiet and listen. Then I made this huge rain stick out of a tube that we received some broom handles in and I was inspired by the work of Henri Matisse, a French artist who used paper cutouts to create images with organic shapes. So I started creating art history inspired rain sticks which led me to use rain sticks as a high school art history assignment. So using defiant poster tubes, even toilet paper tubes, we get tubes from a t-shirt company in our town. We created images on our rain sticks inspired by different artists and art movements. This rain stick is inspired by the work of Roy Lichtenstein and it was created by a high school junior. This rain stick was inspired by the work of Rothko and it was created by a sophomore. This rain stick was inspired by the work of Miro and it was created by a junior. This is our Van Gogh rain stick created by a junior as well. As you could hear, each rain stick has a unique sound depending on nail placement and whatever materials you put inside to rattle against the nails. So once the students completed their design on the outside of their rain sticks, they strategically put nails through the sides and then filled them with beads of different sizes and shapes depending on the sound they wanted to get. The smaller the item, the higher the sound. The larger the item, the lower the sound. Some history on the rain stick. Historians don't really know for sure where it originated but there have been rain sticks found made out of tubes from dried cactus in Chile in South America.


I have a talk coming up, we're running a little bit behind, so I'm not going to introduce this man too long, except to say that he is the greatest Canadian who ever lived. He was actually born a Mountie in Canada. That's not a thing that happens very often when you're born with your horse and your sword. His name is Adrian, he's from Seattle. He works for a company called WellTalk. Let's see if he talks well. Round of applause please. Hi, I'm sorry about that introduction. I'm actually from Seattle, not Canada, but real close. That's pretty good. Oh yeah, go Canadian. So my talk is about deployment Nirvana and it's kind of some of the issues, some of the challenges and some of the stuff that we learned at a company called Moz where I built a lot of deployment infrastructure for our SOA app. Conveniently enough, everybody this afternoon has been talking about services and why they are amazing and they're terrible at the same time. So it's been really a great, great lead-in. So first of all, the web definitely as we build it is changing, right? We're moving more and more from these big, giant monolithic apps and we're starting to split them up into services. And so this actually reflects back on some of the stuff that we used to do in like the 90s with a lot of the ways that we would factor out large enterprise apps. But now we're starting to do it and we're breaking it into these little small apps that communicate over HTTP or NSQ or any sort of message broker or a bunch of different ways. And so we're starting to split them out onto this. And so it's actually not for scaling performance wise, right? So a lot of times we'll talk about like, oh shit, we've got to break out this monolith because it doesn't scale or it's too slow or any of these reasons. Well, it turns out that stuff like this is really, really, really cheap compared to stuff like this.  I mean, you know, we all know that we can get a baby in one month if you just throw nine women at the problem, right? Like it's the nature of the system that we work in that as teams grow, your productivity doesn't necessarily grow alongside. So again, with this, with our movement to SOA, microservices, whatever the current term that we want to use is, what's happening is what used to be a very complicated app, very complicated piece of software that was nice and easy to deploy onto your Commodore 64, it's now starting to become all of these individual little apps and we have to deploy them separately. We have to coordinate them separately. We have to worry about how they intercommunicate. We've got all of this extra stuff, this extra cognitive load that we have to understand. And so the problem that I attacked at Moz and that we're going to be talking a little bit about today and I'm going to be tackling for a while in the future is how do we do all of this? How do we deploy 40 services and how do we do it in a way that makes life easy for developers? As an engineer, I don't want to think about what happens when John bumps a service and blows up the world, because why should I have to think about that? We've got versioning, we've got tasks, we've got ways that we can solve this problem. So hi, I'm Adrian, that's not it, hold on. That's closer, but also there's my space reference, Jonan, you're welcome. Hi, I'm Adrian. So let's see here. So I was at Stride for a while, which is a startup that I had with a few guys. Then I went to Moz, and that's actually where I learned a lot of stuff that I'm going to be talking about now. Now I'm at a company called WellTalk. Basically I am writing code and leading teams and doing all of that, all of that stuff that we do and get paid for and we have a pretty darn good time doing it, actually. So we're going to rewind a little bit to when I was at Moz, and this was about a year and a half, two years ago. So we wanted to build a software. It was this glorious, wonderful, green field app, right? So we all sat down and we actually had some pretty smart folks on the team. We had a couple of ex-Google guys. We had a couple of extremely brilliant, fresh out of college folks. I was the anchor just kind of dragging everybody down, but I could help people laugh, so that was good. So we basically looked at the problem and we built this tool called Moz Local. So Moz Local was primarily and is still primarily, hopefully, I hope it still runs. It's primarily about transforming API data, right? So the core of the problem and the core of the app was that let's say that I'm a small business owner and I want to make sure that my small business listing is up to date and correct across, say, Yelp, Superpages, Foursquare, I guess it's now Swarm, Facebook, all of that stuff. And so we had to work with all of these APIs, and it turns out that a lot of the APIs we were streaming and we had a lot of concurrent connections running. So Node.js actually looked like a pretty convenient, good way for us to build it. Because we were doing a ton of parallel IO because there wasn't really a ton of business logic involved. It was mostly just taking these external APIs, manipulate them, normalizing them. We decided to do it in Node. And because of that, also, we decided to do it as a large pile of independent services. Part of this was just because it made sense from the specific app and part of it was team scaling. So when we thought about why we were going to do a service-oriented app or a microservice app versus not, we thought a lot about team scaling versus performance scaling. By breaking it out into separate services, we were able to minimize the cognitive load for anybody. I didn't have to understand the entire app. I didn't have to understand how our caching layer worked. All I had to know is that as a consumer, as a user of it, I would just interact with it with the published, well defined API contracts that we would give internally. It's very similar to stuff where we would have internal APIs where things would be well defined, what's public, what's private, all of this stuff. We don't really do it very well anymore. A lot of times we can just hack and slash out some code. So it was a lot about minimizing the cognitive load, and it was also about composability. So I touched a little bit. We had a caching layer, and we had a lot of interactions where we would work with an API and think of it as a stream so we could compose together a couple of different services. So all this stuff is looking pretty good for us right now. Well, and then we kind of thought about why not service-oriented architecture. So there were some complex interdependencies in that services relied on each other. We had a service that encapsulated our data persistence layer, which underneath it, there was Cassandra, there was Postgres, there was a little MySQL, there was some Redis. We don't really want to talk about it, which was great as a developer, as an engineer. I didn't have to think about any of that stuff. It was just taken care of for me by this service, but it's very complicated. We had some reliability issues, which I'm going to touch on in a second. Debugging it was nigh impossible. I see an exception, and I have to unwind the entire service stack of how this request bounced through 15 services. That was really painful. The tool chains were non-existent. So this was one of those things, like Forman is a tool now that is very good and is helping us with some of the stuff, but it's still hard. It's still hard to coordinate 15-odd different services or 20 or five or however many we're working with. It's hard to do this in a development environment and think about it well. Integration testing was one that was really painful, and the really subtle foreshadowing deployment was hard. So we made the call, and we decided to go with service-oriented architecture. So we got to work. This was basically our look on all of our faces as we started to get further and further into the madness. And you can get that photo real quick then, because three, two, one. OK. So first of all, we were using a tool called Circus, which I don't know if anybody here is familiar with it. It's basically a process manager that allows us to run end processes or services or what have you underneath it. Don't ask why we're using it instead of in a D or any of the upstart, run it, any of the ways to do it. But this was the tool we decided to use. We also had this thing called Collateral Flapping, which I'll touch on in just a second. Deploy is resulted in downtime and service-level versioning. Again, one of those subtle, subtle foreshadowing things. So Circus was our first problem. And I'm going to call it problem zero, because it wasn't even a problem for us. Before we actually even went live, before we launched, we just killed it and replaced it.  So the next one was we had the collateral flapping issue. So because we sucked at Node, we didn't know what we were doing. Because domains were super confusing, for us, again, point one, we didn't know what the heck we were doing. Basically, what would happen is, say we've got five concurrent requests coming in through this Node service and one of them crashes, well, we weren't doing a really good job at catching those exceptions. So all five of them would just die. So a lot of people would get 500s or broken sockets or any of those really painful things. Well, so what we did is we solved it. We built a service registry and an HTTP proxy, the simplest way, of course, to solve a crashing problem. We built a service registry and an HTTP proxy, and we ran multiple instances of a given backend. We were able to do some kind of cool stuff where if we haven't sent any bytes back down on the wire and we're not that far along on a timeout, if one backend goes down, we just transparently use another one. The end user doesn't even have to know that they killed a backend. They don't know. So the second one was we had deployment downtime. Well, we've got a port registry now. So we fire up a new backend. It'll register itself with our port registry. And then we basically just cleanly repoll the old backends and the HTTP proxy just cuts traffic on over. So now we've got this interesting pile of stuff, these three separate home rolled loosely coupled tools called DMV. That was our proxy because we all know how efficient the DMV is at routing requests. We had Cport, which was our port registry, which we eventually replaced with a tool called Portland. Portland, of course, was some fair trade, free range port registry services. We had a lot of, the read me was literally nothing, just put a bunch of hipster jokes. So that was pretty fun. And then Viaduct, Viaduct was our new service registry and our new process manager, because we all know that the Viaduct's never going to go down. So the laugh there for everybody doesn't know is there's a Viaduct in Seattle that it's been going to go down in the next big earthquake that we have for like 10 years, 20 years, 30 years. So now we're trying to cut a new tunnel underneath it. And that's also gone really bad. So we're like $5 billion now to try and cut a tunnel that's going to save everybody in the next earthquake, but it'll probably just flood. So that was Viaduct. So it was the greatest thing ever, right? We've got these great, awesome tools. They shield us from ourselves when we write bad software and it crashes. We've got zero downtime deploys. We've got all kinds of controls. So we can do some really interesting stuff now that we run the code for our routing layer. We can do some fun stuff around versioning. And we can do some fun stuff around cutting over traffic to different users and all kinds of stuff. Well, it's also the worst thing in the world. So, client tooling, we didn't really have any. Our deploys basically revolved around our intern. He was really good at fast at typing out a pile of commands that sometimes changed a little bit every once in a while, so nobody really understood it. It was bad. We fixed that. So the client tooling was really bad. Development was really, really bad. Again, because we had to run 10 services and you have to run say, oh, I don't know, like Postgres and Redis and another Redis on a different port and you've got to make sure that your Cassandra instance actually has two services, otherwise it's not going to fire consensus and things aren't actually going to get stored. Ton of pain. So that was kind of lame. We made it better. It got better. But the really interesting one though is that inter-service versioning, which is when you've got service A and service B and service C, the versioning between them was kind of unsolved. So service versioning, which was actually touched on the last one with Pliny, it's pretty much a solved problem or so we think. We've got all kinds of ways that we can call for a version on a service. We can do it in the URL, we can do it as a query program, we can do subdomains, we can do what Foursquare does, which is they have a timestamp, which is actually really cool. You include the timestamp of when you want this request, the time of the code that you want this request to run against. And so they like sunset stuff after a few months. It's pretty darn cool. So the problem is is that as an engineer, I have to start thinking about this, right? So I have to think both writing and consuming a service about versioning. We're never gonna be able to get rid of both, but we might be able to get rid of one. So whenever I make a commit, I get back a shot, assuming I'm using Git or any other good version control system, I get back some reference to this code at a point in time, which is really similar to what a version is if not exactly what a version is. So we thought about this because we were starting to have issues with all of our different services and deployments and what versions we're talking to what. So we thought about like, is there anything that we can do here? Like, are we able to actually use Git shots as our versions? And so it turns out we can, partially because of the tooling that we had up ahead of us, and partially just because we built a little bit more stuff. So we've got this HTTP proxy. It's our own custom code. We can do all kinds of stuff with it. So basically what we do is we extend it to route to specific Git shots on a service, and we consider that as a version, right? So we include it as an HTTP header or something else. We're also having service intercommunication via NSQ, which is our message queue. So again, we would include the shot, and we would say, hey look, I wanna talk to the service and let's call this service A at this Git shot. And so we'd actually keep them running long enough and we'd have, we were measuring traffic to individual back-ins. So we would know when traffic got cut away from it and we'd just reap that back-in. And so we also can do things with the domains, right? So we can say that production goes to this Git shot, right, and we can say that staging goes to this Git shot. So we've got all of a sudden getting a lot of power with our versioning. As an engineer, I don't have to think about versioning. I don't have to write versioning into any services.  There's lots of great libraries to do it, but we're at this point right now in engineering that we can build amazing tools and amazing abstractions. So any way that we can make our lives and our colleagues' lives easier, we should probably do it. So there are obviously some advantages and some disadvantages here, right? So as an engineer, I don't have to really think about or build versioning. That's good, right? I know that the software that I'm writing every time I make a commit, I have a version given back to me by the tools that I'm already using. Another one that I can do is we can test multiple releases simultaneously, right? So branches have, or, you know, commits on branches have shaws. So we can, basically, if we have control over the routing layer and where our requests are going when, we can split them off.  I can get back a shaw. I can throw it up into our hosting layer and then I can cut some traffic over to it on a custom domain or say I can give out that shaw to other engineers or marketing or anyone inside or outside of the company and it just rattles through the routing layer and finds its way to my new services. There are some challenges, obviously. Outside of building all of the tooling, there are some restrictions that we had with the Git shaw-based route or versioning. So our deployed services have to be reflective. So what that means is that a service has to know what its shaw is. And a lot of times, when you release code, it's not, right? A lot of times we'll do some sort of tar ball, we'll do some sort of archive, we'll use Capistrano, we'll use any of a number of different deployment methods and a lot of them aren't gonna give us any information on what the shaw is or what the snapshot in time is. It's fine, we can tool up around that as part of the deployment process but, you know, it's still something. Another one is that we have to run multiple instances of a given service, right? So this doesn't work really well for memory-hungry apps, right? So luckily we had, I think, one Rails app in play and so that used a couple hundred megs of memory. But we had a couple hundred little node services rattling around and they shared a lot of memory together and this is on a given server, right? So we were able to run it because of the system and the tooling that we were using to building. And so the last big problem that we kind of fought and that we kind of worked around discovering was inter-service versioning. So that's really where things start to get interesting and exciting and really challenging and nobody's actually really publicly solved this stuff yet. So like, let's say that we've got service A and we've got service B. Let's say that service B is some sort of database-persisted service, wrapping, you know, any number of persistence layers like maybe like a, I mean, I don't know, let's say we've got Cassandra and Redis and Postgres in play and we're using each one for different purposes, right? Cassandra's super good at writes and Postgres gives us relational queries and Redis gives us some flexibility in terms of data stores. So they might need to change in lockstep and so this actually bit us a number of times before we came up with our inter-service versioning. If say we bump B, and we don't necessarily think about it because kind of as an engineering team we have stopped really thinking about versioning, present cons of course, if they don't change in lockstep or if there's no communication across teams, which is another one of those things that hearkens back to team growth, I might break things, you know, I might bump version B and all of a sudden this was like an integer last time and now it's a string that's a number, things that happen especially when you start playing with JavaScript, things that happen can break the world and it's not good. So how do we change these in lockstep? So basically what we did, and again this was all through tooling and all through services and routing layers and I hesitate to call it magic but there was some shady stuff going on under the hood. Basically what we do is we built a release version, so we've got A and we've got B and we've got A and we've got B prime and so even though nothing's changed in A, our application which is this collection of all of our services has changed and it's entirely different application from the perspective, ignoring the hosting layer because we're thinking at a layer that let's just pretend that we've got all of the resources that we could possibly want in terms of compute power assuming we're able to utilize them effectively like say some sort of collection of giant data centers that we can use on an hourly basis and scale up and scale down. I don't know, data centers, Haroku maybe, I don't know. So basically in a perfect world we wouldn't have to have separate A's running, we wouldn't have to have separate B's, et cetera, but from the perspective of a consumer of our API whether it's internal, whether it's external, whatever these are separate applications. So we tooled up around this stuff, right? So at deploy time, we would build a, what we call the release version and so the release version encapsulates every single service that we're about to deploy and what it depends on, all of its versions and then we do an internal mapping between all of them, right? So when I ask for service B at V2 that's gonna give me the Shaw of B prime. It works really well actually. It's a lot of wackiness but yeah, it's pretty cool. We have some downsides though. So we have to figure out some way that our services are able to correctly request versions. They don't necessarily have to think about hosting the right version but they certainly have to think about it from the consumption perspective. So like say, I mean like if we roll back to a slide real quick. When A makes a request to B, right? It's got to be aware of what its version is. So it knows whether it's asking for A at V1 or A at V2. So there were two ways that we can solve this and we did solve this two ways. We did it and solved it two ways. The first one's sucking and the second one being amazing. So the first one we did was we literally would just run separate versions, right? And at deploy time when we encapsulated everything when we bundled it all up we would say, this is your Shaw, this is what you request. The problem is that despite having tons and tons of say AWS capability and tons and tons of different computing resources at our disposal, running five separate services that are all exactly the same, it's kind of wasteful and it kind of sucked. So the second way we did it, and this is the way that it's still running now, is each service actually carries through its tag, right? So let's say that we have a node app and it gets a request coming in that major release version gets carried through every request. And so everything rattles through a routing layer which gives us a couple of really interesting benefits. We have really good metrics on the routing layer so we can watch traffic. Because of watching every single request go around inside the routing layer all of a sudden we can debug now, right? So whereas before like you roll into a black box and a request might bounce around through five, 10 services now we can actually replay all of those traces. So this is all getting pretty good for us, right? So again, I guess there are three ways, right? As an engineer I can think about it which like we don't wanna make ourselves think, we don't wanna make our colleagues think as people building internal tools, internal services, people handling operations, we want to make it so that our users don't have to think about it, right? You know, I mean if we were building an app for small business owners and we said, well, you know, step one, you've got to install Ubuntu and then step two, you can now sell things at your flower shop. That's not really fair to anyone, right? And so like when we've got an engineer who wants to just deploy some software for me to say, well, step one, first of all, you've got to fire up your own AWS instance and get Ubuntu running on it and then you've got to get all these services. Again, not really fair, we can do better. So yeah, let's just throw the whole idea of making engineers think about it out the door because we've got tooling. Another one, which was the first one that we did is you can lock at deploy time, right? So when we roll out end services and we build a revision for them, that's what it gets tied to. The third one, of course, is pass a header around or metadata on a request on whatever your request actually looks like. So we're starting to get good in terms of our deployment environment, our routing environment, all that stuff, but there's still more that we can do, right? So we wanna build the client tooling up so our development environment was bad. Luckily now, there are great tools to do it. I don't know how many people are familiar with Forman, but basically it uses Heroku's proc files to get multiple services up and running or in the case of something that might run on Heroku, you might have timers, you might have a bunch of different processes that have to run. Forman's good, it's not the best, it's not perfect, but it certainly gets us much better there. Another one that's really painful is assets, right? So what you can do is you can 12 factorize things, and I don't know how many people are familiar with 12 factor apps, but if you hit up 12factor.net, it talks a lot about making apps and services standalone work happily in an operations environment. We hearken a little bit to things like Docker, where we're containerizing and we're working on that, but it's really all about working on getting that API contract solidified and concrete in between something like, you know, my application that's running, where's it gonna log?  Well, what if we just use standard out and like push that out onto the hosting layer? So, you know, as long as your stuff is 12 factorized, life is gonna be not too bad. So, the lessons that we learned from this long and painful adventure was that inner service versioning is kind of unsolved, right? And so, it's one of those things that not everybody's figured out when we've got, say, you know, two services, what are we gonna do? Are we gonna, am I gonna have to make sure and version and then collectively bump my version and talk to all of my colleagues and make sure that everybody's on the same page when I bump my version. And so, you know, I'll have to like, hassle Ben to like stop using my V1 API because we're on a V3 net, like no, no, we gotta do better than that. Rolling deployments are pretty much required in today's world. Like, there's no excuse anymore for us not to be able to do this. SIG, I mean, we can SIG winch things, right? We've got beautiful, beautiful tools to do that. And the last one that we really learned with all the work that I was doing was you need to shield your engineering team from complexity. You need to basically have a Heroku-style interface where I push code and it works. There can't be too much magic. I mean, like remember early, early Heroku where they would drop a bunch of gems in and like it's really unclear exactly how the database works and it's a little shady. You wanna get rid of that magic, right? You need to make sure that when, as an engineer, when things go sideways and it doesn't work correctly, you need to know that you're able to debug it. You need to know that you're able to basically get it to work when it breaks. So basically this is kind of my pseudo-closing side is that we need to deploy better. Five years ago when Heroku landed, all of a sudden our deployment world got really nice. Like, we've used Capistrano, we've probably used VLOD, we've probably used a number of different tools. But finally this thing showed up that abstracted away so much of the complexity and so much of the pain that, I mean I know that I deployed like 15 odd different like toy apps in the first couple of months, right? Just like, oh man, I can put this thing on the internet. I don't have to like get some like VPC somewhere and like all of that pain, all of that nightmare. Like, we're in this kind of like amazing time to be an engineer where our tools are getting so good. The APIs, the frameworks, everything is just amazing. For monoliths, deployment's really good. But for services, deployment still is kind of rough. It's kind of a challenge and there's kind of a lot of work that we have to do with it. The other one that we learned of course was that you need to minimize moving parts. Like if I roll back to that slide talking about DMV and Viaduct and the chicken coop out back and all that stuff, you know we had four daemons and we had two persistence layers and this is just to run our software that actually does the customer's needs, right? Like this is madness. So we need to minimize those moving parts and the last one is you need to make it pluggable, right? So one of the other issues that we ran into was as time went, as you know, this stuff software got old there was newer and shinier stuff out, right? Like we first started with Circus as our process manager and then we used Upstart and then we actually ran God for like two weeks, maybe three weeks and then we wrote our own. Now they're hearing right there, but you know, the ability to have things modular and pluggable is gonna really make all of your deployment stack and all of your deployment infrastructure much, much, much easier. So I do have one more thing. I've built this and open sourced it. So basically I took all of the stuff that we learned the hard way writing it at Moz and with actually some of my colleagues who've done some code review and talking with a whole bunch of good friends and good people, I've rewritten it and open sourced it. So hopefully it'll work. It's right now running a couple of my side projects. It's basically a full deployment and tool chain and hosting layer for microservices. As of right now, it runs on your own hardware. My stretch goal that if any luck I'm gonna get it done this weekend is to push it out on Heroku. Because there's no reason not to, right? Like as an end user, I should just be able to call for deploy of all of my services and they're running somewhere on the internet. That's basically the dream and it shouldn't have to be a dream anymore. You know, it's 2015. We've got amazing, amazing tools. We've got great beer. I've seen a reason that we can't get some great deploys. So that's it.