So you finished writing your code, you've debugged it, works great, you've pushed it out live, and now you're experiencing performance issues. You want to know how to fix those issues, and this multi-part series of Visual Studio Toolbox, we're going to be talking all about Profiling in Visual Studio. Hey, everybody. Welcome to Visual Studio Toolbox. I'm your host, Leslie Richardson, and today I am joined by my colleague, Sagar Shetty, who is a PM on the Profiling team. Welcome, Sagar. Thanks for having me, Leslie. Glad to be here. Awesome. So for those of you who don't know, I am actually a member of the Visual Studio debugging team, which is a smaller team in the larger Visual Studio diagnostics world, and that's where the Profiling team comes in. So in this multi-part series, we're going to be talking all about Profiling. But what exactly are we going to be talking about Profiling, Sagar? Yeah. So just to give an overview of the series, I'm really excited for this. Basically, what I thought we'd do is start with a high-level overview of Profiling in the first few episodes, and really by the end of the series, do a deep dive on all the various tools that the Visual Studio Profiler offers. Great. So to me, and I think a lot of other people, Profiling is that thing you do where after you're done debugging things and you go ahead and ship your code, people start complaining to you about how your app is too slow, and then you have to quickly learn up on all those tools that maybe you didn't think twice about in Visual Studio in order to determine how to better improve your performance. But can you tell us a bit more about what Profiling exactly is? Yeah, definitely. So it's a lot like what you say, Leslie. It really is the next level of diagnostics past debugging. So there's definitely, as we say internally, the interloop scenario of day-to-day, trying to make sure that your code, first of all, just compiles and just has a similar behavior to what you'd expect, right? But really, there's that next level of diagnostics past that, which is, okay, maybe my application does compile, but I want it to be a little bit more efficient, a little bit more performant. I want it to be a little bit faster, right? So that's really where Profiling comes into play. To be fair, not necessarily every developer focuses on this kind of diagnostics, but there is a certain subset that does, and we cater to those developers and make sure that they have proper and adequate tools in place in order to be able to solve those kind of challenges. This is done by essentially measuring the performance of the application's data as it's run, capturing that performance data, and then going back and looking at it. And we'll talk more later about how our tools go about doing this. And really, when we're talking about performance, it's not necessarily just one metric we're looking at. I mean, performance can refer to a lot of things, but at a high level, it could refer to kind of wall clock time, just the amount of time the application is taking to kind of run or complete certain operations, CPU times or CPU cycle, the amount of time you're holding up like a CPU or it's taking to process certain tasks, and also memory usage as well in terms of how much memory you're allocating for different objects and things like that. So there are a lot of different metrics we look at, and we have a lot of different tools to support those different kind of performance investigations. Cool. So when you start Profiling, or if you're just curious at all about it, why else beyond just, yeah, like the basic, it improves your performance? Should you care about Profiling? Yeah, so I think there's quite a lot of value in what Profiling offers on both to the customer and kind of on the business side as well. So just kind of give some examples. So on the customer side and the user experience side, I think everyone just likes performance, like better performing applications, right, faster performing applications. I'm sure you've been there, Leslie, where you take a mobile app or web app or any sort of app that you just load it up and it's just sitting there and it's hanging and it's not loading the same forever, right? So there's certainly a lot of value to the customer in terms of making sure that the application, like even if it does work eventually, it's not good enough. It actually needs to like work in a certain time period. So less waiting, I think it needs to just happy your customers in general. So on the UX side, there's definitely some value there. On the business side, there's also, I think, cost to consider too. So I think if you're hosting an application in the cloud and you're having to pay for VMs and stuff like that, if you have certain optimizations and you design your application in such a way that maybe is a little bit more performant from a CPU standpoint or takes up less memory, perhaps, you don't need as many VMs to support your applications and as a result, your costs are lower. So there's definitely a lot of value there on the business side. And lastly, we're kind of living in a world now where you're having to design software for various constraint scenarios. So devices, we support IoT profiling and there are customers that we've worked with and done profiling extensively such as like the HoloLens team and Xbox team and they have products that are operating in very kind of specific and interesting environments such as VR and also just kind of like gaming engines that need to be performant because their customers are expecting a certain amount of performance and if they're not performant, those applications aren't necessarily as desirable to use, right? So kind of thinking about some of those constrained scenarios, profiling is definitely at the forefront of a lot of different people's minds, yeah. Awesome. Particularly the cost point came out, stood out to me the most, like who doesn't like things to be cheaper, especially when it comes to developing large scale applications and things like that, right? Yeah, absolutely. Awesome, so as a PM on the profiling team, what kinds of customers do you typically see using the profiling tools that you create and or what kind of customer base would you like to see using it more frequently? Yeah, absolutely. So at a high level, right? It's kind of going back to performance. So the general motivation is anyone who wants to make their application faster or more performant is kind of the impetus or like the initial reason why you kind of come to our tools. And I will just say that like, if you haven't used our profiling tools before, definitely would recommend checking that. There's a chance that you might get some pretty easy quick wins and really make your application a little bit more performant. But more than that, you know, it's generally people that are investigating issues that don't necessarily break your program in terms of preventing it from compile, but it's just too slow or maybe you're mismanaging memory and you're out of kind of memory to allocate for other things. To be a little bit more specific than that in terms of talking about some like the industries and markets we operate in, kind of alluded to it before, but like game developers are really big into profiling because performance is so key to their product. Another kind of internal customer we work close with is a company that builds a 3D rendering software. And that's like a very graphically intensive tool. So they kind of heavily use our products because performance is key to them as well. Great.  But yeah, and it seems like that goes under the larger just general performance issue umbrella. So when talking about performance, what are some other things that can count as performance issues? Cause we always talk about performance, like it's just this one singular thing when it's actually pretty big. Yeah, that's a great point. And I would say that generally speaking, there's a lot of different tier point performance issues that can come up, but generally speaking, they kind of fall into a few different buckets. So I'll kind of go over some of the common buckets that pop up a lot. So for one, I would say kind of mishandling CPU time. So this kind of comes back to just using bad or maybe slightly not modern practices in code. So one thing I know you're very familiar with Leslie is like asynchronous code, right? And using that pattern. Yeah. So definitely, you know, if you're not necessarily using asynchronous patterns or patterns where you're using nested loops and having to constantly iterate multiple times over large datasets, that's going to slow down your CPU and really make your application less performant. Furthermore, things like not caching properly. So maybe you're making an API call to some sort of external data frame or dataset and loading up like a large dataset. And if you don't cache that properly and every time you make that API call, you need to reload that large dataset. That's also going to slow your application down. Another thing kind of under CPU time is kind of string building and logging code. So this is kind of like a bit of a debugging diagnostics thing, but as you're kind of building out those strings, that operation is really expensive and that cost is distributed across your whole application. So that can really slow down the entire application. So all of those kind of bad patterns, not using asynchronous code, not caching properly are kind of examples of different patterns that may show up under mishandling CPU time. To your point about mishandling memory and that being tough to solve, that's definitely another high level issue we see a lot. So that kind of comes down to maybe allocating too much memory or too often or not deallocating memory once you've kind of like had your fill with a specific object and hanging onto objects well past their lifetime. This kind of leads to not garbage collecting properly. And luckily, if you're operating within some managed memory languages like C-Sharp for example, and using the .NET framework, there is like the runtime does kind of take care of some garbage collection for you. So it's definitely getting better at the runtime level with that. But we do have tools that help you visualize kind of issues with your memory allocation, which is tough to do otherwise without the tools. And also we have tools that can help force garbage collection because even though the runtime does a decent job of kind of collecting on its own sometimes, and we'll talk about these scenarios in future videos where you may wanna force that garbage collection, we have tools that are able to help you do that. And so kind of, so we talked about mishandling CPU time, we talked about mishandling memory. The last kind of major bucket I'll talk about is kind of external dependencies of your app. So obviously you have your user code, but if you're kind of making API calls over different networks or slow database calls you have those external dependencies, right? It's a bit tricky to kind of figure out maybe what's going wrong because you don't necessarily have access to all of those external different bits you're playing around with. So we have tools for that as well that can help you kind of deal with those scenarios. All right, cool. So at this point you've figured out that you're having performance issues in your code, it could be a memory issue, a CPU issue or external dependencies that are just too slow to respond. Now what? Like for most people who don't know these tools, chances are they probably got to play the trial and error game with this, right? So how can we be taking more advantage of the existing tools in Visual Studio to fix those issues? Definitely. So we talked a little bit about what profiling is and kind of the common scenarios like you suggested. And so now kind of wanted to end this video kind of with a little teaser into what's to come and kind of getting started. So I'm gonna launch Visual Studio and share my screen real quick. Okay, so now we're in Visual Studio and basically I wanna kind of answer the question how do I get started and start using some of the Visual Studio profiling tools? So to kind of to mock this up, I have an ASP.NET Core web application loaded up and to kind of get started with some of the profiling tools, what I'll first introduce is the performance profiler. So what is it and how do I get there? So to get to the performance profiler, there's a few different ways, but you can go to debug and then click on the performance profiler in the context menu. There's also this keyboard shortcut Alt F2. I'll just click on this button here. And now we get to this particular summary page. So essentially at a high level and we'll go into the performance profiler more in depth in future videos, but basically it's a suite of tools that allows you to drive your performance investigations and look at a lot of different kinds of metrics. So we have like the CPU usage tool to do different CPU investigations. We have a few different memory tools. You've got a new async tool to kind of see where your application is spending a lot of time with certain tasks. And today I'll kind of just kind of showcase the CPU usage tool and highlight just the basic collection process because this is one of our most used tools. We'll talk about this more in a future video, but you can use a few of these tools in conjunction. There's also a bunch with the terms of optimizing different profiling settings. But for now, I'll just kind of keep the defaults on. This is interesting. It's like the profiler has its own little world. Yeah, yeah. Menu screen and everything. Yeah, it's got a lot of different tools and it's creating this DIAC session, as you can see here. And one cool thing about this is like, once you're done and you collect a bunch of data, you can send it to a colleague and they can kind of load it up on their machine and kind of help you with that investigation by kind of seeing what happened on your machine too. So you kind of have a little bit of that collaboration there. So once you kind of have the CPU usage tool clicked, you can hit start to kind of start the process. In the interest of time, now I've kind of already done the collection, but you'll hit start and kind of let it run for a few seconds. And then there's a stop collection button. But at the end, what you'll get is kind of an output of this kind of a report. So this is our CPU usage tool. And again, we'll have a whole dedicated video to this. But basically with most of our tools, we kind of have this Swimlane up top, which has kind of a graph to show some basic data points over time generally. And then we have some sort of tabular view. And it really depends on the tool, like what that table is showing. But basically here, we have some functions that are of interest and are kind of taking up a lot of the CPU and some hot paths, which are essentially some areas of code that you should really go back and dig into that are also in this case, taking up a lot of the CPU's resources.  Yeah, and you gotta love the pie chart, right? Right? Like when I have to make presentations sometimes, I always get really excited when I'm able to include a graph. It's like, oh man, much more official now. Yeah, for sure. And so yeah, that's kind of basically a summary of the performance profiler. Again, we'll totally go into a more in-depth overview of this in the future. But for anyone that's, for the enthusiastic learners amongst us, if you really wanna kind of dig in, get started, kind of go to the performance profiler, go to Alt-F2, and then kind of play around with the tools. That's great. So from my experience, I am familiar with in the diagnostics hub window, it also has a CPU usage tool. What's the difference between the two? Yeah, so in our very next episode, we're gonna kind of unpack that a lot more, but basically with that one, one of the advantages with that window is it allows you to take advantage of all of our great debugger features that you're well aware of, Leslie. Yeah. And so it's really cool to kind of get that interaction between some of the profiling tools that we have as well as the debugger. So the diagnostics tool window is really kind of using those two in conjunction. The performance profiler is more kind of like the standalone tools, a little bit more robust. And you also get a few different visualizations because we just have some tools in the performance profiler that don't necessarily have an equivalent counterpart in the diagnostics tool window. And we'll talk about some of those more niche tools later on, but yeah. So on the larger scale, before we do deep dives on each different really awesome feature in the later parts, what would you say are like some of the hardest challenges when it comes to developing for the profiling space? Oh, wow, that's a really good question. So I would say kind of high level, there's two main challenges. So one is that we collect tons and tons of data. And so something we really wanna do for our customers is make sure that that data is very digestible and easy to visualize and actionable. So kind of figuring out how we wanna best visualize the large amounts of data sets that we deal with is I think a bit tricky. And then the second thing kind of along those lines is tying that data back to the code and essentially figuring out like how can you connect? Like cause we have tons of data that spit out and again, we wanna make sure it's actionable. So kind of being able to highlight the specific parts of your code that are of interest and the areas that you need to focus on to fix your problem is ultimately another big challenge. That makes total sense, I can only imagine having to try to sift through a giant list of information and being unable to quickly locate the actual hot point. Yeah, that's a lot of data. Yeah, great. So that, do you have any final thoughts before we dive into all these really cool tools and later parts? Yeah, again, just really excited to start the series. In the next episode, we're gonna go over the performance profile a little bit more in depth, kind of talk about more of the tools, cause again, we just kind of briefly talked about the CPU usage tool today. We'll also be doing a little bit of an overview on kind of optimizing settings, kind of going over scenarios where you can start to use a few of the tools in tandem. And then also hopefully mentioning the diagnostic tool window that you brought up, Leslie, the one that it's more integrated with the debugger. So again, kind of going back to the idea of like, how can we best use a few of these tools in conjunction and not just focusing, just kind of on one tool. So yeah. Fantastic. Well, thanks for joining me, Sagar. And I look forward to having you join us for future episodes to come. Thanks for having me, had a good time. Great. See you all later.


our first presenter Dr. Carolyn Zachary from the Adult Education Office at CDE. All right, thank you very much Melinda. Welcome to our webinar. I hope that you're ready because we are ready to start remote testing and I am happy to be here with the cost esteem. We'll be talking about who will be giving you all of the details. I'm giving you a real high level view right now and that is the fact that we are going to be instituting a remote testing policy if you are going to be doing this. You will need to amend your local assessment policy and that local assessment policy is going to need to identify what tests you're planning to administer in a remote setting. Included in that will be an assurance that your agency is going to follow all of the remote testing requirements and an assurance that all of your staff are trained to administer those assessments. This is following OCTAE guidelines which are part of our NRS reporting requirements. So this will include that you have some way of or process for identifying the student to ensure that your proctors are completely trained for proctoring the test remotely, that you're also ensuring test security and that you are providing live proctoring. A template and guidance will be available for agencies to use by the end of the week. You may notice that in TOPSPRO Enterprise known as TE that there is an optional box for students that says force majeure not able to test. This is what we will be using as we provide you the guidance so don't do it now but as we provide you the guidance for what to do if you are not able to do remote testing with students. The reason we're asking you to do this is because OCTAE is very interested in the data that will be coming from states with those those numbers of individuals they weren't able to test. We will be including this information in our narrative report to OCTAE. Okay I know these two words cause a lot of anxiety for everyone so I want you to all breathe and I want you to be aware and know that the adult education office knows that you are very concerned about your payment points and that you're concerned that COVID-19 is going to dramatically impact the work that you're able to do with your students and the results of all of that hard work known as payment points. We will be evaluating all of the data that comes out of this after the certified by your agency and that doesn't happen until August where you do your certification so we will not be evaluating this data until the fall. I want you to understand that the data from the 1920 year is used for your 2021-22 grant award year so it's not used for your grant award that's coming up here in July. We really want to evaluate the data very carefully and ensure that agencies are not dramatically impacted by COVID-19 and the inability to test students because we know that this is an equity issue across the state. What we really want you to focus on right now is on your students and using testing to help your students see their growth as well as your teachers so that your teachers can see the growth that the students are making in a remote setting so we encourage you to use it for those purposes, relax about it for payment points. We will be focusing on those in the fall when we have the data which is what we really need to have to be able to look at all of this. I would also encourage you those of you that have issues with connectivity because of the remoteness of where your agencies are located to look into the Rachel device. This could help with your connectivity and I've asked Penny to put information about this in the chat and it is from the World Possible Organization and this may help you in your remote setting and with that you're welcome to email me any questions that you have so with that I'm happy to hand this over to Pat Rickard, president of POSIS. Thank you very much Carol and thank you all for joining us today to learn more about remote testing for the NRS approved COSIS pre and post test. We're very excited to share with you what we've learned, what we're doing and what we plan to continue to do to support you and we really thank those of you that have willingly agreed to pilot test and have given us some really great feedback. I do want to many of you joined us last week on Thursday to learn more about remote testing for the co-apps and sit and I want to put this session today in context of the other two. I'm looking at it as three different levels of remote testing. The first level would be our performance-based co-apps and those are developed by the local agency and you have in place trained staff and the security on those, definitely we want security but it's something that is within your agency and not across the state or national. The second level would be our performance-based citizenship interview test. It is standardized for use across all agencies in California that are offering English language citizenship preparation, you know, civics citizenship preparation and we have trained SIT assessors throughout the state in your agency and we talked about simply providing some additional training for them in order to administer the SIT. The third level which we're going to be talking about today and Linda is going to be covering is our nationally approved NRS classes pre and post testing which is multiple choice. We will also cover the EL civics government and history test which is multiple choice. Each of these three levels the co-apps level one the performance-based SIT level two and the NRS pre and post multiple choice level three each have separate guidelines and each have separate go remote instructions.  Thank you Pat. Hello everyone. Glad to be able to present with you today and thank you for taking time to come and hear about this. Next slide please. Yes here's our agenda and as Pat mentioned we are focusing today on the multiple choice tests the NRS approved pre and post tests and government history for citizenship. We're going to give you an overview some background about remote testing and how COSIS is approaching it. We're going to tell you about our clinical tryout and the pilot phase that happened mostly in April and then get into what's happening starting today May 4th where we're moving into phase two still piloting but also rolling out for agencies that have been piloting and rolling out different approaches that I'm going to describe then also we have a phase three which is you'll see then we will tell you about next steps where to find all of the documents and supports that we're going to be sharing with you today and then at the end and actually I'll be stopping along the way and pausing to see if there are any burning questions that haven't been able to be answered in the chat box. As there are so many people on the call we may not be able to get to all of your questions in real time today but we will compile them at the end and post that on the website so we'll hope to answer everyone's questions at some point if not today then afterwards. We're going to get into an overview now that is going into a little bit more detail about what Carolyn introduced in terms of the Department of Ed the office of career technical and adult education octay. They put out a memo on April 17th that stated very clearly that states have flexibility at this time with this disruption caused by COVID-19. States may allow local programs to exempt students from peer and post testing those that are still enrolled doing distance education and also octay the federal government wants to know what the time frame is for that in your programs and who or the extent of those exemptions. They know you're not going to be able to test everyone under the circumstances and they would like to have an idea of what happened. As Carolyn mentioned there are certain guidelines that they put out that are to be followed when doing remote testing whether with CASAS or with another test publisher and these are in these very broad areas of making sure that the student identification is authenticated making sure that test security protocols are in place and making sure that proctors that are administering remote testing have been trained and know what to do. On April 9th octay met with all of the test publishers and CASAS of course was there and then about a week and a half later we just we CASAS met with octay presented them with a plan to roll out remote testing. They have given us permission to do that and we were happy to hear that they were quite impressed with our remote testing plan and with the speed at which we were able to pull that together. So that is something that has given us made us feel that we are supported by octay and we are looking forward to rolling this out with you. We did conduct a webinar on the 20th of April with state directors in 30 states and told them what our plans are. We have actually had pilots going on in the last few weeks in probably 15 to 20 states including California. So it is really being tested by programs like yours all over the country. We are hearing how it is going and we are making adjustments in very real time as we go. We have been updating the guidelines pretty much daily or even sometimes more than that. So to really make it as streamlined as possible. So what are some of the benefits of doing remote testing? One certainly is that you are able to continue providing services in terms of assessment. You are able to post test. Now of course we know that you are not going to be able to test all of your students. We really want to make it clear that that is not the expectation. But during this time we hope that you will take advantage of it to try out remote testing. Post test perhaps select certain students that are priority or that you would like to see how they do with remote testing. So you will also of course be able to see how they did in their coursework. If you have incoming students and we know you do have a lot that are recently unemployed therefore might have more time than they had before to study and taking advantage of that. We know you would love to get baseline scores for those students and so if you choose to do remote testing you would be able to get those using E-Tests remotely as well as use these assessment to guide where to place these students in distance learning. We really want to emphasize that this time also gives you the chance to develop your agency's capacity to learn how to test distance learners remotely. And when all of this is back to at least some semblance of normalcy that will add a lot of flexibility to your assessment programs so that what we envision is that all of our assessment programs will consist of both a face-to-face component as well as a remote testing component so that you will be able to test students that have difficulty coming in to test at your test centers. So it looks like remote testing is the wave of the future so that we can do distance learning as robustly as we've now learned how to do or are still learning how to do and we actually are very much in a learning mode but feel that we're improving day by day in terms of understanding what's possible, how to do it more efficiently, more cost effectively, and with additional types of technology. So we do know that we're learning together and look forward to that. Now challenges, I'm sure you all could fill the chat box with all the challenges that you've already encountered with distance learning, getting that going, one of them first and foremost I'm sure is the students access to technology, the equity concerns related to that, the digital divide that has come into such sharp focus these days. When you're going to consider doing remote testing you'll need to have certain devices for proctors and students and I'll be giving you details for that shortly. Smartphones are too small as you I'm sure can understand for the test items that we have in our NRS approved tests that they would not be able to see them well on a screen and that would mean that in turn they would not be able to perform well on the test. So unfortunately although most of our students do have smartphones that is not a possible device for the pre and post testing. It can be used if you attended our session on Thursday, it can be used for the co-app assessment deliveries, it can be used for the SIT but not for pre and post testing. In terms of internet bandwidth that can be a challenge as well with Wi-Fi being much in demand in a given household there could be kids trying to use it at the same time as their adults working from home trying to use it. So we know that our students may have a challenge getting access to the bandwidth that they need so that's something that will have to be planned well ahead and hopefully dealt with in a way that will work out. Also in the students test environment at home they may not be able to get a private place to test or it may be challenging. Many distractions with others in the household so here again there are many challenges that we are well aware of and I will say now and I'll keep repeating to really reassure you that all of these challenges are things that we have considered as we are writing our guidelines. So whether it's for SIT or co-app or these pre and post tests, the remote testing guidelines that we are going to be posting that you'll will require you actually to study very carefully before you embark on remote testing. They have very detailed instructions step by step, procedures, scripts, here's what you do, here's what it looks like, we're going to be I'll tell you more later but we're planning to do some follow-up sessions with videos of what settings in certain types of platforms so we really are aware that you are going to need that kind of very specific help and that is provided in these guidelines. In this webinar today we don't have time to go into all those details I'm going to get into some level of detail but please be assured that these guideline documents have very very specific instructions. So it is going to be somewhat of a challenge for you and your program staff to learn and implement these new processes but based on our pilot testing with a little help from CASAS and a little practice we're sure you're going to be able to do a great job. We are we know that your proctors have been trained to give E-test space to face but they have not been trained to do that in a remote manner so we are relying on you to ensure that they have read the guidelines carefully that they take advantage of the training that will be posted as recordings on the CASAS website and that they are ready to go when it's time to go. Test security is extremely important and you know we're not going to be able to make it airtight but we hope that we can close up as many loopholes as possible and we really are going to count on the goodwill of our students to some extent but we are putting in place as many precautions as we can without making it too cumbersome for you or the student. Equity concerns have already been mentioned to some extent with the technology but there may be others. So as we researched all of this we looked at some third-party vendors there are lots of them out there we met with many of them. What we found is they're expensive a little bit steep for our adult ed budgets 15 to 20 dollars an hour to have a live proctor in a third-party vendor so that's not what we went to first we may reach out to them and perhaps have be able to crack a deal that is affordable at some point but at this point in time we are not offering that kind of solution nor an artificial intelligence solution at this time and we looked at a variety of video platforms we talked to state directors and their staff we talked to a lot of folks who are familiar with distance learning to learn what we could from their experience with that we made weight multiple factors um cost which in terms of the technology to use for remote testing uh we think is fairly minimal however there is a cost in terms of staff time and that is because uh with remote testing and the need to be able to observe the student in real time it's necessary to limit the number of students that can be tested at one time and so it's between one and five we are allowing at this point if we find we can allow more great but we understand that that is going to be labor intensive and therefore costly for your programs depending on how many students you decide to include again though I want to emphasize that this is a time when you're trying this out and you are not obliged to do this remote testing but if you do want to try it it will be more labor intensive because of the high ratio of proctor to just one proctor to only a maximum of five students so we understand that there's really I'm sorry nothing we can do about it but that is the situation at this time we know that there may be technology constraints both for proctors and students which perhaps you've already been working to alleviate when you've done uh distance learning now by uh learning out devices so perhaps that's a way that you can work this out as well test security is something that we'll talk about it takes somewhat different forms with remote testing ease of implementation uh we've tried to make this as easy as possible though it does take a bit of doing in terms of understanding how to set up the video platform and etc all the different steps but it's possible and doable and we know that many of you are very eager to get started so we've tried to address that urgency by making this available on a very short timeline from compared to how we normally develop new types of assessment in terms of the delivery it will be live proctoring so that's in real time synchronous uh there's something called a remote testing agreement where we'll want to have one per agency and that will cover all of the different remote assessments that are possible there's an agreement for proctors that you will have them fill out and keep on file at your local program and then students uh this is not something we've had to do before but because they're at a distance and not uh under our watchful eye uh next to us we will ask them to verbally agree to certain uh test security promises in terms of delivery uh it is on e-test so we need you to use certified e-test proctors who've already been uh ideally who already have experience using e-test so they must be certified proctors there's no by the way additional certification for the e-test administrators just the proctors and these proctors will need additional remote test training as we just said um we had originally talked about videotaping but we have gone away from that uh after thinking it through a bit more and getting input about it we have decided not to require videotaping of these remote assessments it's just a little too cumbersome and hard to figure out how to store them lots of reasons so that'll make it'll easier for you uh and also could be a privacy issue so no videotaping required also just to be very clear the CASIS remote testing cannot be delivered with paper tests uh if you're doing remote testing it can only be with CASIS e-tests uh for the pre and post tests now uh in terms of co-apps or SIT you may um you may the conditions are different and you still need to be able to see the student live but um there can be some paper involved but we'll explain that that that is explained clearly in the previous webinar and um we can clarify that it's all laid out in the guidelines for co-apps and the SIT as well but as far as the pre and post tests no paper tests also we know that in some programs they have figured out ways to do face-to-face testing even during these times uh of the COVID-19 crisis that could be in corrections programs we know that some programs have gotten creative and have found ways to have students take the test in their car getting a box you know that's that they come and pick up but they and it's never never touched they never touch uh uh anyone on the staff's hands anyway it's there are ways I guess to do it where you can observe social distancing and still have the student be doing it in person so any situation like that where you're giving the test face-to-face essentially uh the basic normal rules still apply that a proctor must be physically present so if you are doing that that the the usual rules apply and um you need a proctor so which tests can you use for multiple for remote testing with multiple choice tests it's the same ones you've been using the goals series reading goals and math goals for ABE and adult secondary for ESL it's life and work reading and the life and work listening 980 and also though it's not it's not a pre-test but it's a post test really the government in history for citizenship test a certification test that is given to ESL citizenship students is also a multiple choice test and can be used with remote testing following the very same guidelines that we're going to tell you today for the pre and post tests now this slide shows the phases and the one we have just completed is phase one where we did clinical tryouts and piloting and we restricted that to one-on-one one proctor one test taker and pretty much used only zoom because that made it a little easier to see what was going on so that was I would say quite successful we got a lot of great input and we incorporated those suggestions into our guidelines starting today we are moving into phase two where we are going to be having if you're just starting up we're going to ask you to pilot that is you know not go full scale right away but start small pilot and then roll out whether zoom or another platform like go to meeting webx or other ones one-on-one is one option another option is proctoring with up to five test takers and that can be with the with various devices I'm going to explain the details soon and that will soon include Chromebooks it does not include Chromebooks at this moment in time that's something that we hope to be able to have available within a week or two so we're talking weeks and not months and it's very very shortly we hope to be able to make it possible with Chromebooks but starting now you may test up to five test takers with these other platforms like zoom and go to meeting for example and then we are adding on the ability to do remote testing with the listening 980 series as well as the government in history for citizenship test in the future we are still interacting and exploring with third party vendors if we can possibly work something out to make that work there are other options we're looking into that if they can make this a more efficient and less costly proposition we will certainly look into them and let you know as soon as we've worked anything out all right let's go on to the next slide then and we'll so now I'm going to tell you in some more detail about the three approaches that we have come up with so far and that we're making available the first one is one-on-one and in this approach the test takers have a variety of devices that they can use that's one of the advantages of this approach they can use a pc they can use an ipad or they can use a mac and in this scenario the proctor needs to register their computer and it has to be a windows 10 because the proctor is going to run e-tests on their computer and then remotely share their screen with the test taker through the web conferencing platform so that's why this can only be one-on-one because the proctor is loading one test onto the proctor's computer and then sharing that with one student the proctor and the student are connected and be able to see each other real time using a mutually agreed upon web conferencing platform with a webcam and they need an internet connection that is two mbps or higher okay let's go on to the next solution in this one it is possible to test up to five students however the I would say challenge here is that the test takers have to have windows 10 PCs so that may be a high bar we know that but we do want to make it available in case there's a chance that you can provide students with these or if some students do have a pc with windows 10 then they could and you could also take advantage of testing more students this is necessary because that's they have to be able to run e-tests on their computer so another aspect of this approach is that the test takers computer has to be registered and just want to let you know that for the purpose of remote testing we are allowing there to be only one person that is the proctor to register the student's computer normally we require two certified staff to do that but for remote testing we are waiving that requirement and so the proctor would be able to register the student's computer then using the web conferencing platform that you select and the webcam you the proctor would be monitoring these one to five students and again you need a and I believe this is not a very fast internet connection but again given that it may be in competition with others at home using the connection it might be challenging so part of again these detailed guidelines step-by-step instructions you're going to get there is a link and all the instructions for how to test the proctors and the students internet connection speed there are a number of websites that do that and that's something that you'll have to do on the day of testing you can do it before to double check but also I mean to see if it's even close but also on the day of testing and this is in the guidelines you'll be checking their internet speed to ensure that day that they're going to be able to have a fast enough strong enough internet connection to do the job the third solution is coming soon and we know it's eagerly anticipated and that is with Chromebooks because we know that many of our students either have a Chromebook or their students were given a Chromebook or you can give them a Chromebook because they're so affordable and so we are moving as quickly as we can to make the Chromebook solution available and in this solution the test taker would register the Chromebook on their Chromebook that they have at home you would help them set it up register it again just only you and not two people then like in solution two the proctor would monitor using a live web conferencing platform and you can do up to five students but the difference here is that with Chromebook you can only run up one application at a time so in this scenario with Chromebooks the test taker would have to use their smartphone with the video cam to and they would set it up on the side so that the proctor would be able to see their screen and to see the test environment while they're taking the test so that would be how we do a workaround with Chromebooks because you can't have the web conferencing platform on working on Chromebooks at the same time as e-tests by the way one thing that is again in these very detailed step-by-step instructions is that we asked proctor to do a sweep of the testing environment and this is something we explained to the student beforehand we encourage you to have a session before the testing day or before the test session to explain to the student and they so that they can be aware and anticipate all of these situations that would be new to them including this requirement that the proctor will need to scan the room they'll let the student will have to scan the room so the proctor can see this is to make sure there are no people there who could be giving answers to the student no inappropriate documents papers aids that might help the student answer questions so it's a test security measure we know it could make students slightly uncomfortable but we hope that you'll have a way to explain to the students that this is necessary for um test security that it's only for that purpose we're not looking at anything else in the room just looking for that and this would be done before and after the test session or if the if the proctor senses that there is something amiss the proctor can also ask the student to do that anytime if they're concerned and there's also instructions in the directions that uh if you if a proctor sees a reason to be concerned then the proctor can interrupt the test at any time and try see if there is a real concern in which case they would they would stop the test and then as i think you all know you're able to start the test up again if appropriate or end the test and end the session so this is a mock-up version of these guidelines just to let you know they do exist they're quite detailed about 20 pages with examples and scripts etc so there is something called an agency remote testing agreement this is a form that will ask every agency that intends to do remote testing to fill out one per agency it will cover all of the possible tests that you can use for remote for EL civics or pre and post that would be the cost of center as to prove pre post tests the co-app assessment deliveries the sit or the government in history for citizenship we will ask you to tell us how you're going to verify the student identity and in that we're going to ask you to use the procedures and the ways that you normally do that in your local program so nothing will change there but you may have to figure out some ways to do that remotely that are different than you normally do it perhaps and then another aspect of this age and very important aspect of this agreement is that you attest to having read these remote testing guidelines thoroughly and that you agree to abide by them so now we're going to go out of the PowerPoint and into the actual document so you can see the this agreement and I'll show you a little bit more detail that's in this test remote test center agreement so okay yes so that at first it says you've read the guidelines then that the second paragraph down there is that you know we would like you to stay informed because we're releasing the version 1.0 but it's very possible likely I would say that we will be updating that with additional refinements and improvements so we will let you know when we release new versions and they'll be on the costus website we're going to have one page dedicated to California remote testing on the costus website and that's where you'll find all these documents all these resources and the trainings including any updated versions of these guidelines again to reiterate we're not asking you to record and that we would like you not to record because we would not like these recordings to get into the wrong hands so we thought there was really possibly too much danger in recording because you would need to then ensure that those the privacy requirements would be maintained also for test security all right so then here we would have you list which test series you're going to use by checking off which ones you plan to use and you know we're aware that when you fill this out you may perhaps decide to use certain of these tests and not others so if that changes over time that's fine you can work that out with your program specialist and your cde regional consultant to see if you want to revise this but just please do the best you can at the time when you want to begin we know it may change then we want you to indicate which of these approaches it might be more than one that you're going to be using in your agency so are you going to have the proctor heavy tests on their computer their windows 10 and have students have more options for the devices that's number one one on one are you going to opt for a solution where you can test more students at a time but they need a windows 10 and if we can get that changed at some point in time we will let you know it would be nice we know but at the moment that is the requirement for this option number two and then option number three which if you're going to fill this out this week you would not be able to check this one because it's not quite available yet as soon as this Chromebooks option is available we will definitely let you know and then you'll be able to use it so just to reiterate Chromebooks is not currently an option as of today but in the next week or two we expect to be able to have tested out all of that that whole process and have good clear guidelines for you to use Chromebooks as well all right then just briefly to share with you about the co-app assessment deliveries and how this is also in this agreement so if you're going to venture into delivering the co-app assessments remotely we want you to list which ones we want to you to briefly describe how you're going to adapt them if you need to for remote testing and then on the right we've listed from low to high tech some options for the technology you may have think of others but and you can if so you can certainly add them to this document when you submit it but this will give an idea of what you're planning in terms of the assessment deliveries of your civic objectives and then for SIT if you're going to do remote testing again it's optional but if you decide to you can use the if you happen to have the printed COSIS paper SIT test booklets either you brought them home or your agency can get them to you that's perfect and then really nothing changes you would as you would normally it's a consumable test booklet so you as you're giving the test as you know you write the 012 rubric responses right in the test booklet and that's how you administer the test that's if you don't have access to one of those paper booklets we have created and this is new a fillable PDF version of the form 973 and 974 they're not quite yet available but they will be available by next Monday the 11th of May that's when all of these things will be on the COSIS website so please have a little more patience we're almost ready with to roll these out but when that time comes that those booklets will be available for you to use if you don't have the paper booklets you might decide to use both depending on what your SIT administrators have at hand and then for SIT remote testing there's just two options for the technology and platforms and that's smartphone with the video camera or computer with the webcam phone then we already talked about the identity requirements and then when you're going to start estimated is what we want to know okay back to the PowerPoint and so then the proctor remote testing agreement is something that will also be on the COSIS website it will basically have each proctor attest to the fact that they've read the guidelines that they will ensure that the student verbally promises to also abide by the guide these agreements not to cheat or basically to follow all the test security rules and this agreement if you have proctors that do more than one assessment they just have to do one of these forms and it's something that you keep on file locally with your local assessment policy now we're going to get into the remote testing version of the going remote checklist we've had this type of thing for E-TES onboarding called going live with E-TES but this is the going remote with E-TES process so the first thing and extremely important is that your agency has already implemented COSIS E-TES and if you haven't already but you do want to try this out you can during this time take advantage of the time to go through the certifications and get all your staff certified and get set up in E-TES that's something that our tech support and other staff is available to help you with even during these times so please feel free to seize the opportunity to do that now if you're not already on E-TES you'll need to complete that agreement we just talked about the agency level agreement you'll need to make sure that the proctor has the equipment necessary to implement the scenario the approach that you decide to use or approaches and that ideally ideally that they're experienced in E-TES so that they will have a little bit of an easier time doing this you'll have them complete that proctor remote testing agreement then you'll make sure they know how to access the training materials on the COSIS website there'll be the recorded trainings there'll be some youtube type how-tos there'll be FAQs there'll be there'll be the detailed guidelines step by step there'll be also a quick start like a short version of that that once they get good at the using the the detailed script then we'll have like we do for E-TES the steps for testing day kind of document where they can have a shorthand version of that that they can refer to easily now that the training is not required but it is very much recommended and so it's optional but we do highly recommend that you have your staff do this training then and as far as your test takers having the technology they need or if you need to help them by providing them with some of the technology in order to make this happen registering the test stations is something that the proctor needs to do with the one-on-one solution where they share their screen or that the proctor needs to do for the student in the second and third approaches where the student takes E-TES on their own computer then you're going to review and select the appropriate test sessions and you really want to be ready with those at the time of testing so you're not holding up the process as you go to find those very important step nine is to conduct a trial run or more than one trial run with colleagues not with students but to try it out and make sure that you're comfortable with the process you really have thought through what might happen and how you're going to guide the students when you actually work with students and finally once you're comfortable with that then number 10 is go remote so now we'd like to go out and just show you the slightly more detailed version of this just see that it does and we are going to be adding to this you can see on the first one we've added the link to the going live checklist for getting started with E-TES itself so we'll be adding links to this document so that you'll be able to easily find those documents on the CASAs website that would include that second one the agency agreement the proctor agreement and number three if you scroll down a bit you'll see that there's some there's a little bit more detail about the requirements for the proctor they need a webcam of course then the training again we'll have a link to that into the various possibilities for training step five is all of this all of the different variables and alternatives for test takers so there are a lot of different ways to go and you'll have to see which ones will work again I want to apologize I know it's frustrating that it's limited to only five students or depending on what you kind of technology you have just one on one and we know that you're not going to be testing your usual 25 students in a remote proctoring mode it's simply not possible but we do hope that you'll take advantage of this opportunity to learn how to do this and prioritize the students that you think would most benefit from doing it now or that for whatever reason I think you'll have to talk internally about which students you decide to do this with and which teachers which classes I'll register the testing stations I think we've talked quite a bit about and I think the other steps we've already covered again the training tools these guidelines documents are very detailed FAQs will be forthcoming and will be ongoing we will keep adding to them as we get additional questions from the field and in terms of the training webinars there will be an initial one that will be up on the website no later than next Monday and we will also be offering subsequent mini webinars on different topics those will be rolling out starting next week to follow up on things like you know how do you do security settings what does that look like or how do you do the sharing part or what should you look for when you're monitoring and proctoring so there's there are quite a few topics that we know you're going to want a little bit more guidance on so we're going to be creating short how-to videos and short mini webinars to help you get more comfortable in different modalities for how to do these remote tests and there will be these for not only for the pre and post testing but also for the co-app assessment deliveries that I'm sure we're going to have a lot of really creative ideas for how to adapt the assessment deliveries for the civic objectives for remote testing and then also we'll have a whole set of materials also for the just to kind of reiterate that in terms of the preparation this is so important that before you're actually giving the test we'd like you to meet with this test taker it could be the day before it could be the same day confirm with them that they have the technology they need make sure that they're aware of the test procedures so they're not surprised by anything answer any kinds of questions to make them comfortable with the process especially help them understand that there will be somebody watching them and that can be uncomfortable so but if they're aware of it then hopefully they'll at least be prepared for that situation and then on the testing day although you've done things like actually beforehand checked their ID and however you do that locally and done an environment scan the room just so they understand what that's like you still do need to redo that on the day of the test because of course that could be different also in terms of checking the speed of the internet all those things you have to do again on the day of the test unless you do that that preparation session just before the giving the test which is possible easier to schedule but maybe not quite as comfortable for the students so that's up to you to see how you're going to schedule that but we do you do need to be sure that on when you're actually delivering the test that you confirm everything that you had practiced and explained about before okay then this is a screenshot that shows the proctors computing using the one-on-one approach where the proctor has e-tests on the proctor's computer and then is sharing their screen with the student you can see the webcam where the proctor can see the student can see the proctor this is also something I wanted to mention that we have the e-test sampler which is on the costus website it's free I hope you're all aware of that but if not please check it out it's called it's the practice items in all the modalities for all the test series it's called the e-test sampler and it has the all of these tests in the e-test format and these are the practice items for reading in the e-test format so this is something else you could do in the practice session with the student is to familiarize them if it's the first time they're taking a test with e-tests show them this e-test sampler show them how it works how you navigate through it and because they're going to be taking the test on their computer they're going to be controlling the arrows and and progressing from one item to the nest next when the when the screen is shared by the proctor it's actually the student taking the test on their own computer even in this one-on-one mode where the proctor has their the test on their computer the student is taking it on their computer so or on their iPad so especially I want to say that if they're taking a math test it would be really helpful for the student to see if they haven't already taken a math test how it's done any tests including the online calculator and other types of things I do want to mention that one of the recent decisions that we made is that we are going to allow for math only we're going to allow the student to use scratch paper and to use a handheld calculator if they prefer they can use the online calculator but they could also use a handheld one we we're going to restrict this but we realized it's just too difficult to take a math test without scratch paper so the the requirement is that you'll have to make sure that that's a blank piece of paper when you start the test and and see what the student has there at hand and at the end you'll have to make sure that the student tears up that paper in little bits in front of your eyes to make sure that it doesn't go anywhere else after the test so these are some of the things that are this is a very good example I would say of the kinds of things that are listed in great detail in the guidelines I see a question here about what does it look like when you're delivering the test with more than one student and I don't think today we can go through that in detail but what I can say is that in the guidelines it does go through step to step how you work it out in zoom for example they have what are called breakout rooms in other platforms they could think they call them private rooms where you can have the students all together then you can take the students into the break it into one by one into a breakout room to check their ID and work at work it out in that way so we have that all detailed out in the guidelines and yes that's how it would work all right next slide please so one of the things that is coming soon is that we are going to be able to we need to be able to identify which tests are given remotely and which tests are given face to face and we will very soon find a way to embed that in the e-test software so that you'll be able to indicate that that will enable us to compare student performance on remote as compared to in-person testing we certainly hope that with good preparation of the student that they'll be relaxed and will be as well as relaxed as you can be taking a standardized multiple choice test but that they will be able to do their best and have comparable performance as they would on an in-person test I mean they'll be in the in the comfort of their home perhaps who knows maybe they'll do better but we do want to see and also we will be as we as we always do we have ways of reviewing test security issues in the back end where we can see for example if a test was taken very quickly that can be an indication of guessing there are other data type indicators of test security problems so we will continue to do those sorts of checks as we roll out and ongoing as we do remote testing as was mentioned in the beginning there is a field a new one in Topps for Enterprise called unable to test due to force measure this is where you will be able to indicate students that you were not able to test because we know that there will be many of those and maybe I'll ask Carolyn to pop in right here and just comment on this because I want her to be able to tell you the guidance for this so the guidance for using the unable to test due to force for sure will be coming out later this week with coming from both the CDE and COSS related to remote testing we do have to work on a policy for this and that's in development but we need to we'll need to get that out to you later this week thank you okay now we can move on to some next steps just to make it clear where again kind of I think we've talked about this throughout but wanted to pull it all together by showing you that by when and where you'll be able to get all of this where is on the COSS website the there's going to be a page that's devoted to California remote testing on the COSS website and by next Monday May 11th we intend to have the following materials there one is the remote testing guidelines there would be the document for pre and post testing and government in history for citizenship there would be other documents other remote testing guidelines documents for assessment delivery of civics objectives for co-ops and another remote testing guidelines for the SIT for the going remote checklist there will be one for multiple choice tests for co-ops and SIT as well so there'll be different ones the one we saw today is for the multiple choice tests those will be there the agency remote testing agreement will be there as well as the proctor agreement that you'll keep locally on file the FAQs will be there and we'll be addressing FAQs in all the different for all the different assessments and then the resources for training will also be on this website as far as today's webinar the recording the PowerPoint and the Q&A responses will be on the OTAN website and just a minute I'm going to turn it over to OTAN so you can see exactly where that will be which I think you are familiar with the COVID-19 page and also on the CASUS website at this link is where we will have the recording of this webinar if you have technical questions you can as always contact techsupport at CASUS.org if you are interested in pilot testing with remote testing we would very much welcome that and if you would like to do that for the pre and post testing or for the government in history or for co-apps or the SID please contact your program specialist and you can talk about it with them see if it's something you really want to do and if so we can go forward and help you out to get started well in that case I'll ask Pat if she would like to make some closing comments. Thanks Linda I just wanted to reaffirm that the at this time with the COVID-19 trying to do remote pre and post testing using the CASUS assessments is not required this is a time when you might take the opportunity to get staff trained to actually try out and practice within the agency and to to build capacity and so we really encourage you to look at the long at the at the long-term outcome which is building capacity of staff in order to on an ongoing basis be able to serve students both in class students and remotely with following up with co-apps with the SID and with the pre and post testing and we are all very much aware at this time that the students lack that many of our students lack the technology in order to do this and I think Carolyn may talk about and has talked about the CARES Act and the ability to apply for funding for the technology so we really encourage you to think about this as a longer term capacity building opportunity and we're more than willing to share with you what the pilot sites are finding out we're going to be sharing best practices going forward we're going to be putting recordings up for the training that that are available 724 that your staff can access and learn more about the remote testing so we really thank you very much for on this very interesting journey with us and we are going to be working with you now and into the future so that we can really make this work for all of our students so thank you very very much for your participation today


Accept responsibility. Who here has a tendency to blame someone else? I'm raising my hand too, okay? And if you're not, we need to figure that out, okay? Who else has a tendency to complain? Who else has a tendency to blame COVID when they don't make any sales, okay? Who else has a tendency to make a lot of excuses for their lack of success, okay? I'm telling you, the moment everything will start to change for you is the moment you start to accept responsibility. You don't hit your target, your fault. You don't make a sale, your fault. The leads suck, your fault. You don't get referrals, your fault. Point goes horribly, your fault. I don't get what I want in life, my fault.  And if I don't succeed, I'm blaming me and nobody else. The moment you actually start to do that, your entire life will change. Appreciate that, thank you, thank you, thank you. I got one question, okay, as I kick this thing off. Who likes money? Who likes money, okay? Come on now, come on now, all right, all right. Okay, two more, two more, sorry, I missed you, okay. Next time, next time, my video guy wants money too. I already give him money every week. Thank you guys. So the question is, what just happened here? What did you notice? Roger just talked about take action now, right? Yes. Okay. Also, what else did you learn? Do you want to be sitting in the front or in the back? Okay. Honestly, when I go to an event, I never sit second row, ever. I don't even care how much it costs.  But I'm like, dude, I'm sitting in the front, okay. Because over these last, I would say that like, what I'm going to do, is my life has really changed the last 39 months, specifically the last 39 months. But for everything to change, we gotta, we gotta, you guys are still with me. We gotta, you guys are still awake. We gotta, come on now, Kevin. We gotta, the question is, who's going to change? Why are you going to change? When are you going to change? And at what point in our lives are we going to wake up and realize if it's meant to be, it's up to me?   I'm looking at myself in this, in this, Nate, it is Robert Graham, by the way, okay. But who else is freezing, flipping cold right now? Okay. I was going to, I was going to speak at a dresser and I'm like, I will freeze. I can barely feel my toes. I'm like, I got to wake up, right? It's cold. Who's seen the movie Molly's Game with Molly Bloom? Poker movie. So, such a good movie, okay. Such a good movie.  And they're hanging out and talking at the end and he says, I'm going to break all the rules. I'm going to give you all the answers. And I'm going to give you three years of counseling in three minutes, right? For the last few years, I've been doing a lot of one-on-one coaching. There's days where I'll walk into my office and I'll have 13 Zooms in a day. I was, it was so funny. I was telling somebody the other day in 2018, I was charging $100 for a coaching session, okay. The last coaching session I think we sold was $3,500 for 60 minutes. I just made the decision the last several weeks. I'm never, I'm not doing one-on-one coaching at all through Zoom ever again. Someone wants it, they got to come to my office. Why? Why? Why? What? All true. And time is our most valuable asset. For me to get to where I want, I've got to be focused on driving the brand, right? Driving our team, paying attention to all the little details. So, as I use money to have a little fun, okay? And if you think we're having fun now, can you imagine my breakout sessions on Saturday morning, okay? Come on now, come on now, Brad. Come on now, Pete, okay? I can tell you if you think this is crazy, wait till Saturday at 8 a.m. and 9 a.m., okay? Raise your hand if you know, without a shadow doubt, you will be at one of Cody's breakout sessions on Saturday, okay? Hey, hey, good freaking decision, by the way, okay? Because the title of that's gonna be how to go from six to seven figures with recruiting. What I believe we all need, Sam and Sam, is people. What I've learned, most people think businesses build people. I'm here to tell you that's wrong. People build businesses. I am beyond fortunate.  And several of them paid for their own way, by the way. Because I'm not paying for their ticket, okay? Why? Because, and if you paid for somebody else's ticket, I'm telling you, don't do it next time, okay? Why? Because you have to pay to pay attention. I'm telling you, the moment you start to actually spend your hard earned money, you start to actually pay attention to what's going on, okay? So when I talk on Saturday about how to go from six to seven figures with recruiting, it's because over these last 39 months, I found incredible people, right? I either solo-owned or co-owned several companies in Springfield, Missouri, with over 100 staff that will do over $13 million in total revenue this year. Just bought a climate-controlled storage unit two days ago that sits right behind our office. Why? Because really what happens is in the cycle of a business, you have a successful business, it produces cash, and then you invest that. I always miss the third piece, right? I'm 30, I invested in nothing for 10 years. I made a decision in 2021, it's time to invest. It's time to start thinking about my future, right? It's time to start taking cash and making it easier on me in the future. So when I talk about people, I'm telling you, you need incredible people. Who knows, they need, who would like a lot of amazing people around you, okay? Who would like to have a sales team that produces revenue? I think our marketing team's done, I don't know, 110 grand in revenue this week and I haven't even been there, okay?  That's pretty freaking cool.  You need people, right? You need an Andy who freaking runs things. You need a videographer. Like Perry Lunsford does an unbelievable job of getting a ton of leads and appointments and everything else, why? Because the dude is putting himself out there. Who's scared of putting yourself out there right now? Honestly, talk to me, okay? If we can't be honest here with all the craziness going on here, you can't be honest anywhere, okay? What we've got to realize is quit worrying about what everybody else thinks.  We've heard that all day today, like I am chasing my dream. Who's gonna make a decision to start to chase your dream today? Who's gonna make a decision to start to change, right? Because if everything's gonna change, I gotta change. I interviewed Brian Tracy and Les Browne and they both said the exact same thing in an interview. They had one little nugget that they both said. You wanna know what that is? You wanna know what that is? You wanna know what that is? I caught it and I'm like, oh my gosh, this is incredible.  They said, both said, you have to accept responsibility, both of them, two of the best speakers and sales trainers on the planet. And they both say the exact same thing in two separate 30 minute interviews and I had no idea they were gonna say it. Accept responsibility. Who here has a tendency to blame someone else? I'm raising my hand too, okay? And if you're not, we need to figure that out, okay? Who else has a tendency to complain? Who else has a tendency to blame COVID when they don't make any sales, okay? Who else has a tendency to make a lot of excuses for their lack of success, okay? I'm telling you, the moment everything will start to change for you is the moment you start to accept responsibility. You don't hit your target, your fault. You don't make a sale, your fault. The leads suck, your fault. You don't get referrals, your fault. Point goes horribly, your fault. I don't get what I want in life, my fault. The moment this little mental shift starts to happen, of dude, I'm in control of everything around me at all times and I'm doing my thing and if I don't succeed, I'm blaming me and nobody else. The moment you actually start to do that, your entire life will change. I promise you, okay? Quit blaming everyone else, quit blaming Nate for a lack of training or he gave somebody else the Glen Gary leads and he gave you the poo poo leads like I'm telling you, okay?  And I did it because I talked to four agents in one single morning. They all blamed 2020 failure to COVID. 2020 was the best year of my life. And it could have been the best year of your life, the moment you accept responsibility, okay? I am controlling my own destiny and you are too. Now let me get into some coaching, okay? I gotta set the mood though, I gotta set the tone and you gotta be in the right mindset before you could even start to absorb some of this stuff. True or true? True or true? You gotta stay with me awake or I gotta give out more money or what the heck's going on, okay? You with me? Okay, they may not be because they want me to give out more money, who knows, right? Hey, you guys messed up that one, okay? Who wants to be successful? Who, if I told you that the most successful people in the world do one thing every day, would you do it? Yes. Who's heard that successful people write down their goals? Yes. Okay, some of us. Who's heard that successful people write down their goals, okay? You gotta have them before you can write them. Who wrote down their goals this morning?  Who wrote down your goals this morning? Who wrote down your goals this morning? Who wrote down your goals this morning?   I write that I own a beach house, a vision jet, and a helicopter. And my team's like, why do you need a helicopter? I'm like, dude, why not? I don't know why I need it, but I'll use it, right? Perry, I'll do something with it, okay? I write that we have a four story glass tower in Springfield, Missouri, with a helicopter pad on top. I write that I own 1,000 apartment units. I write that our sales team sells over a million dollars every 30 days. I write that I help every insurance agent in the world. Like, I'm writing all these things. I write that 8% nation has 10,000 people at it. Why am I writing this stuff? Because for me to go there, I gotta go there, right? For me to go there, I gotta go there. For me to go there, I gotta go there, as you're taking notes today.  And a lot of us are floating through life. We're floating through our day. We're not accomplishing anything. And we're not setting out to make sure that we have our best day ever. Accept responsibility and write down your goals. If you wanted to, who could do that? Based on the fact that everyone could do that, who's actually going to start to do that? Hey, if I told you, you were 100 times more likely to get whatever you wanted in life if you wrote down your goals every morning, who would actually start to write down your goals every morning? I'm telling you, get anything out of the whole weekend. If you will just do this one simple exercise to start every single day, please. I promise you, you will look back in 12 months and say, holy freak, who was that dude? Or who was that chick? I can promise you that.  I had someone do this at a power day. I had someone pay 10 grand to come spend a day with me in our office in Springfield, Missouri. And I walked them through this exact stuff that I'm giving you for, well, you didn't pay me, so nothing, okay? I'm actually paying you, right? Technically, okay? And these dudes named Joe and Alan, they came in the office and we spent some time together. They were felling miserably. They were not doing well at all. I walked through this exact same stuff that I'm walking through with you and I'm gonna need to speed it up for sure. The next week, they had the best week they've ever had. And it was five times better than any week they'd ever had in the past 12 months. So should they have paid me 10 grand or should they not have? And who else should too, truthfully? If it's actually gonna help that much and you're gonna have your best week ever, your best month ever, your best year ever, would you trade 10 grand for your best week ever? Your best month ever, your best year ever, okay? Nate, I'm not pitching, I'm getting, I'm doing that in the workshop, okay? I'm not pitching at all, okay? But you should, true or true. I get a little excited and aggressive today, okay? I got my workout in this morning, maybe that's why, okay? Second thing after goals is coaching. Okay, I guess you could say third after acceptance responsibility. Who heard when we first started today that the best people in the world have a coach? Was that Bill Gates earlier that said everybody should have a coach? That's ridiculous, that's amazing, that's crazy. Ridiculous in a good way. Who's currently paying for coaching? Okay, these are some honest questions you gotta ask yourself.  My buddy, Coach Michael Byrd, will be here tomorrow, I think, right? I guarantee you'll take your money, okay? Guaranteed. If the most successful people in the world that have coaches, why don't you? If the most successful people in the world write down their goals every single day to start their day, why don't you? Right, like we play with this thing. Like we're like, dude, I want to be successful. I hear it all the time, Cody, I will do whatever it takes. And then I'm like, go do this, this, this, and this. They're like, well, everything but that. Like one of the things I do to start off my day every morning, right? Marlon does it too. Cold shower. Right? We do that separately, but we both do that. Okay. Why the cold shower? Why the cold shower? It wakes me to freak up, it gets me out of my comfort zone, right? And it forces me to do something I don't want to do.  The cold shower here is a joke, by the way. It is not cold at all. I could have stayed under there all day, okay? It was not compared to Missouri, okay? It was not cold at all. If I told you you would have better days because you spent a couple minutes in ice cold water every morning, who would actually do it, okay? Now we're gonna really say who actually wants to be successful, and who really doesn't want to be successful. Everybody says I want to be successful. I'm telling you what I do that made me successful. I'm not special, okay? Yes, I got my dad's work ethic. Yes, I show up when I'm sick. Yes, I do whatever it takes. However, there's nothing special about me. I outwork people, I get more attention.  And I write down my goals. Anybody can do that, okay? I went to Grant Cardone's $50,000 mastermind about 14 months ago. You say, well, why would you do that? The most successful people in the world have coaches. True or true? True or true? You guys still with me? Okay, I got to check in, right? Part of sales training is you do not present to someone that's not paying attention, okay?  Right? If they're not paying attention, I'm not going to go through the sales process at all. If Miss Betty's got the TV on and it's affecting her ability to pay attention to me. Hey, Miss Betty, can you do me a favor? I'm having trouble focusing. Can we turn the TV off, please? Right? I'm telling you that. There's a big sales nugget in that. I'm not proposing to somebody that's not paying attention. They're not fully bought in. They're not engaged, right? Third, limiting beliefs. There are specifically things that are holding you back from taking that next level in your life. I guarantee you that. For example, most people care about what everybody else thinks. Who struggles with that one, okay? I struggle with that one greatly previously, okay? Or, afraid to get out of your comfort zone. You can't force yourself to do something. We're at my house the other night. I'm in the hot tub and my wife's like, I'm like, babe, we just opened the pool. It was the same day we opened it. And we're in Missouri. It's still pretty cold up there, okay? And it's like, oh, it's like 10 days ago. And she's like, yeah, it's like 50 degrees. She's like, I bet you won't jump in it. I can't be a hypocrite, okay? So I go, look, it's 53 degrees. What would you have done in that moment? I made sure I got on video, but I jumped in the freaking pool, okay? Right?  Apparently I'm okay giving it away because I know I'm gonna get it back. Who knows they need to invest in something, someone, et cetera, some idea, but you're currently afraid to do it, okay? You know that you need to do something, but you are honestly afraid to do it. You need to pull the trigger. If it's gonna get you closer to your mission and your dream, you need to do it. And I promise you, a weight will be lifted once you do. I'm just telling you the stuff that worked for me. There are specifically things holding you back from taking the next level in your life and you gotta identify them. Like Nate talked about, right? A lot of it's our parents, how we were raised, our teachers, right? Our coaches in basketball or whatever else, right? Like that's a lot of it by the way, okay? That's a lot of it. Fourth, your ideal life. I'm currently designing and I'm always redesigning my ideal life. What I want, 10 years, 20 years, 30 years, 40 years, 50 years, 60 years from now.  Well, Roger's dad's 91, I'll be 90, okay? In 60 years. Have you went there?  To go there, you gotta go there, right? To go there, you gotta go there. Have you went there? Have you sat down and actually started to create what you want that to look like? How do you travel? Who do you travel with? How many team members do you have? How much revenue do you do? How much do you give to charity? I don't care what you've come up with, right? But I'm telling you, this single exercise of actually sitting down and mapping out what you want in the future and then back, taking steps backwards to get there will change the game. Because I know what I want. And it's, yes, it's freaking insane, by the way. But I got a better chance of getting there if I actually go there first. True? True? True? True? Who likes money? Okay? True? Who likes money? Okay? Okay, okay. Okay, three more. One, two, I'm sorry, I gotta give it to Joe, okay? He's gave me a lot of money, so I gotta give it back, okay? All right, give these guys a hand, okay? All right, you guys are still awake, right? We're still awake, Nate. We're still alive, Nate. Are you going to end up with what you want? Are you gonna end up going there? Are you gonna end up with your ideal life or are you gonna make excuses for why you didn't make it?  Here's a little trick that I do when I set goals, by the way, okay? You'll wanna write this down. I make sure they're extremely large to the point that they scare me, by the way, on purpose and I'm always creating short-term targets. I'm talking about, like, for example, I just did a challenge. I was raising a million bucks in 30 days, okay? Between our sales teams and what I was doing, I think we reached about three quarters of a million dollars, okay, so did we fail? I don't know, depends how you look at it, right? But why would I set it to begin with? Never done it, wanna see if I can do it, freaking crazy. I ran a half marathon last year. I ran an Iron Man last year. I'd never ran more than a mile in like seven years, okay? Why? I wanna step outside of my comfort zone. I wanna wake myself up. I wanna force me to do stuff I've never done, right? I wanna go there, okay? Thank you, Chris. I wanna go there and I know who realizes that if you're in better physical shape, it helps your ability to go make money. I guarantee him Marlon will agree, right? He said he lost like 60 pounds. Dude looks incredible, okay? He's kinda trying to compete with me with my abs, okay? And I guarantee he has more energy right now than he's ever had. Do you need energy to be successful at sales? When you're selling, what are you selling? Are you selling a product?  Are you selling you, right? Are you selling conviction? Are you selling passion? Are you selling your ability to get someone to pull the trigger, right? Are you selling your ability to help someone do something today? Are you selling a relationship? Are you selling your ability to help someone build rapport? Here's what I want you to do. What do you believe the number one quality every great salesperson has? Boom, Lori is absolutely correct, by the way, okay? Give it up for Lori. That was quick, she gets, so she said confidence, by the way, okay? Who assumed that was probably it? How you talk to yourself matters? The question is, how are you currently talking to yourself? Are you speaking positivity in yourself?  Are you walking into a cell saying, I am closing this deal today? Or are you saying, ah, I had a burger for lunch. I don't feel real well. I'm not sure if this is gonna work out. Or, also, when someone's like, I'm not interested, okay? Or, I don't have any money, or I can't qualify, or I'm busy, I love that one. Like, they answer the phone, but they're busy. Total sense, okay? Makes total sense, right? And you hear some of these objections. How are you responding? Are you responding with confidence? Are you pushing through? Or are you listening? We had a lot of new agents in here. Or are you listening to the ridiculous stuff that prospects say to you?  I'm not interested. I wouldn't be interested either. I'm broke. I wish I had more money, too. I'm in awful health. Doctor says, I could pick it up, too. I'm with you. As you're taking notes, the first step you have to do when you hear an objection is you have to agree. Doesn't mean that you totally agree with whatever they're saying, but you're not gonna get where you want by being disagreeable. True or true? Right? You notice I'm not giving false answers. Why? Because it's correct. What are you gonna be, what are you, what do you have to have to be great at sales? Confidence. I'm not gonna have you raise your hand, but who could use more of that? Right? I'm not talking about arrogance, right? I'm not talking about walking up here and showing off with money. I'm talking about having personal confidence in your own ability. Like waking up saying, for example, for those that don't know my story, I made $117,391 in 13 cents in my first eight months at 20 years old. While I was in college, while I was playing basketball with no leads, I cold called and cold door knocked because I had no sales experience. And sold life insurance, cold door knocking, okay? I didn't know anything, I didn't know any better. What did I do? I went out and just outworked people. But that first year, I made a decision instead of goal, I will earn $100,000 my first year. Are you committing to goals and targets of what you're gonna accomplish this year? Okay, we can't make this stuff up. At some point, you gotta say, here's what's happening this year, here's what I'm doing, and you're not saying I may do it, I hope I do it, maybe we'll get there. Our companies do $100 million a year. Now, do they? No, but that's how I write it. And will we? Come on now. Right? Oh, you just talked about confidence, I can't lose it. Right? I'm telling you guys, you have to believe in you. How you talk to yourself matters.  Quitting is a mindset. You should write down one of the things I do that when I'm taking that cold shower and I'm talking to myself, I talk to myself and I tell myself, I never get tired, I never quit, I'm a machine and I'm unstoppable. Why? Because I never get tired, I never quit, I'm a machine and I'm unstoppable. The question is, it's not even a question, you are also unstoppable, the moment you start to realize it. The question is, are you telling yourself those type of things? Are you thinking I may quit if I don't start to make some money? Or are you telling yourself, I will never quit. Quit plantin' that doubt in your head of I may not make it. I'm telling you, you will make it. The moment you start to say, I will make it. If you do not quit, you cannot. I'm at zero seconds, thank you guys. Anne, Anne, Anne, Anne, okay? How awesome did he do? I absolutely love this dude, okay? Once you get to know him, he's one of the nicest, the most amazing people on planet earth, okay? Also, I can tell you it's about time he's opening this up to everyone outside of just his team, okay?  The moment you start to pay attention, realize, take notes and actually become an implementer and become a master communicator, everything will change. And also, when you show up for my breakout Saturday morning, everything will also change, okay? Hey, if you love this video, you're gonna love, okay? I got behind the scenes with Jordan Belford at our conference about how he can help new agents and what he would do if he was you. Check that video out.