So good afternoon, everyone. So I think exciting day so far. And there are so many wonderful talk. I've been here since morning and listen to love talking about the security. And also Rahul talking about the complete front-end architecture, a lot of optimization they have done, the micro front-ends. And then Mohit realized us that the performance are building for one side of the community is not the only responsibility what we carry. We have a responsibility towards other people or other customer as well. And Lena talked about the micro front-ends are not only way to think about there are systems which can be done beautifully or can be worked beautifully using monolithic. And so my talk is more about building a high-performance mobile-first web applications. And before I begin, let me introduce myself. So I'm Upendra and I lead a web platform for Mentra and Jabang both. And I also lead some of the backend functions for Mentra and Jabang both. So let me walk you through with the agenda. So for any problem you saw, so first you should know the problem. And that's where I'm going to talk about the defining the problem, what we used to have in Jabang mobile experience. The next will be via PWA. There is a lot of talk about the PWA in recent times and a lot of companies has started building PWA. They have made a lovely PWA and I'm going to talk about why we decided to build a PWA. So next is early decisions and technology choices we made. So I'll walk you through the sum of the decision-making factors in our PWA implementation and the overall consumer experience. Implementation details and relations we learned. So I'll go through the, I'll deep dive basically all the details about the implementation and some of the mistakes we made and how we learned from that. And the last thing which will be more about accepting performance in the organization as a first-class citizen, how we started thinking about the performance as a go-to strategy for every release which goes to the production. So let's talk about the problem statement. So mobile traffic was increasing. I think this is nothing new to the community or I guess in past three, four years we have more mobile user in India than typical people who have computers, right? So this was a trend across the technology. Suboptimal performance on mobile, this was definitely a problem. Once you build a software and once it gets old you always have this suboptimal performance. One of the key factor of having this poor performance was desktop first design. I think when we built a new web in back in 2015 so that was a approach we took which was desktop first design and a lot of components we implemented which was mean to be for the desktop user and we carried those into mobile application. There are components which was not primarily even required for the mobile web. It was actually there in the web pages and that was creating a lot of performance bottleneck. Outdated UX, so this was another problem in what was good or what was very user friendly in back in 2015 was no longer was good enough for the consumer to engage. And so these are the four fundamental problems what we have seen in our application and today majorly I'm going to cover about the performance optimization what we did for the, when we build a Jabong PWA. So why we need a PWA? I guess one of the core responsibility for us being in the consumer domain which should allow or empower our customer to choose what platform they want to use rather than we are asking them to come to the mobile app or mobile web or desktop. And this was the go-to strategy about thinking about the optimizing the user experience and PWA is a term which is primarily for the user experience, right? All the capabilities PWA provides which is maybe you can think about having the app icon on your mobile phone or in desktop this is more about the user experience engaging user experience. That's what that's what was the thinking behind building a PWA, right? And this was the tagline which is more about the fast integrated reliable engaging user experience across the platform and any kind of a network condition, right? Be it slow 3G or a fast 3G or 4 3G or it will be a web app or desktop. So some of the early decisions, right? I think Arvind covered beautifully about the role of the people building any of the best consumer product, right? And how their startup being able to produce some of the good quality product with very lean team, right? And so that's where the people are very important when you start or when you think about any product. So having the right people for the right job with the right skills and the right mindset is very important. And one of the very fundamental thing when you build a product. So this is more about not about the code you write. This is more about the problem you solve and the technology you use to that solve that particular problem. And that's where the people mindset is very important. They're focusing more on the solving problem and using technology as a tool. Domain knowledge, customer behavior, and so these are the two important aspect when you solve a problem because once you understand more about the problem you understand your customer. You know the domain you are operating in then you would be able to take the right decisions. And this was one of the key factor when we started thinking about the design and the architecture. And we made the decisions about the architecture and technologies based on the domain knowledge we had and the behavior we have seen about our users. Microfrontense, I think already one of the session Rahul has covered about the microfrontend importance. And that's where we, and Mintra by the way has implemented this like four or five years back and when the microfrontends were not very popular. So in Jabong when we started implementing PWA we have also gone ahead with the microfrontend approach. And this was also very critical and played a big part in terms of shipping entire product in six weeks. So from the typically any consumer domain product you might get a lot of pushback building anything on a mobile web especially, right? Because the apps are already doing a lot of a good job. So that's where we have got only six weeks to push ship this particular PWA to the production. And this particular approach actually helped us in terms of building, separating the concerns and we divided entire monolithic web application to three parts. One was the discovery where most of the product search happens basically you choose the right product and the payment and the order. So in this particular session or talk I'm mostly going to talk about the discovery microfrontend app which we optimize. And the next was based on the experience in the past, right? So we have been doing this react angular a lot of technology in the last past one year, right? And we have a good enough experience in terms of how to think about the building and application what could be the right bundle, JavaScript bundle size in order to get the optimal performance of the web application. So initial thought was just go ahead with the 20 KBs of JavaScript bundle size. Some more about the technology choices. So choosing react over react, primarily two reasons. One is its size which is a three KB and second was the consumer behavior. So in typical e-commerce and discovery kind of application. So most of the things what happens user lands on any of the landing pages. He clicks on any of the banner goes to the list page where the catalog user can see multiple products. Click on the one of the product and goes to the product. So this is the most flow which is related to a navigation of pages, right? So routing kind of an application. So that's where we don't really needed a lot of optimal algorithm in terms of having a virtual DOM and very fast processing diff algorithm. So that's where the technology choices choice becomes very clear that we really want to go ahead with the very optimal or very minimalistic JavaScript size library. Then we have started building and obviously the redux was another choice for the state management. We started building with the redux. Then we realized that when we are starting using redux we need to leverage the react implementation and also the router implementation of the redux. And again, this was not the original thought we had because we really wanted to keep the bundle size as low as 20 KBs. So then we thought there is not much benefit out of using redux. Then we thought redux is generally a philosophy, right? It's a guidelines how you can interact or how you can manage state. So that's where we thought of building in-house system to manage a state. And then we have a Rota library, which is I guess 700 bytes for the state management. And we named this one JS. So I'm not going to cover more about the one JS and then the third party libraries, right? So I think using third party libraries is again carrying a lot of burden, right? So it's like, I see this as a problem. Let's say one family has four people in their home, but they think about they might be having six people or maybe once in a while or twice in a year, their parents or their grandparents or there are a bunch of relatives visiting them in summers, right? So why should we buy a bus, right? Because when we need to go out with the entire family, we should have a bus because everybody will be in the same vehicle and we can interact with each other. And so exactly the same problem. Having a library which gives you a lot of features which you might not need, right? And so that's where we would have not thought about the 20 KBs probably we would have end up using a lot of frameworks or libraries. So that's where that's hot behind that only use the bytes, number of bytes you really want to use, right? So do not become greedy to use so many features in a library. So this is the idea behind that apart from using preact and preact router, we did not use any third party library at all. That's strategy. So by the first principle, it's if we go back to the fundamentals, right? A website is a collection of web pages and a web page is a composition of styles, your markup, few JavaScripts and interaction between the elements in the web page. If you could optimize these resources, a web page is constructed by, right? So if you could optimize rendering JavaScript, CSS, fonts and images, the entire website is already optimized and we have gone ahead with the same strategy. So let me walk you through with the implementation details now. So the first thing was rendering optimization. Again, the principle was very simple that put first things first, right? So everything which is required in terms of making your page interactive, having the content which is visual to the user and making the website meaningful for the user, only ship those resources. By the same lines, what we have done, we have used some of these methods like hybrid rendering, reduced initial payload, and lining critical CSS, mark the word critical because that's where this is very important. HTTP to server push, preloading critical resources. So these are the basic methods we used in order to achieve a very fast meaningful paint. So what is hybrid rendering? So if you break your page into a web page in two parts, right? First is above the fold and second is below the fold. So above the fold, which is user is interested below the fold, they might be interested they might not be interested. So by the principle putting first thing first, so the whole focus is on ensuring that we are rendering the first fold as soon as possible. So what we did, we applied server side rendering for the above the fold content. So everything from the image to the URL, everything was built in the server, right? So we are not resolving even the image URL in the browser for the first fold. And the second fold, it was mostly a reactive strategy. If user will scroll then probably we are going to show this otherwise we don't really need to bother because user don't have no intent of looking at the below the fold content. So this was the typical strategy and this is what I mean by rendering hybrid rendering. So let us look at the code basically and this is a exact code which is doing hybrid rendering. So you could see this on demand loading which is one simple utility we have created to just make any component to follow the reactive approach. And this is the below the fold content. So anything which you see above the widget group is basically the primary content we are more intended towards optimizing. So this is typically the implementation of hybrid rendering where you have below the fold only rendered when user scrolls are shown some intent towards looking at those content. So reduced payload again on the same line. I'll just add how we are fetching the resources which is required to render the below the fold content. If you see this clusters of resources there are dots small dots right. So these are the resources only fetched only requested when user scroll from the homepage. So this is how we reduce the initial payload in order to optimize the overall rendering performance also the reduced the payload of the webpage. And then critical CSS again the word critical is important. We need to identify the critical resources right. So usually we have when we build an architecture we have a tendency to think about style guides responsive designs adding a lot of reusable Java CSS in one common file. So for example, if I talk about bootstrap as a library or as a utility right. So it has a lot of CSS components. So you have a dialogue box you have a form field elements you have responsive nets responsive grids and accordion blah blah blah so many things right. So that's where it is like generic CSS in order to find the critical that means what is required in the webpage. So for us what was required the images right. Probably the grid probably some of the text we were showing so that's where what we did. So we just put only very critical part of it nothing else right. So if I talk about so it could be the heading or copy text or some of the element which was related for the having the responsive images kind of a thing right. So identifying the critical CSS is very important. Just we need not to be very greedy in terms of adding so many things from if we are coming from the old school of designing a webpage. So server push and multiplexing. So multiplexing comes by default if you enable HTTP2 in your application you will get the multiplexing. You can see that's the same connection ID being used to fetch the multiple resources right. So in HTTP2 if you create more connection once probably you can use for multiple requests. Server push was important for us because we were using CDN and Akamai. So Akamai has the feature to do this for us. So we offloaded this to Akamai. They have their own machine learning algorithm to find out all the critical resources required in a particular webpage. So they have this URL to resources mapping and by enabling that we were able to have all the critical resources pushed from the server even before it was asked right. So this was actually very helpful because I think this helped us in terms of improving the first meaningful pain. So next is preload critical request. So I think so again so because when we talk about the critical this is again very important to have your enough context about the finding the right critical resources. So apart from typical JavaScript CSS and fonts the most critical part was having this hero banner preloading because this actually helped us to preload the banner much earlier when it was requested. And this is when we have improved the first meaningful pen significantly. If you see this particular comparison before and after in slow 3G. So you could see the when the hero banner is getting rendered in after as compared to before. So this is the impact of actually realizing that you could even preload some of your the main content or the hero banner on your web application. So next is JavaScript optimization. I think I covered a lot in terms of optimizing the boot up time because reducing the payload or reducing the main bundle size help in terms of having a good or optimization towards the load time and the pass and compile time. So by optimizing the overall bundle size the first problem was solved. So next was more about the execution of JavaScript. Right, so then we focused more on the execution which could be the runtime execution or the execution when JavaScript loaded first. So code is splitting and after they're splitting pre-caching those resources which was required in the future maybe the route level components or the component level bundle itself is pre-caching with the service worker and then optimization of the runtime JavaScript which is a responsiveness of JavaScript how soon basically you respond to the scroll event or any kind of animation which triggered by the user action. So I'm just give you a very brief overview about the one JS here. So this is a typical one JS component look like in our code base. So these are all the pure functions, right? So you could see this is a pre-act code but it doesn't seem like a pre-act code, right? And this is a pure functional programming approach where we have created a small function which is a maintainable, which is testable, which is basically reusable. So these are the lifecycle methods. We just created a different name because internally we are going to bind this with the pre-act. These are the actions. You could simply think this is the action in Redux or any kind of a similar statement management library but these are the pure functions. Event handlers, typical JavaScript event handlers. These are the properties in state. So any component in our code base can subscribe to these properties and once these property gets changed or updated, the component is going to be re-rendered. So typical anything which is a react or pre-act world scenario and this is actually the component which is we can say the presentational component, right? Because all the markups, so because I think there was always a confusion or debate on the smart component or dumb component, presentational component or business components, right? So that's where we have actually segregated. So this looks completely neat and clean. There is no markup at all. Now all the markup, all the, so JSX which is all the HTML goes in there. So let's come back to the code splitting. This is the home page, only one JavaScript bundle which is initially it was 15.4 KB but later when we did some of the more polyfills and all it got a little bit more bytes but this was the home page rendering. It just clicks on any of the banner. We are loading all the resources which is required to render this particular page. Again, even if you are able to see that even the script related to Google Analytics and click streams or any kind of analytics or third party things also coming on as part of the route transaction. Again, pre-caching of these resources. So obviously when we are leveraging the capabilities of the browser we are ensuring that we are having all the critical resources pre-cached by the service worker. Again, once user clicks on any of the product page gets loaded and the resources which is required or responsible to fetch this particular page are loaded by through the service worker. And then again, the route label and component labels code splitting are implemented across the website. This is a typical page, product page look like. So you could see the primarily three or four section in one of the template or the markup. So I will particularly talk about the bottom two which is a deferred component of the on-demand components. A deferred component basically a lazy loading of the component. Once your pages are rendered from the server side or it just first above the full content are ready and you might have some dynamic content like you. So we had like offers. So offers are dynamic. So that's where it goes in the deferred component because these are the important ingredient of the interest of the consumer. But anything which is not very primarily required part of the above the fold it has gone again on-demand. So this is again the reactive approach, right? If users is going to show some intent towards looking at those content which is at the bottom of the page then only we are going to construct that, right? And this basically worked pretty well and I'm going to show the business numbers again which shows these are the actual, these approach work pretty well for us in order to have the more user engagement and having the better business matters. So this is again the comparison of the JavaScript optimization earlier it used to be 88 KB after that vendor become 8.3 KB JavaScript before was to one 8 KB later 122 KB how this is 122 KB because we need to count Google Analytics, GTM a lot of third-party libraries like Facebook or marketing pixels, right? So that's counting for this 122 KB. 18 KB JavaScript, initial JavaScript which is responsible for making page interactive. So it's just 18 KB. Runtime performance optimization. That's again so it was more about when user click on any of the when you start interacting with the webpage we need to ensure that all the identify the long task which is taking more than 100 seconds or 200 second and breaking those long task into the smaller task. If you have studied about the real model which was advocated by a lot of blocks in Google by the Google. So this particular explains that 100 MS is the right time to respond to the any action user takes on your webpage and all the visual changes basically, right? So any animation we need to show that has been gone into the request animation frame in order to we should avoid any friction or any kind of a jank in a kind of showing animations. Instrumentation and analytics. Definitely we have users web worker ensuring that we are not blocking user action where we are not doing anything which is basically engaging the main thread, right? So keeping the main thread healthy and ideal was the go-to approach and that's where a lot of computation, a lot of heavy lifting has been done to the web worker in order to manage data manipulation, massaging for the instrumentation or analytics. Memorization of the window function. So there is a lot of time we need to call windows APIs in order to get the typically in e-commerce and the application nature of the application what we have. So we have a lot of images to show and in a single page might be a different size of images we need to show. So calculating these images, right dimension for the images, responsive images we need to get the viewport height width and so right. So what we used to do earlier, we used to call these window function every time we really needed it. So we have seen it was causing a lot of reflows when we are calling these functions. So we started memorizing these functions, right? So the next time, so once this function is getting called, it was just save all these dimensions or all the window properties which was required in the future into the memory and next time when we used to call these functions this was fetched out from the memory. So this is how we optimize the JavaScript runtime performance, right? Yeah, so image optimization. Again, this was a very basic hygiene I would say. I think in first place it should have been fixed but we were never enough not able to crack this because basically the problem with the approach where we were building, we have built entire application using desktop first approach, right? So a lot of images which was even not on the correct size. We were using the images which is primarily for main for the desktop or maybe probably for the tablets. Using the right image dimension, I think this is a need of today's any mobile web application because you have a different browser having a different kind of image optimization, right? So WebP, for example, is a Chrome but might not work in Safari but Safari has its own version of lightweight images. So a lot of these we did with using again third party because this is not our core business so we leverage a lot of functionality from the cloudinary in order to optimize the images having the right dimensions, having right types, having right quality of the images based on the network and based on the user device. Again, there are many times when we use JavaScript, we were using basically a JavaScript to construct the image URL, right? So which dimension it should be, which type it should be. So that was actually delaying the constructing the, showing up the images, right? So we optimize and minimize the use of JavaScript in order to construct the image URL before it shows up. So these are the basically the image optimization methods we tried and you could see the difference, right? I think that's what I was saying. So we were not doing a good job. 940, 54 KBs, MSB4 and 56.6 KB images after, right? So this is something which is fundamentally wrong or we did wrong basically. So this was a basic hygiene of the system what we fixed as part of the optimization. And these are the visually, so difference. So earlier in fast 3G we used to take 18 seconds after the optimization, all the optimization I talk about we were close to 3.5 second. Again, slow 3G, if you see the difference again, it's a 3X faster now, 42 seconds, no way user is going to continue with your app. Most probably it will not come back again, right? And 11 seconds probably in 3G we still have a scope to optimize it. We're still on it, right? So performance is something which is not done one time. It is a continuous process. And we are still trying to optimize it, how good we can do. These are the Lighthouse scores. You can see the suboptimal performance numbers on top and then later on these are like the perfect hundred. So I think happy ending to the entire optimization project. And these are the more detailed one. First content full paint 0.8 second, meaningful paint 0.8 second, speed index, that means the visually completeness of your website which was 2.1 seconds, 1.1 time to interactive. So this is something which is, you can see about the user experience. So probably you will like the new one which is much more cleaner, much more faster. And the left one which is, sorry, even I would not like to see that again. So the performance, first-class decision. So what we did basically in order to achieve that, right? So managing performance budgets. So we are using Lighthouse performance budget tool in order to ensure that every time our Jenkins jobs are running, it is actually not breaking the rules we have set. Webpack plug-ins also we are using. So every time developer is going to run the build, if the JavaScript size is going to exceed by 20 KBs, it will, the build will break. So user are more agile or more thankful about adding any feature they can go back and talk to product that we need to remove something or we need to think about that how we could ensure that the bundle size is not going to increase. So these are the Jenkins jobs for the daily reports. We have set up a performance testing as a release acceptance criteria. So none of the releases are going if the performance criteria is not met. So these are the basically shifting towards or the developer is starting adjusting to the first having this performance as a first-class decision, right? So nobody goes or think that if we are not meeting the performance criteria, things are going to work, right? So this is also how we started shifting towards or having a better mindset about the performance for the developer. Business mattress, there are business folks possibly they would like it. So the all greens and basically once we have completed this project in six weeks, right? And this is actually the pitch for the getting the another project where we are going to optimize and we actually got the another project where we optimize the payment app again. So I think this is important because I think everything I talk about is not something very smooth, right? It's just we tried it and everything worked, right? This is a very important thing. TTI optimization was tough. Though we have shipped the entire project in six weeks to the production but TTI optimization was good enough, no. So reaching 85 Lighthouse score was easy but it took another three months to reach to the 100, right? Even as very minute details we have gone ahead and deep dive and reaching there was actually a tough task. It's not easier, I could say. The utils trap again, I was talking about just putting up like a lot of JavaScript or the code into a single file. And generally this is what we do. A lot of common functions we put into a single file that gets heavy over the time and eventually we end up having a big JavaScript file. So this is again challenges. We need to refactor code a lot in terms of ensuring that we are reaching the 20KB size what we have thought about. Analytics and other third party JavaScript was nightmare, still a nightmare, still be fixed again. If you go to probably if you run a test again for the Jabong possibly you might not see 100 because of some third party library getting included. So this is a part of parcel of the business, right? So we need to have a revenue, a lot of metrics, things has to be done, retargeting pixels. So again, this happens over the time and we need to keep talking to our business folks and getting these optimization. So this was a challenge, still a challenge and going to be a challenge. Polyfills, again, polyfills we need to, generally what we do, we add core JS or polyfills, Babel polyfills, which is the 30KB size again. So how we did it, basically we have gone ahead to MDN, pick that particular polyfills implementation at the MDN and put that into our code base. So this is how we reduce from 30KB to 1.5KB for the polyfills. Side notes, ability to push changes to the production daily and experiment friendly environment. This was actually a big help because we were able to not going into the too many discussion, we were able to push to the production validate the concept we were thinking or the methods we were thinking to optimize the content. Ask experts for help. I guess we have done this a lot. We have consistently worked with the Google folks for past six to nine months in order to get a lot of technical advancement or the help from them. The two hours through six week product shipping was not easy. So we used to have a two hours check in daily basis. If anybody is blocked or to ensure that we are unblocking that guy or at least having them to solving the right problem. We are still learning. So while I'm saying all of these things, still we haven't learned a lot. We are still trying to optimize things. We are still learning new things and putting those into the implementation for our web app. The summary, so this is everything we did in order to achieve the numbers I have shown. So the passive listeners and probably the intersection observer I did not talk about but this is also we have the network API. So we have used the API in order to have lower quality images for the slow 3G devices so that the user is still able to see the images and can do the intended behavior. So yeah, that's it. Thank you. So I think I have just one minute. So if you guys have any question, possibly shoot at me. Hi, you talked about the container management that you guys have written in about 700 bytes, right? The store or the container management, you wrote in-house for about, with about 700 bytes. State management? Yeah. Okay. So can you talk a little bit about it, like what's inside just 700 bytes? This is, that's what I'm saying, right? So Redux is just a thought process or maybe the pattern you could use. We implemented just the same thing. Possibly we did not needed a lot of things which Redux offered you. We just have done the same thing. Having a, rather than having a multiple reducer, we have put everything in our single, all the properties in a single file and we have just had two methods which is a subscribe and a subscribe unsubscribe. So every component which is subscribing those properties were basically able to just call those subscribe method under the hood and subscribe to those properties. And internally that implementation of that react component with Redux were able to actually high level component, typically in any react application you have seen this provider thing, right? So we have implemented the provider and provider basically as the top level component which used to take care of the rendering of or calling the render function of the component which is actually subscribed to a particular component. Once the component was actually getting unmounted, we were just able to unsubscribe those components. So it was a very simple implement. It's the same as the Redux. Thank you.


OTAN, Outreach and Technical Assistance Network. Hello everyone. Again, I'm Will Natterson. The counterpart of me is Laura Flores Miranda, who's not with me today. You'll hear about barriers, and I'll explain why Laura is not here, but she has taken another job to support herself and just couldn't make it for the session today, but has been very active with me in our midterm project. So just to make sure that Laura is present in the words that are here and the slide itself. So let's see if there we go. So Tuscany Adult School is located in Orange County, California. Orange County is the small county, but has quite a different range in who we serve throughout Orange County. We're part of the South Orange County regional consortium. So we're the lower half of Orange County. Our surrounding cities are the city of Irvine and Santa Ana. To our north is unincorporated Santa Ana as well as Orange, the city of Orange itself. So, Tuscany Unified is the school district we are part of. It's a K-12 district or TK-12 district that services roughly right now about 24,000 students. When looking at our makeup for Tuscany Adult School, we currently have active for this school year, 458 students. I give last year's numbers just to comparison though we are seeing a definite drop in this year's enrollment. Last year we had about 1,239 students. So it's pretty significant and we've been fighting to have students come back on the campus or online as well. Tuscany Unified did open back up in September. We for the fall semester did not open back up in person. We had all of our classes online and then the spring semester we attempted to open some in-person classes in the evening. And even with that, the numbers are much lower. I can give you a total number for all in-person classes that we currently have. We had about a total of 28 students that are attending. So we had to clap some classes and things like that. But with that makeup of who we are for this year, our ESL population is at 262. Our citizenship preparation is at 27. Our ABE is at 21 students with a combination of a focus in reading, writing, and then in mathematics as well. And then our diploma and equivalency program is about 159. I think for this year that's been a number we've seen increase or grow compared to our ESL, which has typically been our largest component of our school, making up almost 75% of our population. Definitely a shift for this year. The age range of our students is 18 to 81 and we're pretty proud of that 81-year-old as she is taking the online learning and running with it. I don't have an image of her here, but it's been pretty amazing to watch her try and work when I've been in the classes observing. There are 12 various languages besides English that are spoken at home. On the side are all, again, pre-pandemic except for the one in the bottom corner of the gentleman Omar carrying his American flag. That was right after he passed his citizenship test in November. So very proud and sharing it with this teacher. But we had to have a special outdoor one-on-one diploma celebration and virtual celebration for graduation. And that was our way of being able to have graduates celebrate if you see the image of the graduate there. And so with that, let's talk about the makeup of our school of teachers. So we have 25 part-time teachers. Again, as I shared, we have ESL, ABE, and then our adult secondary education. Prior to COVID had some basic computer classes, but those had to stop because we didn't have access to the lab. So it's just those four strong, I mean, three strong areas that we currently are offering our classes are mornings. We separated Fridays just out because it's a special group, but mornings from 9 to noon. We had some afternoon our ABE classes, those parents have shared with us, they would prefer to come in from 2 to 5 or be online from 2 to 5 and then our evening classes from 6 to 9pm. And then with our in-person classes starting back up in spring, we have a Saturday multi-level ESL class that is going on. So that is the makeup of our teachers and just the image is one of our department meetings of some of our ESL teachers. So the makeup team is Laura Flores-Muanda. She's considered our lead. She is the instructional coach for our school as of January 2021. That was a new shift in trying to look at support for technology integration, as well as we're a new school to WIOA, bringing co-ops into play as well. So Laura's role has shifted prior to that. She was for five years our lead ESL teacher. The one who suggested, hey, let's apply. And I said, all right, let's go for it. So with that, I'm the other part of the team. I joined Test and Adult School in the 18-19 school year. So this is about my third year in adult education. And then those not present with us when we meet in D-LAC meetings, but who've been present in trying to create our distance learning plan and really trying to think of our steps forward has been support from our school leadership of Virginia Burroughs, who took over the ESL lead teacher position from Laura, and then Stacy Sevkic, who is a counselor and our lead ASC teacher as well. So they are in the background, no images of them, but definitely a part of the work that we've been doing. So this quote stands to sort of start where we were with applying for D-LAC, you know, change begins at the end of your comfort zone. So change when you're really comfortable or think you're doing well. Again, we were seeing growth prior to the pandemic. In the 17-18 school year, we had almost 1800 students that we were servicing. So we're pretty proud of that growth. And knowing that four years earlier, we were just rebuilding from flexibility and all those components. So there was a comfort zone saying, hey, we're doing well, we see some growth in our assessments. We're pretty comfortable. But you know, really to look at change and the need for things, you've got to go past that comfort zone. So with that, there was a nudge towards change. So when we applied for D-LAC, it really was to look at the integration of digital literacy skills into our teacher's instruction. Again, ESL being our core, majority of that was paper pencil in the classroom. We, our paper pencil costs us to school have not done e-testing. So it really was, our students didn't have access to a lot of devices. I bought our first iPad card when I came on as an administrator, but I'll be honest. It's up for about two months because teachers were nervous about it in students when they were brought out weren't sure about it as well. So we really, the conversation was how do we integrate these digital literacy skills into instruction to make it feel comfortable. So the classrooms themselves tech being used before this was just a document camera and a projector and a laptop to help if they needed for running our student management system ASAP. But that was the basics of what we were doing. Textbooks were all hard copies. They weren't, there wasn't a digital format of that except the one exception is for our high school diploma and high school equivalency programs, we had started to use a online platform. So for our high school diploma section we use AdminTem and for our HSE preparation, we use Aztec software. So those programs had been doing that again even prior to this application. We knew that we need to do invest more and push more but really building those digital literacy skills. The true that was a nudge, the true push to change making us really look past just digital literacy skill integration was COVID. You know in March when we were told all the school sites were shutting down. What were we going to do for our students it was right at our spring break, having to pivot and think what could we do for our students. What could we have our teachers do as their tech savvy skills. We're very limited. What did we do so we went to an asynchronous situation where we taught our teachers how to use screencastify to record lessons we put it up onto a distance learning website that housed all of our teachers so students could go and select it and watch those videos. Our teachers attempted to email students to keep an interaction or use remind for those teachers that were using remind. So we saw that but we knew from that and our teachers found through that it was really hard to to engage students. So emailing students, not a lot of response coming back remind started to drop off in the middle of April. We weren't able to again do any post testing. So there was a lot of struggle with that so we knew for 2021 that we as a school needed to be different and, and structure ourselves so we can engage our students you know, and that had to start with how to encourage enrollment. And so we had to move to an online enrollment form Google forms became our friend, trying to create that form, put it out on our website so students could register for school or start the enrollment process. And then we even had to then say, okay, what are we going to do for testing. As I shared September test and open back up for the K 12. We were able through executive cabinet approval to have limited testing come in so we started scheduling testing sessions so we had students selecting dates and times that they would come in we'd know those students coming in for contact tracing and having to follow those procedures. So that was the starting of our change in our shift. The other part of that was having our teachers learn how to use a live web based video platform for testing testing unified overall is a Google district. Our students though didn't have Google accounts our teachers very inexperienced with using Google so we taught them Google meet to try and get students to engage with them on so those are just some of those shifts that happen and made us had to understand what we were going to do and really building our distance learning plan. Also, not having Google classroom available to us and testing restricting certain components of access. We went to Google sites to try and make it a platform where teachers could house information, put up assignments for students, and even if they had some recordings that they wanted to put up, trying to give them to shop for our students to access gain information that they might have questions on the other side of that was helping the office staff be able to say well hey let's go to this teacher website. So that was another new learning component that came up with our teachers. Another quote to talk about change here. All great change were preceded by chaos. So COVID made us feel a little chaotic, it was, we're very reactive in our change, instead of being purposeful in our change and what we needed to do and I think that's where being accepted into the process of starting to go through the process has helped us. So we're not having chaos everywhere but and being reactionary, but trying to be purposeful in our change and looking forward to that. So with that ideal 101 comes in and really does it helped us focus on what change did we want not because we were responding to COVID and thinking oh we need change but being purposeful in that change so really going through the chapters of ideal one on one in that is what the stage is going to look like, how are we going to recruit what's screening what's orientation going to be really did provide for us a lens to say hey let's pay attention to those components I think we were narrow. I'm sorry I know I'm doing it into a screen but we were very narrowly focused where I deal sort of brought it open and said hey let's look at all this. And the survey data that we were completing really helped us through ideal one on one and then asking teachers questions as well and asking students access points to technology and those components really helped us. We really hadn't started talking about screening students and orientation of students. The online platforms and making sure students understood so that was a great thought process for us to really go through and then just the insight that destiny would give and provide on each of our components and our proposals really helped us. We learn I enjoyed the questions you know what about this or did you think about that and it just spurred more thought and conversation that we brought back into leadership and really creating our distance learning plan so ideal one on that was a great focus for us but the other component of that was really saying where do we want to near our goals I think we could address everything and it sort of feels like we might be addressing everything but we really in the leadership team and with staff talked about what is it that we need for our students to look at being successful in a distance learning plan, but also bringing students back onto campus. What is it they need going back to that original idea of their digital literacy skills and access to technology so as you can see these are the five areas that we've come up with to focus on for our plan and that's technology access for our learners and staff. I'm talking about recruitment again I'm getting knowledge with you like I shared earlier, our numbers are significantly down from the year before so how do we recruit, how do we start making people feel comfortable whether it's coming back into person, or being online, how do we support that, going through a screening and orientation process to really be able to say, hey, can we make some recommendations for you compared to just what the cost of score gave us to talk about your ESL level or should you be an AB class or an ASC class. We're really looking at professional development and support for our staff. We as COVID hit and what we learned from OTAN was opening up office hours was an amazing thing that we started to do to provide teachers an opportunity to come in and ask questions. But we really want to be purposeful and focused in that and I'll talk about that a little bit more. And then instruction and assessment is always there always an important component to be reflective on and know that we need to continue to build. So what we've accomplished so far in our distance learning plan so we've purchased new devices for staff so all of our staff members have a surface pro that our district adopted we've used funds to purchase that for our staff members to give them the up to the date device to be accessible in our classrooms connecting to our TV monitors. We've created an orientation video for students a simple video. Laura loves Bitmoji so you see Laura's Bitmoji down there, just explaining hey this is what you're going to need to understand to be successful as a student. And then the recruitment side of that trying to look at using Facebook Twitter and Instagram to recruit students. We had a Facebook page but it was sort of stagnant and I'm going to be honest as the leader of our school. A lot of balls and near and I think needing to know how to delegate that out so we have worked on that delegation and it started with that out there so we're getting some chatter from that which is always a good thing. And then also looking at when students are coming in. We completed their online registration come in for their scheduled assessment. We're giving them a digital literacy screener that really just ask these four basic questions. What are your skill levels on being able to work with apps, send an email be online, just accessing the computer so that way it helps us to see how we will hopefully in the future be able to provide support, but just give us an understanding to ask those questions in our enrollment form as well. But it's been interesting to see how the answers change when they come in person to answer that so it really tells us at home, completing the enrollment form that there's somebody there with them you know a great example yesterday in classes. We have a teacher doing co apps and we're using Google forms to do some of that collection and this was an in person teacher who said you know what do I do if the student doesn't have an email and that's sort of ironic because to enroll you have to have an email with us so we can send you out the scheduler to set your appointment to come test with us. So we joked about hey that is definitely somebody else who's supporting and working with that individual so they can get to that point of learning so this is what we've worked on so far. So, the other thing to talk about is the, the other trainings that we've learned from Dr Porter conversations with our coach Susan culture, and then just penny and net a year influence as well as other OTAN conversations that we've had as well. So we're able to look at you know team building looking at that creating a culture of change and handling conflict you know if you think about change. It can bring up some defensive walls to say I don't know if we can do this and trying to bring that through and then developing those communication with people informed and valuing our individual strengths you know and and looking at not only us as the D that team members, but we talked about it in leadership and then talking about with our, our, our staff as well as how can we delegate some of that work, building on the skills that that our teachers have so again that whole point of putting us together it's a great puzzle that we all come together and and can create something wonderful, but we've got to build it together so how we did that so part of that team building is really going back to norms you know norms have always been present somewhere but it brought us to back to that point of saying, hey what do our norms mean and one of them is really making sure our teachers felt like they had a voice. It's been pretty amazing for people to feel comfortable enough with the norms being the the the frame for us to exist in meetings to really hear those voices and then really talking about those individual strengths. Just letting teachers share what their comfort level is with certain digital tools of being online itself and how they're engaging their students. It's it's been pretty powerful and then communication. We have weekly communication that's going out, and then handling conflict you heard me talk about that as well. Our challenges ahead. It's a part time staff and trying to keep people engaged as you see Laura is not here. She's had to take on another position somewhere else. Funding for professional development and for technology so we can learn out more components and then building enrollment. It's a huge component for us. If you don't have the students, it's hard to keep things going. And with that our next steps, we're going to be building an onboarding workshop to build on digital literacy skills from our screeners, as well as from teachers, building a marketing presidents. We're going to start class C testing and then establish a clear distance learning class expectations. What is it that we really want distance learning to be, instead of being the actionary. And with that I close the secret of change is to focus all your energy, not I'm fighting the old, but I'm building the new and that's what we're looking forward to.


key moving forward and I think when you talk to Nets players you know they all say well we think we have the pieces it doesn't matter who's injured and who's not but I think deep down they know they've got to figure it out even with guys on the bench there have been guys on the bench that have really stepped up you look at Lonnie Walker you look at Sharp you look at some of these players that have gotten in and they have really I think done what they've needed to do in order to help fill some of the shoes of the players who haven't been able to be consistent we haven't seen Ben Simmons in a little while now now I think he facilitates the entire offense even though he's not a big scorer but not having him on the floor creates some more chemistry issues and especially defensively because he's a guy who blocks shots and does it on both ends of the floor as well so I think consistency yes chemistry moving forward will be a real key thing they've got to get used to playing together and playing well together


Yeah, okay, perfect, great, thanks a lot. Well, welcome to our session for Manager Essentials, focusing on the candidate experience. I think the candidate experience as we start to really grow as an organization is gonna be at the core of everything that we need to be focusing on, given the fact that the landscape has changed throughout the last several years on what candidates are looking for, what candidates are demanding. So a couple of things that we'll focus on as let me go into presentation mode here. And you guys, can you see the slides okay? Yep, okay, great. So looking at the agenda here, we'll quickly look at defining the experience, the importance about a good candidate experience as also what happens if we get it wrong. And then we'll talk about a little bit how hiring managers and recruiting can partner together to really focus on the candidate experience. Then we'll look at some interview best practices and then highlight how they find us and how we find them and then have a conversation, open discussion. And throughout this entire session, please feel free to chime in either on chat or prefer to even have your mic open, it would be preferable as well. So let's look at a quick definition. It starts with them researching our company and as everyone's probably aware, Glassdoor has become the norm where candidates go to review what the company is from a culture standpoint, how people like working there, what people's experiences have been, not only working there, but throughout the interview process. And we'll take a look at some of those a little bit later. But as you start to look at Glassdoor and GitLab in itself from a company perspective has probably one of the best in the industry or even from a company standpoint with a rating of 4.8. But then when you start to look at some of the reviews based on our interviews, it doesn't get so good. So we'll take a look at what some of the candidates are saying and talk about how we can help manage that. And then look at, it continues through every touch point. It's before they interview, it's during their interview and after the interview is the life cycle of the candidate experience. And then it doesn't end after the decision is made. It's the lasting memory of the total experience. So even as they come here and once they're here as an employee, we still gotta make sure that we're manning to that experience and having it be an extremely positive one. So next, I'm just gonna play, I got a couple of videos in here that I'm gonna show you as well. I believe I got to take my headset out for you all to hear it. But this talks about the candidate experience and the importance of it. When it comes to discovering, attracting and retaining the best talent, one thing we should never underestimate is the power of great candidate experience. Because if an employer gets it right, the results can be truly out of this world. And getting it right has never been more important. Competition for talent is reaching astronomical proportions. Candidates are vital beings getting on with their own lives, on their own planet, sometimes far, far away, or so it seems. And those who require their skills often don't know where to find them. But perhaps the right candidates are closer than they think. But even once the prime candidate is found, employers can struggle to understand them fully, which means reaching out to them correctly can be another haphazard mission. It is therefore beneficial to study candidates more closely, their habits, their preferences, their motivations, in order to get to know them better, communicate wisely, and win their trust, ensuring nothing gets lost in translation. This kind of research unearths facts that are critical to securing talent. For example, we know that job seeking is now a 24-7 activity. And with mobile technology, 38% search during their commute, 30% whilst on the job, 18% even admit searching when they're otherwise engaged. And although candidates consult up to 18 different resources throughout their job search, we know that most turn to their friends first for information or advice. So despite technology, word of mouth is still key and reputation speaks volumes. Either way, the candidates' experience will be directly related to new prospects. If this experience has been good, others are more likely to jump on board and referral campaigns can be great for attracting newcomers. But if the experience was bad, this can upset the shift and ties can be severed with competitors gaining advantage. So news travels far and wide and fast. Only 25% of employers regularly request feedbacks from candidates. 78% of job seekers say they've never been asked. This feedback could tell you all you need to know to improve your prospects. Although the world of recruiting has gone nearly 100% digital, the old rules work the best. Listen more, seek feedback, keep things simple, communicate often, set expectations and walk in the candidate's shoes, however different they may be. The journey will be one you'll be glad you made as getting to know your candidates can change your work for the better. So, good, perfect, awesome. Thanks a lot for sitting through that, but there's a lot that can be unpackaged there. And as you can see, things have really changed in the way we view candidates and the way candidates are viewing us. This is a slide that I shared when I did our functional group update, but really think of it as one of our eighth values because it is truly that important for our organization. And it's really gotta be, before the interview, during the interview and after the interview, and it's everyone's responsibility. It's not just recruiting, it's not just the hiring managers, it's us as an entire organization. Just like you look at diversity as everyone's organization, it's not one person's responsibility. And when we start talking about setting clear expectations, I think we do a good job of that within our handbooks and making sure that candidates are understanding what they're coming into when they start interviewing with us, but it's not setting just an expectation in writing it. It's through every step of that process. Once they're done screening with a recruiter, making sure they understand what is that next step gonna be. If they're not moving forward in that process, making sure that they clearly understand that. We are going a 100 miles an hour here, things are gonna fall through the cracks, it's never gonna be perfect, but that means more so than ever we gotta pay more attention to it. And always listen, they always say the customer's right. And sometimes the customer's not always right, but the candidate, if they're not always gonna be right either, but we need to listen to them first. And then obviously be fair and demand accountability. Accountability for us and also accountability for the hiring managers that they have to take ownership in that. These are some characteristics I won't read through them all that of a great candidate experience, but it really is responsible, it's the response and how are we getting back to them. And when I show some of the feedback that we received on Glassdoor, I think we need to take a look and sit back and say, okay, what is it that we're doing today that we need to either iterate on because it might've worked when we were a much smaller organization, but is it gonna be scalable on how we are going about our interviews and throughout our process? And how can we fix each step of the process to make sure it's time back to a great candidate experience? Are there any, I'm gonna take a quick pause, any questions, any comments? No? Okay. Then this might be a quick training session. You know, why focus on a candidate experience? And then also think about what percentage of those prospects or candidates may be our current potential customers. And that's where we wanna make sure that if they're having a good experience here at GitLab, maybe they didn't get the job, but they had a great experience. So they might take that back to their current company or to whatever organization they go and join and then potentially become valuable customers. So this next one is a video. I'm not gonna play the whole thing just a couple of minutes of it. It's actually a mighty Python skit of a bad candidate experience. So you can click and watch the rest of it, but obviously just throw a little humor in there and see what a bad candidate experience could look like. You know, I really enjoy interviewing applicants for this management training course. How is it now? Do you mind just standing up again for one moment? Good morning. Morning. Why do you say good morning when you know perfectly well that it's afternoon? Well, you said good morning. Good afternoon. Oh, yeah. Do you ring bell? Why do you think I ring the bell? This is the interview for the management training course, is it? Yes, yes. I don't think I'm doing very well. Why do you say that? Well, I don't know. Do you say it because you didn't know? So as you can see, not a good candidate experience making him feel uncomfortable. Now, can you hear me? Yeah. I have to plug it in, plug it in the back, okay. Let me just check out some chats here. Let me see how I can get to the chats. Oh, there you go, pick it up, perfect. So as you can see, made him feel very uncomfortable during that interview. And in the way we go about interviews being totally virtual and on video, one thing we want to make sure is that they feel comfortable. Even if, you know, things can happen with technology as everyone knows from clearly, even with my mic. So I want to make sure that we make that candidate feel as comfortable as possible from throughout every step of that process. So I want to throw a couple of snid bits of what I'm not going to obviously read through these. You can take your time and go through them. But this is what candidates are actually saying about us about some of our interviews on Glassdoor. So we want to make sure about the process, making sure it's not too tedious. We're asking candidates before they're even 100% interested in us that we're asking them to take some lengthy assessments, which may not be the right approach, right? We want to get their interest level. We want to make sure that we're interested in them before we're asking them to commit substantial amount of time in investment in us if we're not even making that connection point yet. You know, when we start looking back at the impression, you know, I never heard back from anyone, even when they are going through the process, even though they came back here and interviewed, the fact that we didn't give them a response back is not okay, right? I understand if people are applying and they're being disqualified right at the resume stage, but if we're engaging with them at any step in the process, we need to figure out a better way and more efficient way to get them the proper feedback. And it's not always coming just from the recruiters. Most of the right feedback that we want to get them, the constructive feedback, is going to be coming from the hiring managers. So making sure we hold them accountable to get us that feedback. Well, they completed an assessment, but they didn't pass that assessment. Well, why?  Well, why didn't I fit values? Because then they're going to take that feedback and hopefully better themselves, but also make sure that they're not putting remarks out on glass door like this. Any comments to that or questions? No? Okay, I'm going to go to the next slide. Scott, who should provide the candidate feedback? Should it be the recruiter or should it be the hiring manager? Yeah, that's what I was saying. I think it should come from the hiring manager if it got to the hiring manager stage. If it went, you know, if the recruiter disqualified them at their screening stage, then I think it should be the recruiter. But I think the most effective feedback, just like candidates are more responsive, if managers reach out to them initially to try and attract them, I think that's going to be more receptive if they receive authentic feedback from the hiring managers as well. Unless someone disagrees with that. So I'd love to see, you know, as much if not all of the feedback coming from the hiring manager when possible, because even if it's bad news for the candidate, communicated right from the hiring manager is going to make it feel like it was more honest and transparent and they had that connection. So in terms of getting positive referrals and ratings from people who maybe weren't a fit for the role, I think that the more they feel like they were genuinely connected with the hiring manager, the better. I tend to agree because I think the hiring manager can also use it as an opportunity to provide coaching when the right candidates will appreciate and may come back to us in the future. There's always those candidates who aren't interested in feedback and don't want to be better, but we don't really want those people anyway. So I think that investing in people, even if we didn't want to hire them right now, but giving them the feedback they need to maybe come back next year and try again, I think is great if the manager combines them with some of that coaching. Yeah, and then also if the candidate knows why they weren't a fit or were they lacking a certain skill, they'd make him back and say, ah, you know what, actually someone I work with would be a perfect fit skill wise. Why don't I send them over to you? So then you can start getting additional referrals. But if it's just a blanket response from our system that's saying thank you for coming in, we're moving forward, it's not going to get the same effect. So are we, do you provide that immediately? Because I find that some people are just happy to know what the status is of their application. Like, okay, I got the client and I'll move on. And some people do want feedback, but it's better to kind of separate the message from, hey, you're not moving on from, and this is the reason why. Can we, what are we doing? Are we sending the feedback out immediately or do we first say, hey, you've got the client and if you want feedback, please let us know? Yeah, Nani, if you don't mind, I'll throw that over you because you're closer to that and getting that feedback. When do we typically try to get that back to them? At the moment, because of the volume we tend to, especially if it's at screening level, we don't provide feedback, but 90% of candidates will ask and then we do provide feedback or further feedback. I have taken to Bobby's training recently that said, hey, decline right away in the call, it's a lot kinder. So I've started doing that and I do find it a lot more effective and people really take to it. And they even give me feedback in the call then to say, hey, I really appreciate that you were honest. So I actually prefer giving direct feedback in the call. At the moment, from a volume perspective, no, we're sending an email to decline, we're not giving the feedback, but we are following up if we then give further feedback. If the candidate was further in the process, I highly recommend that the hiring manager does the feedback and not us. It's a much better experience. Said as a good example, he always declines all these candidates directly. It doesn't just take pressure of me to try and waffle some excuse of why we don't want to hire the person. It actually gives like a clear answer or a clear direction for the person as to why we're not continuing. I think that you need to use some good judgment. I hear everyone's different and the report levels you build are different and the level of directness and honesty are different. So there's not necessarily a one size fits all to this. This is where we have to understand what kind of connection and relationship we have built with the candidate and what it would be most likely what they would appreciate and respect and would work best for them. I don't think you can say it's gonna be the same for everybody. And I've had different experiences where we decline and then I schedule a meeting for later to offer feedback. I've had candidates who just aren't interested at all in that and I've had some that are very interested. But if I were to climb someone and they were to ask me why, it would be silly for me to say, I'm not gonna tell you that right now. That's gonna be a separate call. It's much more organic if they want details of why that you would have that answer. Otherwise it feels like you're saying no without knowing why you're saying no. So I think that it's important to have your facts straight when you go into that decline in case you are questioned. Don't make it a debate. Don't make it an argument. It's just a matter of here's what our perception is and here's our decision based on that perception. Yeah, and I think there's no one size fits all here. Unfortunately, I had a lot of success with declining people straight in the cold, like Nadia said, people appreciate it and the honesty. But I also had someone recently totally blew up on me when I at the end of the interview, I explained why as it was declined. So it matters a lot. It matters what stage someone is in. Like the further they are along, the harder to blow. What the interview is, is it a video call or in person? In person will be harder. What kind of a pet depends on the person. It depends on the reason for declining. If the person just doesn't have the skill set, like the level of skill that you require, that's gonna be a tougher message than like, oh, you're really good, but we're looking in another dimension. We're looking for, I don't know, public cloud expertise. You don't have that. That's a different message than, look, you're not aware of modern X methods. That might be the truth, but that might be the truth that you don't wanna have at the end of that conversation, but just spread out the interview from the decline from the feedback. And I think in general, for feedback, the rules, people have to be open to it. And a good way to measure that, if they ask for it. And even if 90% asks, I think it's good for us to just wait for it. Because that 10% that don't wanna hear it, it's pretty rough on them if you do provide it unsolicited. Yeah, I agree with that wholeheartedly. The unsolicited is gonna be rough, but it can also, I think, whether solicited or not, it's important that it feel like feedback and be delivered that way, not judgment. And I think that's already a sensitive moment for the candidate and how we deliver that is also important. And I think you need to do it, as if you were talking to a friend of yours who didn't get a role and deliver that in a way that feels like genuine, honest, interested feedback, not judgment. Yeah, and I did a training. I just pushed this in the chat box. I don't wanna hijack spots training here on the whole candidate experience. I think this is probably worthy of its own training on how to give candidate feedback because there are things to avoid and there are things to lean into. I did a training for the recruiting team on this and I think it would make perfect sense for us to do something for the rest of the company as well. So we will add that maybe for our next training that we can go into more detail there on specific candidate feedback, which can be different from current team member performance feedback and things. Hey, Scott, you said we only provide like, we only close the loop if we engage with the candidate. Don't we always also close the loop if they just send in their resume and we never engage? No, no, yeah, absolutely. Yeah, so if we never engage with them, they'll get a response saying that they're not moving through the process or the resume wasn't accepted. That's more of a candid response versus because of the thousands of applications we get. But if we actually have a connection point with them, that's when it needs to be more authentic. Okay, do we have that documented somewhere where they can read it? Like, hey, if you just get a decline based on the resume, we're not going to provide details because we got over 400 applications every day. I'm not sure I can take a look and see and if it's not, we can do a handbook update for sure. Cool. Yeah. We definitely have a template for it and we are using that, especially if it's completely misaligned. Quick example, Barista applying for, now I'm going to wing it, but back in CICD, just using an example. Then we literally send that mail to say, hey, thanks so much, not going to provide further feedback. But yeah, but I think Chloe did do a handbook update. I'm going to try and find it. Okay, great. Thanks, Nadia. Yeah, and as you can see, the recruiting has changed forever. The talent crunch is getting worse. Just today in the US, our unemployment rate went down to 3.9% or 3.8%. So there's a lot more, especially the talent that we're going after, is probably a lot less, probably less than 1% of unemployment for the type of skills that we're going after. I assume that's US-specific, Scott? Yes, yes, the 3.9, that was US-specific, yep. But I think from a skill set perspective, it's probably globally, where there's probably less than 1% with the development staff is my assumption, because it is difficult to find those folks in other geographies as well. Candidates are obviously a lot more informed. They've got choices now. And candidates are picking companies and not vice versa. They're picking us. Yes, we're going out there and going after them. And I always say, go find that passive candidate. Well, with all the social media and tools out there, everyone's really an active candidate because there's so many ways to define information and opportunities. It's just a matter of, are we the right fit for them? And when we start looking at the different workforce that's coming out there with the millennials, they're gonna go out there and they're gonna do job shopping. They're gonna find that company that's gonna make an impact, that's doing things to change the world. So we gotta start, and this kind of ties candidate experience and employment branding kind of ties together in some way, because we need to come up with what's our story, what's our message from an employment brand standpoint that's gonna get candidates excited on what we are doing, because we're doing great things that can change the way people are doing development life cycles that people would get excited about. We just need to get that out, that messaging out there to them. And again, anyone can be found with all their social media out there. It's just a matter of getting the right messaging out there. And then speed is the new currency, right? We gotta move quickly right now. Our average time to fill is right about 56 days. I would like to see that sub 40. There's a lot of steps in our process that I think we can tweak up. And speed is the utmost importance here because we do lose Canada. We can probably pull some stats and I'll have Chloe pull some stats around candidates that are declining our offers, maybe because of the time it took throughout the process. So that is something we definitely need to be cognizant of and see what we can do to help it, not only from a recruiting standpoint, but also with the hiring managers. So we're looking at some tools that might be able to help with that and get it streamlined. And then once the candidate receives an offer letter, we're talking about this in our onboarding committee. How do we keep them engaged? That's not the finish line once they sign that offer letter. There's a period of time to when they still can come back and say, look, I received a counter offer. We've witnessed that this week and a candidate that we were really interested in. We have a couple of candidates that have accepted offers, but they're not starting until August. So we need to figure out how do we keep them engaged? So we're putting some strategies and thoughts together on outreach, not just recruiters doing it, but their managers doing some outreach to them as well to keep them engaged there. Scott, I would chime in on that. I think especially when you're talking about outreach, post offer signing, the managers got to be all over that. That's not recruiting's job. You've gone through this whole process to bring this talent in. You should be, if that person's really got a pretty long timeframe, you're effectively doing onboarding and engagement as you're leading up to that so they can hit the ground running. But that's the most important retention period. It is super critical. And a super critical once they start to, I heard a story recently about a candidate starting in Europe who didn't hear from their manager at all in their whole first week. Not acceptable, right? We've got to be more hands-on than that. Yeah, yeah, absolutely. So next kind of tying candidate experience into the recruiting and hiring manager partnerships, one thing, we're very reactive throughout our process.  I call it the post-and-pray model. We want to shift that in paradigm to be 100% direct sourcing, but there's got to be a real good partnership. And it starts with the job description and making sure that we're not just receiving a job description. The recruiters are reaching out to that hiring manager and having what I call an intake call. So they truly understand what it is that they're going out there looking for. What are the right industry titles? We've done some iteration on it, some of our titles that might not be what other organizations are calling them. So we're never going to find that candidate. So that's an iteration that we have to look at. What are other companies calling this position? Even though we might have a different internal title, the external posting title might be something we need to help attract that talent. And then what is our strategy from just posting versus sourcing? And develop an outreach strategy that lays out different events and not have conferences and all that. We don't do a lot of that, but we should be getting out there to some of these conferences that have that talent. And I'm not familiar with all of them, probably some of the hiring managers are, where we know these debat folks or SDRs might be going to so we can get a presence and get our messaging out there. And then schedule update meetings, not just have an intake hall and we just start working on the REC. We need to be from a recruiting team, we need to be more proactive at partnering with the hiring manager and giving them regular schedule update calls. If they're weekly, bi-weekly, whatever they may be to make sure that we're all still aligned because more so than not, RECs tend to morph over their life cycle. What it was today, and we've interviewed some candidates through calibration exercise or whatever it may be, could be a little bit different tomorrow or next week. So making sure that we're still staying aligned or if we're continuing to get candidates that are being declined by hiring managers, have a net recalibration exercise and say, okay, here's what I'm seeing in the job description. It seems like you're looking for something a little bit different. So that's something that we have to start working on a little bit more as well. And then our participation, like I said, discovery meeting, ongoing touch base meetings, pre-brief meetings before the interview, making sure that we've got the right candidate or the right interview team lined up. Everyone knows clear expectations and then trying to get them through that process as smoothly and with the best experience possible. Hey, Scott, quick question. Are there processes right now where, well, I mean, you guys answer yes, because I've been a part of some of them, that we go ahead and begin interviewing without having the meeting and the pre-brief prior to interviews? Yeah. Is that the exception or is that more common? So I would say that the pre-briefs aren't happening. We're being more reactive. I think we're getting recs thrown to us and we're just posting them and then start and manage the candidate pool from there. But I think we need to be more proactive that having those pre-brief meetings with the hiring manager and debrief meetings, right? Once we've gone through a selection of interviews, we wanted pre-brief, here's the candidates that we've interviewed this week or interviewed over the past two weeks. Let's have a discussion on who we're moving forward with on the next step. Yeah, it feels like that could just be a gate, right? You don't get to post the job until we've had the pre-brief meetings or you don't get to these start interviews. We won't schedule any interviews until we've had that. Yeah. And make sure that that's a priority. This is so important for a manager. You should absolutely have the time and interest to make sure that happens quickly to keep everyone on the same page. Yeah, thank you for that, absolutely. So, let us help you as the hiring managers. What is the target population? You're living and breathing the skills and working with the technology. So during those pre-brief flights, it's important to say, okay, what are the competitors we want to go after or don't want to go after or who's hands off? When we do executive search briefings, we have those discussions. Who should we go after? Who shouldn't we? Because they're a customer and we don't want to rock that boat. What are obviously, what are the software and technologies, industry deal names? What are the soft skills? Yes, we have a culture within GitLab, but that's not just about soft skills. It's what's the culture or the skills that the hiring manager is gonna have that we can match up with the right candidate? Because it's more importantly, it's having that right culture mix with that specific manager versus the country and making sure that they're gonna be able to work good together. And then obviously looking at employer referrals, that should be the first question we ask before we ever post-direct, is what employer referrals, what referrals do you have? Who should we reach out to your network? Even if you don't have specific relationships with them, point us in the right direction. I think we've made some great strides over the past couple of months with doing source-a-thons where we actually sit there with the hiring managers and look at your profiles online and do calibration. And then they'll say, go out there and reach it. Even the executive staff has done that. And we'll take that list and go out there. So we need to make sure that's happening on a regular basis and not just for a one-off rack, but make sure we're doing that on a consistent basis. And then I just threw some interview best practices here. You can read them at your own leisure, but I think one thing I wanna focus here is looking at the way and there is a handbook update where we're trying to get, and I think you think of it as going to an onsite interview, right? If you're going to an onsite interview, you're gonna meet with several people during that day. You might come back maybe one other time, but right now we've got time slots where a candidate might stretch out a week or two. Let's try and make it so it's a better experience where we carve out, we leave their setups in interview days, or we do a couple panel interviews, which we've seen to be very successful, at least on my recruiting team, we've done those, as well as maybe back to back. Let's carve out two hours out of the candidate's day to have interviews with a couple higher managers. So at the end of that day, the recruiter and the higher manager can partner up and say, okay, you interviewed so-and-so candidate, are we moving forward to the next step? Or even a couple candidates in a day and then make a decision at the end of the day? Yeah, I think this is super important, Scott. I think that in the past I've been at companies that prioritize interviewing as talk to all other meetings. And to me, that can be even more possible at GitLab because we talk about our meetings being asynchronous and making sure that people in different times those can participate. I also think that can apply to if right now I could have been interviewing a candidate instead of being part of this training, I could have still watched this training and I still could have gotten access to the content. But I couldn't necessarily get back to that candidate. And if we make candidates wait too long in the interview process or it feels too broken up and not fluid, it's not a great candidate experience. I think we need to do more to prioritize interviews in our schedules when we have important roles to fill. I agree, I think one of the constraints that the traditional onsite interview process at traditional companies is a benefit because someone's flying in or driving in, you sort of have to make sure you get all the interviews done. And so of course, everyone seems to find the time and you make good use of their time, but you also can communicate really efficiently across the team. I also think when you do those blocks, it's a little bit easier to make sure that you stack up the responsibilities for each of the interviews. But the interviewer has a different responsibility so that they're not getting the same interview questions over and over again. If you are coordinating a series of two or three or four, obviously you're not gonna wanna make that candidate after the same thing four times. I think that we probably are a little bit less disciplined about that when they've got four over four weeks. Each person starts setting their own agenda for the interview. Yeah, and the candidate starts forgetting about what they've already covered. The managers can stop forgetting, wait, I saw a candidate I really liked last week. Were they better than this one or not? It's been a while. I had to go back and review my notes. It's just the doing things in a tight timeframe, I think is super important when it comes to hiring. Yeah, absolutely. So during the interview, just a couple of things, obviously interview etiquette, making sure that we're making them feel comfortable, making sure that we are prepared as well with what we wanna be asking as far as questions and prepared to, you know, prepared to talk about values, talk about GitLab as well as the position. And then like I said, we just talked about managing the interview, making sure that it's clearly defined and then setting the right expectations. That is the most important is making sure that we're setting those expectations. And even if we say, look, this interview process might take a couple of weeks and you're gonna interview three or four different people over a period of time. If we set that expectation, it goes a lot further than just waiting and assuming something is gonna happen. And then the selection process profile, obviously profile match, skills, culture match. I think we do a really good job with the culture fit, making sure that career goals align and then obviously the technical skills. And then the rest is just, this is how they find us, this is how we find them with the tools that we've got internally. And we're starting to look at other ways on how we can reach out to folks and how we can engage with them. Cause it's more about engagement than outreach. We can find people as to how do we engage with them. And then so right now we're currently using as our technology, obviously lever, as everyone knows, LinkedIn. We've recently pushed out a broad campaign, well, job posting campaign and as well as ad impressions, which has generated tons of activity. I think it was last week we had upwards of 3000 applications. And one thing going back to the Glassdoor review and I saw stat, I think it was 90% of the people before they apply to a job go on Glassdoor to read the reviews. So just think about that, 90% of those 3000 actually went and looked at our reviews before they applied. So now that puts pressure on the recruiting team because now we've got to manage that flow and make sure we manage that experience because if there's applications that are not getting reviewed, we need to get communication back to them on why they weren't fit or if they're gonna move forward. And then we use Stack Overflow as well to engage with candidates. And then we're also using some diverse sites as I mentioned up here, like Power to Fly and some others. So open discussion. I know we only got a couple of minutes left. Is there anything else you wanna cover or discuss, debate, anything? Scott, can we go back to slide number five? Slide number five. I don't have the numbers here to say but when was that? Career experience, the trash truck. Oh, the trash truck, okay, yeah. So someone correctly pointed out that it will be our seventh value candidate experience since we have six values. But I don't think you should think of it as a separate value. I think you should see candidate experience in the light of our current values. So our values are collaboration and that means not treating the candidate as someone on an equal level, not as some pawn that we're moving around, but a future colleague and someone we should treat even better than our current colleagues. It's about getting results but at the end finding the right person. It's about efficiency, not only for us but also for the candidate. Not letting someone know that their state changed is these people hanging, these people wondering, they can't move on to the next thing that they can close it off in their head. So that's not efficient for them. It's about diversity and attracting a broad range of candidates and making sure that our filters don't discriminate unfairly. It's about iteration. I'm not sure how to weave iteration into it and the last value is transparency. It's about letting the candidate know what states are at and if they want feedback, give them that feedback and give it in a way that's actionable for them. No, thank you for that, Sid. That spot on, I think, yes, and I'm not sure where I got eight from but we'll change that for sure. But yeah, you're weaving it in and again, that's kind of, maybe it's to the core of our values. Candidate experience can be at some of the core of those. Thanks. Anyone else? I think to add to Sid's, I think value's point is it definitely is within our values and really every manager should be thinking about this as a customer experience. We're trying to delight each of these candidates because obviously, hopefully whichever one we select, we want to accept but all the others might be future candidates or certainly referrals to other future candidates. So we should be thinking about this as how do I get through this process and select somebody? It's how do I really make this a delightful experience and if we were to follow up with a survey to each of these candidates, they'd say, yeah, it was actually a great experience and I actually really liked, they wouldn't say that guy who interviewed me. It's like, Mike, I really enjoyed the conversations with Mike and I want to keep in touch with him. When I decline a candidate and they send me a LinkedIn request and want to keep in touch for the future and want to grab coffee someday, that's a good sign that we had a good process. God, just out of curiosity, we've been really highlighting referrals in our team calls. We've been trying to ask for those. How has that gone? Are we being successful at making every team member feel like they're a scout for this company looking for the best talent or is there more we need to do there? I think there's more we can do there and I think with pulling referral programs, it all starts out to making sure that we are reaching out to that referral and I did a survey back in Northrop Grumman days and asked, what can we do to improve the employee referrals? And some people came back all for more money but it was like less than 1%. But most of the response were, just reach out and consider my referral. Even if there's not a job today, there may be one tomorrow. I think we need to do that more often. But again, I think that's something that we need to prioritize, right? Because we're moving so fast trying to hire so many people but I think our messaging needs to be better to our employee population and how to refer people and then where are the referrals within the process? Have they been reached out to? Are they being interviewed? And I think there's ways that we can use our tools to be able to do that. Yeah, I totally agree. If there's referrals that we're not following up on, that's obviously a big mess, not only for the person that referred them but for us a referral is presumably filtered through someone that we already trust their judgment so that that should get a pretty good look. Yeah, and not I'm reaching out to the referral but the person that referred them and just saying, hey, I reached out to them. Doesn't seem like they're fit for what we've got here today. Love to keep them within our pool for our other future opportunities but thank you for that referral, keep them coming. Anything else? Well, thank you everyone for your time. This is my first training session I've done in a long, long time. So hopefully it was valuable value add to you. And if there's other things you wanna discuss around this, please feel free to reach out to me anytime. Thanks, Scott. Thanks all.