I've been in the 3D industry probably about 15, 16 years now. I've done commercials, films, AR, VR, games, back to short films again. So kind of, if it's 3D, I've been around it or done it. I'm a principal art director for the Fuzzy Pixel team. Our Fuzzy Pixel is a creative team within AWS.  We're a group of industry vets. We come from kind of all over the industry. And our mission is to use our products and services as our customers would. So we're doing real-world productions, we're creating the highest end content that we can possibly create to really battle test all of our services and make sure that they're ready for creatives when everything goes live. And we're kind of referred to as a customer zero. We hit everything first and make sure it's ready for you when it comes out. So our Fuzzy Pixel team, we produced four shorts to date. We did Spanner back in 2020. We did a couple of very short shorts. Shockingly fluffy and getting fuzzy in 2021 just to kind of wrap our heads around Blender in preparation for our bigger short, Pichu, which we put out this year in 2022. This is our main character, Mayu, and our splash art for Pichu. And then I just wanted to share something from our director, Amaro Zez, about the story of Pichu. So this is from Amaro.  She taught me the importance of using my work to contribute to society and the technology could be used with a purpose to create and transform. Connecting technology with the heart to tell the story transforms the work a team involved, its leaders, and the people who watch the story. It moves us to act in our reality and that transforms us as people. The concept of Pichu was a story that I had already been working on since the end of 2020. While writing this story, I came to the cruel realization that Ecuador is not the only country that struggles with children's education. It's a global problem. I wanted to create a story that has a great social impact. Pichu is the story of an Andean girl who will have to face the challenges of life with the support of her mother. With great pride, Pichu is based on the Ecuadorian culture using its beautiful landscapes, clothing, and Andean music. So one of the things that Amaro mentions there is authenticity. And this was really important to us. And one of the things is that we really wanted to ensure as we were creating Pichu that we stay true to the Andean region of Ecuador. And it was important that the people from Ecuador look at something that we create and say, hey, that's me, that they really feel connected to it. They don't feel like we did a poor job of representing them. We really wanted them to be proud. And you'll both see and hear how we explored the Ecuadorian culture throughout the architectural environment, clothing, and the characters. And to that end, we got Amaro's mom involved. She plays one of the characters, Kapa. She plays the mother in the film and is also our cultural advisor. So as doing anything with art, it all starts with reference. I needed to immerse myself in the cultural aspects of the Andean culture. I'm looking for, to back up just a little bit, I'm a character artist at heart. That's where I come from and then we're doing production design for the film. So I always start with character. So I'm looking at skin tones, eye, nose, and mouth shapes. You can notice things in the characters, like they get very sun, wind-burned cheeks. And these things are very important to represent in the film. We also got to learn things like there's different ponchos between the boys and the girls. The girls end up being more of a blanket that they wrap and pin together, where the boys poncho is like a whole poncho that's all one contiguous piece. And then hats and belts and all the costuming and things like that. And to that end, I created a series of 14 speed sculpts of which you can see a few here. Some of them pretty good, some of them kind of crap, to be perfectly honest. But that's important. It's just try and fail until you get it right, basically. So I started off with something more realistic as the top left one there and then tried to push and pull. And by the bottom left one there, I had something looking sort of decent, but then I wanted to push it to something more unique, created an unique style for Pichu and ended up like pulling her face horizontally, created more of a chunky round style where everything's a little bit inflated, a little bit puffy. There's no hard edges to anything, like you'll see in some of the architectural and things like that later. And I hear like much larger bevels, slightly exaggerated proportions. But we did want to have a material truth. So like the cloth feels like cloth, skin feels like skin. The eyes use refraction shaders and things like that to pick up the light properly. And then blending the style and realism together to create our own unique look. And this, if I can get it to play. This is a turntable of our characters. So Mayu there in the red poncho, Cop of the Mother, and then her friend Saewa in the lavender poncho. How do I get this to move on? Go to the next slide here. Can't you? Oh, here. Okay, thanks. You can also see how in the foliage, these are paper trees. It's very iconic to the region of Ecuador. They grow up in the high altitudes. And then how we put the style pass on that. The image on the left is sort of a less stylized paper tree where there's lots of little leaves, like it's in the minis it would normally have. The leaf colors are more warm tones. And the paper tree on the bark is a little more detailed, things like that. But when we put our style pass on it, we removed about half the leaves, inflated them, made them much more puffy. And then we even pushed the leaves cooler color. With the intention, we really wanted to be intentional with our use of color throughout the film where our character Mayu is in bright, vibrant, saturated colors of reds and things like that. And then we wanted to put her at visual contrast with the environment to make it feel more dangerous as she's moving through her journey. So we made the environment much more cool in color. Just to, one, draw your eye to the character, but then again, like I said, to put her at odds with the environment just visually. And then we went through the architecture as well. You know, everything is rounded bevels. This image on the left is actually a door that Amaro's dad produced. He handmade, because he's an architect. So as part of his thesis project, he made that. But then we put it through our style pass so it has no sharp edges. Again, we made the colors more cool and desaturated. But we still have our material through so the metal feels like metal and wood feels like wood. Another example of the house. And then the school where Mayu ends up. And then we also, like I mentioned previously, we wanted even the music and the sound to be authentic to the Ecuadorian culture. We hired local artists and singers. All the actresses in the film are from Ecuador. Our local musician here, he actually made all these instruments and played them for part of the film. And he even went out into the wilderness and recorded like the river sounds and sounds of birds and frogs and things like that. So everything we could was as authentic as possible. And then at this point, I'll hand it back to Olesky to talk a little bit about the pipeline and things we use for Nimble. Thanks. So there's the recipe for getting ovation. You just need to pass it over to Snow. So as I already mentioned, we made this short completely in the cloud using a managed service from AWS called Nimble Studio. You see in the top left corner. So it's kind of overarching the whole thing and abstracts the management of the underlying pieces. And these are depicted here. So we used different workstations for the artists. Some of them were running Linux, some were running Windows for different kind of DCC software that the artists were using. And yeah, I think some of these, Chris, we'll go into some more detail later on. We'll be milking this like ovation thing. So there will be a couple of transitions. But so before going into more detail on what Nimble Studio is and what it does, let me take a step back and look at a typical pipeline for such a project. You might have seen this before, and this is an overly simplified version of what's really happening on a project. So a movie consists of shots and you move those through this pipeline, right? So you need assets like sets, props, rigged characters, and so on. And these are created in advance. And then every shot goes from layout to animation to character fix and so on over to finally to editorial. And this is kind of linear, what we show here, but in reality there are iterations in every step, correct? So you want to make it better and better and better. And the cycle repeats many, many times. And the more iterations you put into it, the better kind of quality level you end up with. And this is an important point because the number of iterations that you can make actually depends on kind of capacity that you have in your underlying infrastructure. And it's important because, like, supporting this pipeline is technology. So it's artist workstations, storage, so for assets, asset repository, and render farm. Now that we speak about render farm, probably all of you know that, but let me just repeat it once again. So rendering is taking geometry, lights, and camera relative locations and giving it over to a machine and saying, okay, compute what it looks like from the camera perspective. And usually before it looks nice, this process is pretty time intensive. So for many films it's typical to have render times of couple hours. Sometimes, like previously we were talking about days of rendering time per frame. You'll know there's like 20 or at least 24 frames per second, sometimes more. And of course it doesn't make sense to run it on the same local machine where the artist is working, so we uploaded to a render farm. And render farms were logically the first component that studios tried moving into the cloud because it has been one of the most prominent bottlenecks in a studios infrastructure mix. If we take a typical project, a studio usually starts having a certain pre-allocated server capacity, so over on the left side of the slide before things go wild. And then they notice at some point that they run into a bottleneck of not having enough of this capacity. So artists submit jobs, at some point it's too much for the render farm to handle and all available resources are consumed and then artists have to sit around waiting for their renders to come back before they can continue with their job. Not very nice situation. And then the studio usually responds by adding some capacity, depending on the duration of a project you do it several times and then you embark on this game of catching up with the needs of the production, adding capacity and figuring out how to do it at the same time juggling other tasks that you have. So there are certain inefficiencies to this approach. So anytime the studio has too much capacity to fill, this represents the wastage resources. So these light areas where no jobs are submitted to use the capacity that we have in place, essentially this is capacity that you're paying for but not using. And when there is a utilization above capacity, this means the studio is using all their resources, which is good, right? But this could be a potential bottleneck. So artists are waiting until jobs that are kind of going over the capacity will render to continue their work. Now the situation gets more dramatic if we take a typical real-life project where everything results in a crunch. And there can be many reasons for this. The typical suspects are story being revisited or character designs being altered late in the process. Whatever reasons maybe the result is that a lot of work is delayed and then it all has to be done in very limited time because the deadline is looming and you cannot move it. So teams try to iterate as many times as possible before the deadline to bring that to the quality level that they're shooting at. So basically it's a very difficult situation to handle. Of course this was an awesome opportunity for studios to turn to the cloud and use the virtual instances as render worker nodes. This was a huge win because it worked and they began looking for other opportunities to move other infrastructure elements to the cloud too. But curious enough, the studios didn't use cloud render farm quite the way we were expecting or like they were expecting. We assumed that many would just take the peaks of high utilization that we saw on the previous slide closer to the end not necessarily in the very beginning of the project and send these extra jobs to the cloud when the capacity was no longer enough. And mainly closer to the end of the project that was happening. But it turns out that this wasn't the case. Instead, studios spread out their jobs and took advantage of all this excess capacity at all times. Practically this was resulting in more iterations taking place throughout the whole production process not only during the crunch time. And yeah, so this results in more iterations, more reviews, more nodes, higher quality overall. Now let's zoom out a little. Rendering is not the only part that you can move to the cloud as I mentioned. And essentially you can build a studio in the cloud with all the other resources as well. So you can place your artist workstations there, you can place asset repository there. You no longer need beefy workstations in everybody desk. And this can be difficult. We released a guide describing how you can build a do-it-yourself studio in the cloud and this was hundreds of pages. Not everybody has like an army of cloud architects to build and support this. And we thought in everybody probably thinking there must be a better way. And there is a better way. And this is where Amazon Nimble Studio comes in. This is a service that allows you to deploy cloud infrastructure and be up and running in a few hours. So you don't have to spend these hundreds of hours or months before you can start creating. And it's a service that was created by creatives for creatives. Back when Nimble Collective came up with their idea, they were at Dreamworks and already had a lot of experience in the industry. And throughout their careers as animators and other professionals working in the industry, they saw that many artists were struggling to get their ideas off the ground. Because bigger studios only invested in limited number of movies, but at the same time smaller studios couldn't afford the tech necessary to realize their big ideas. And this team's main goal with Nimble Collective was to enable smaller studios with the technology available to bigger studios. And in 2019 AWS acquired Nimble Collective and we took some time to make the solution well architected and enterprise ready and we relaunched it in 2021 under a new name, Nimble Studio. On a very high level, Nimble Studio provides exactly the building blocks that we were talking about previously on a very high level. I'll dive a little bit deeper later. So these are workstations, shared storage for assets and render farm. And an important point, it's all in the cloud so you don't have to invest upfront before you can start creating. If we zoom in a little, here's what a simplified architecture looks like. Let's take it step by step. So on the project you will have artists who do modeling, animation visual effects, compositing, et cetera, and they need access to powerful workstations, preferably GPU based for hardware accelerated graphics. With Nimble Studio, admin can define templates saying, okay, this group of artists needs this kind of software and these other groups need something else and then you just let people spin up from these templates, spin up instances based on what they're supposed to be doing in a project. So it's not like everybody has access to everything. And the good thing is that you connect into it using something like Nice DCV or Teradishi. So it's really bandwidth efficient and you can use the peripherals that you're used to using like your Wacom tablet and it will support several high res monitors so you are not limited to only one screen and also some specific features on the peripherals that you're using will also be supported with this kind of connection. And one note here is that we recommend placing these workstations into the nearest AWS region. I'll show the map later because latency is important. So here you want to make sure that it doesn't exceed 50 milliseconds and for luxury performance like 20 milliseconds. But you have a very good setup here in Europe with connectivity so it's workable. And the left slide of this architecture diagram shows a mix of infrastructure components that the pipeline depends on. I'll go through them real quick and on a very high level not to bore you to that but just to understand what it does. So there is storage in the form of just asset storage where your raw original assets are stored but also FSX is a file system, a cloud-based file system that you can mount on an artist workstation or a render node. A license server for software that needs it so you can run it. Active Directory and SSO are components that allow you to create users and for these users to log into the portal. I will show what the portal looks like. Deadline is our render management software which is free for use on AWS and the compute represents the actual worker knows that the deadline wins will spin up when you need to render stuff. Customized are the templates that you can create to include all the DCCs after that you need and you can actually customize it with plugins, typical plugins that you would use and then reuse this customized template over and over for all artists who need it. So what it looks like for an artist, let's say we have Carlos who is a designer on our project and when Carlos logs into Nimble Studio Portal he will see one environment that he can work in called design, logically. And to begin working he will just click launch here and Nimble Studio will spin up a machine for him with all the tools he needs to do his work. In this case he's using Blender who wouldn't. And it's important that Carlos can operate Blender as freely as he would on his local machine. Those of you who probably saw our booth at SIGGRAPH or IBC could probably also try what it feels like. You cannot tell a difference between local machine and accessing a cloud-based instance. Like you move the mouse, the cursor responds instantly so you don't feel a difference with the delay that we're talking about with the latency of 20 to 50 milliseconds. And you can use your Intuos Pro or Cintiq whatever you're comfortable with. And let's say the artist is happy with their work and now they want to submit a render and they can use the integrated deadline submitter. It's a plug-in for Blender. And what this will initiate is that you have a queue of render jobs and it will spin up the workers these will be so-called EC2 spot instances the cheapest way to run is to instance like virtual machine instance on AWS and these will only be spun up when there are jobs so they will not be running at all times and then after the tasks are finished they will be also brought down automatically by the deadline. Important point is that instances are only used and launched based on what is in the queue. So it means that you only have a render farm when there is stuff to render. This is like a departure from where we were before and this also means that when you're early on in production the farm, so you aren't paying for render capacity that you're not using yet, right? And then when there's a crunch you are able to scale as quickly and as much as you need depending on how much you screwed up with the planning. So now animation is a team sport and it's not about one person being able to work it's about multiple people being able to work together so just as we have Carlos doing some design work we also want to have Anna to do some compositing. Notice that her setup is, it has more options than Carlos because she has more types of jobs that she can handle so as a studio admin I can create multiple environments and say okay this artist only gets one and this artist can do more jobs so she gets more and these environments will have the pre-installed software and so on. And it is configured in the form of so-called launch profiles so you say okay it will have this template it will have access to such and such studio components like render farms, storage volumes, license service, etc. And artists will be able to pick between such and such sizes of an instance so you also have these guard rails of who can launch different sizes of instances. So if you know that this group only needs to work with Photoshop you probably will not give them like the beefiest GPU based instance out there. And this is what it looks like for Anna it's completely abstracted from her so she can choose machine size, the template aka the launch template, the AMI and how to connect to it. She can use a browser or she can use a native client depending on whether she does feel like installing something on her machine or not. And ultimately the underlying complexity is abstracted and Anna can focus on work, collaborate with Carlos even when they're both using different OSs they can have access to the same assets because it's all in the cloud and interoperable. And one additional perk of running a studio in the cloud is improved visibility. When you create studio resources you can assign tags and these tags allow you then to pull up reports how much resources were used for which tasks how much render capacity you were running on throughout the project. This will give you visibility of how much you're spending and which tasks cost you most for example but also it will help you for the subsequent projects you will have better visibility into the breakdown of your costs. Before I hand it over to Chris again and I want to stress one point placing your instances close to where your artists are is important, I would say crucial. Currently Nimbus Studio is offered in four regions across US and Canada. It's in London for Europe and it's in Tokyo and Sydney for Asia Pacific. So expansion is planned with the current state of things is on the map. Now over to Chris. So I'll get back into Pichu and how we use that specifically with Blender. We use Blender as our DCC backbone same way we would use Maya or anything else. We can use whatever applications are sort of best for the task at hand so maybe I use ZBrush or I use Substance Painter or something else as an asset creator because that's what I'm familiar with as an artist or our contractors that we're bringing in but ultimately it all feeds back into Blender. We did some asset creation, we did all our rigging, our animation, we made use of the new geometry nodes and we did all our lighting and rendering in Blender with Cycles. So I'll start off with the bad news first and then I'll get back to the good news some of the stuff we liked or didn't like was like some of the lighting we thought we had some missing features a lack of light linking sort of hurt us creating a stylized film like a good example is you want that very tight specular highlight to hit right in the character's eyes in a very specific spot throughout most of the film a good way to do that is to associate a light with a specific piece of geometry we couldn't find a way to do that easily in Blender so that part was difficult for us we also do a lot of things like you want to put specific rims on a face or things like that so we used a lot of cards to kind of get around it a lot of blocking cards, a lot of foliage that we turned off the primary visibility on things like that just sort of make up for that another thing was a lack of check pointing so doing rendering in the cloud we could get a frame that's like a two hour long render frame it's about 95% of the way through you're like okay great I'm almost done, I can start compositing and then the instances take it away from you because it's on spot so somebody willing to pay more on demand they take it from you with check pointing it would allow that frame to pick back up at 95% but since Blender doesn't have that it would start back over from zero so you lose a little bit of rendering time there it can be a little frustrating also for hair we moved that whole pipeline from the most part into Houdini just for the simulation and grooming we found it much easier that the grooming tools in Blender weren't quite of what we needed them to be for creating braids and the hair that we were doing in the film and then staying with the simulation theme, our clothing we moved that into Maya and used in cloth just because the Blender simulation one our artists weren't as familiar with it and two we couldn't really get the results that we wanted out of it in a short amount of time so we quickly pivoted over to other solutions now for the good stuff the geometry nodes, they're awesome we used custom node setup for scattering some of the stuff we pulled off the marketplace that really allowed us to very quickly populate the foliage and all the rocks and things throughout our sets the marketplace setups we also used our own custom geometry node setups which I'll talk a little about in a minute and then our animation one of the things that was pretty neat is you could use reference footage and the compositor to very quickly put things together so the director can see what you're thinking before you actually do the work just to make sure everyone's on the same page and you can move forward more execution instead of kind of stumbling around and trying a bunch of things also the pose library is pretty cool so we could thank you we can use the same poses on multiple different characters because we're using the same rig for basically all the same characters so we can have a key artist come in, do all the very nice poses and then hand that out to all our contractors to keep everything on model as best possible and then lighting with cycles full disclosure like I've been lighting in Arnold from Softimage and Maya and things like that working in cycles I actually found really really nice the speed and stability is often times better in Arnold in my experience and working in Maya and things like that and the shader is in my opinion way better Maya's pretty clunky and it has a lot of quirks that are just weird to work around a lot of like just tribal knowledge how to make things go but Blender was very straightforward easy to dive into and I really enjoyed that part of it so far our geometry nodes our effects artist Joaquin he was able to create our own custom setup for the snake grass that you can see in the film it's part of this area that Maya has to navigate through but he was able to make it so that you could customize the size the shape even the animation so it had to blend to where you know when the grass is touching the ground it's not bending so it doesn't feel like it's wiggling through the ground but then up at the top it's wiggling quite a bit as it's hitting the wind and things like that and then also the marketplace tools that we're able to download to really speed up our production process and then for animation I mentioned like using the compositor and the pose library has a nice easy to use interface and really smoothed out that whole process some animators have never animated and blended before but get them in get them working quickly and make the production nice and smooth and here's a couple shots working through that animation process and you can see Jason he filmed himself, composited himself in so he can act out the part of Maya and it's like is this what you'd like her to do is this the way you think she'd react and show her mom's like oh I'm coming home from school we'll check out what I did mom so it's a very nice and quick way to get on the same page make sure everybody's thinking the same thing and then you can move forward to execution so you can move forward into the blocking pass make sure everyone's cool with that and you get a bit more polish but at this point you know everything's still kind of penetrating each other with the cloth, things like that and then you add the hair and the cloth which just was a more complicated process if you want to know about it later but going back and forth between the mom touching her daughter and her cloth and the daughter touches her and the back and forth that has to happen with the cloth and animation is pretty interesting but you can see the final result here talk a bit about the lighting like I mentioned before one of the things that was really nice was the speed we rendered Pichu at 4K and for the most part it was about an hour per frame sometimes a little less, sometimes our heavier shots were closer to two hours but that's I think pretty snappy overall considering the size we're rendering things at I can mention the stability was great we had a lot of very heavy scenes you'll see a lot of foliage in there they actually opened comparatively quickly to the previous project we did in Maya where it would sometimes take 20 minutes to open the shot our shots would open in like a couple minutes and you're ready to go very few crashes you know start up the cycles render move things around, scoot it around without restarting so I could kind of see it happening live and that was a really great experience I know in Maya if you do that too much you need to save a lot because you're probably going to crash it and then the shader editor I mentioned that it's easy intuitive I found it very powerful and flexible it's like Blender has all the little Lego blocks you need to kind of if you know how you're doing it just to make whatever you need to happen like one example is the refraction shader on the eyes I was able to make it so that normally you look at an eye straight on and you know it looks fine but you turn from the side and a lot of things it will you just kind of look right through it in a stylized sort of film which is not what happens to a real eye you actually see the curvature it refracts the color and the shape of the eye at a side view and I was able to do that and Blender I thought it was pretty cool and then we also used the shader editor to mix the physical sun and sky with HDRIs for our lighting here's one of the lighting shots that you can see you can see a lot of the cards that we used in the background as blockers all those trees on the side there aren't actually rendered, they're just there to cast shadows and block lights and really allow us to craft each shot as we wanted to and then we did our compositing in Nuke I'm sorry but we went into Nuke and actually that pipeline was great because all our composites that's what we were comfortable with that's what we knew we needed to move fast but the pipeline from going to Blender into Nuke was super easy and a lot of our artists used the tools that they knew and were comfortable with and we did our final color in DaVinci just to get that last little touch of goodness make sure all the highlights are just right you're balancing all the colors with the character in the background and all those good things just to finish it off and at this point I'll hand it back to Alexi briefly to talk about moving your production to Windows don't go far away, I need you so I just wanted to say a couple words about bottlenecks and I brought a picture of bottlenecks so I have all the components so the list here is your typical bottlenecks that you run into when you're running your on-prem infrastructure at a studio but these are also kind of a list of things that people talk about when they say why you cannot do animation or why you cannot work remotely and you will find articles on the web explaining why exactly it's impossible most of these articles are pretty old so the laws of I don't know, is it Murphy's law it's no, it's another one, it's Moore's law yeah you have to know your laws so it's progressing so none of these things are showstopper anymore and I just wanted to make sure we're all on the same page about that but it's not only technical bottlenecks that helps studios back from realizing their big ideas some of the bottlenecks are kind of non-technical nature and I listed them here and curiously or like interestingly enough cloud also helps solve those so you have a bottleneck for example location talent so you're not limited to your local talent pool but you can hire from everywhere and to the next bottleneck time and money you can start small in the cloud and you can take it from there get up and running in short time you can deliver with a higher quality level with the resources that you have you're not limited to your one workflow that your existing infrastructure is limiting you to you can try out real time workflow you can try out creating for AR VR for multiple different platforms and if something goes wrong, some experiments don't work out you can just turn it down and you're not paying for it anymore, it's not standing there in your server room accumulating dust and depreciating in the process and also security so for a very long time securing your IP was the top most it's still top most but in the cloud it's doable with a set of tools that you have there and many companies who move from on-prem to the cloud they say the level of security they are able to achieve in the cloud is at least on the level, if not higher and today we talked about content production how it can be done in the cloud our focus with media entertainment doesn't stop there internally we divide it into different other areas apart from content production there is media supply chain there is broadcast, there is direct to consumer streaming read Netflix here data science and analytics from media these are different areas that we internally at AWS divide media entertainment industry into and in every of these sections we manage solutions similar to what you see with Nimble Studio it's an abstraction layer that makes it easier for you stay creative and do things you want to do without having to worry about the underlying building blocks and orchestrating those so I encourage you to check those out and there will be a link on one of the subsequent slides and over to you Chris again for the last time all right thank you Alex so yeah if you like to watch the short film Pichu it's on YouTube hit the QR code here if you'd like to please give it a look we think it's a nice looking film with a good message it took us about nine months and 23 25 artists spread over six different countries and then part of our mission as FuzzyPixel is to consistently get back to the community so we've made Mayu the main character her notebook and a tree branch available on Github so you're free to download the assets play around with her use her in Blender and do whatever your little heart desires and then if you want to know more about Nimble Studio or other managed solutions for media and entertainment you can check out Nimble Studio in our M&E stuff here if you have any questions how does it work for the licenses? we have to pay for those licenses we have our own license server in the cloud and then we just run it from there we have a distributed simulation system for the media that can use simulation on multiple machines yes and no it was a little more manual we didn't have anything that was working like we could send it to the farm to do simulation type stuff but what we could do with Nimble is we could spin up multiple instances in the cloud so RFX artists would spin up like 5 or 6 machines and then hand distribute those to all the different machines there I was also as a lighter able to spin up 2 or 3 machines and lighting can take a while sometimes so on one machine get something going just let it do so I can see what it's going to look like in the end fire up another machine get that going and it just allows us as artists to work much more efficiently the way we had it no currently I imagine if you know what you're doing though you can make that work for us we were using Slack a lot we also use Sync Sketch for all our reviews so we use like a zoom or chime call and we all get on that and we can use Sync Sketch to talk but directly built into Nimble no not currently we did have the shared storage and everything just so we can share all the same files and be looking at the same things but as far as collaboration tools we went a little bit outside of Nimble for that yeah over here we did that through Kitsu I believe that's another open source thing we had our engineers hook that up right into stuff for Blender and then we used the system where we were versioning as we went along so we have all the versions of files so if we need to roll back they're always there but then we also had an unversioned file which is what we're using to link our reference into our Blender scenes that way you can open it up and you can always have whatever the latest approved file is at this point I know with GPUs getting cheaper that's a bigger debate I think the nice thing like Alessi mentioned is that you can get rid of that GPU if you're not using it so you're kind of only paying what you're using for at the moment which can lower your costs a bit but if you're using that thing 24-7 maybe a different conversation to have probably if you're going to sell it later you need to understand like how much it will how much value it will lose until then so we'll be able to buy a new one as the better version the whatever current best thing is with that same money probably not so it's this kind of conversation there are many other aspects that we can talk about not necessarily about the cost but yeah the producer will take over my 390 Ti connection and that's it okay to see my seeds and I told them that's buy a cloud solution IP security is another point for example with remote access that we used using nice DCV it's just pixel streaming you're not downloading any assets on your local machine so basically assets don't leave wherever they're supposed to be and this is another plus so you don't have copies of your assets on everybody's machines in their homes are we are we it's usually a conversation because there's a shared security model so usually you end up in a conversation and then they go through and analyze what you set up but we have to go through pretty crazy security in order to be able to be able to be out there so that's something to do is your solutions architects sort of run through your system to make sure everything is all okay I mentioned that Nimbus Studio was revamped to make it well architected and one of the so called pillars of the well architected framework is security so make sure it's up to the standards of how you make things secure in the cloud but when it comes to be compliant with a certain standard you can do that and there's really a big bit of a security stuff oh yeah we used the limbic yeah I think for the most part the limbic worked pretty well one of the things I found was that I wasn't able to modify the limbic afterwards to do like shot sculpting so I tried to use geometry notes for that at first but I'm not the best at geo-nodes yet but it worked and I defaulted back to Maya because then you can modify that and then pump that back in yeah so if you could modify the limbic inblender I would have just done it there but I just couldn't figure out how to do it under the time constraints yeah we didn't use usd on this one but yeah there's got to be ways to do that shot sculpting work there at the end just to make everything look one last question sorry they're cutting us on time no it was not a huge problem one of the ways we worked around that is you can sort of diversify the types of instances you're getting so it's like oh maybe we get a slightly slower render but we know that you know customers aren't constantly pulling that type of instance so we're able to get our renders through that way we're able to scale pretty well I think it was sort of just an issue at first till we recognized the problem and then we sort of found work ways or work arounds to get to you know reliable renders and then we also found that a lot of those on-demand type instances are pulled during working hours and then so like 5 o'clock hits and then everything comes back and you can get a massive amount of capacity at that point and really shove everything through but I think we're out of time at this point yeah so thanks everybody for showing


Frightfully Forgotten merchandise is available now. T-shirts are available in five designs. The classic design, the retro design, VHS Forever, Keep Drinking and more. Jason Voorhees wants one. Freddie just loves his. You can wear yours as you collect your VHS. Play the guitar, bust some ghosts and drink. New for 2022 is the Frightfully Forgotten the drinking continues design in which you can mentally prepare and gather the courage to shovel snow and of course drink. Also available is the Frightfully Forgotten soundtrack. While listening to, you can vacuum the carpet, dance with a skeleton, smell your VHS and of course drink. So click the links in the description below and get your Frightfully Forgotten merchandise now so you can stay warm, look cool and of course drink, drink, drink and drink. So until next time, keep buying and keep drinking.


My name is Karan Nandwani. I'm a PM on the NuGet team. Today, we'll be looking at how you can secure a NuGet package ecosystem. In the recent past, there has been an increase in the number of supply chain attacks, and they have made the news. We have seen that there has been a propensity for using package managers as the mode of attack for some of these things. Today, let's recap some of these incidents and see what we can learn from them. So, Homebrew is a package manager that you can use to install software on Linux and Mac OS. There are many use cases here such as setting up a DevBox really quick with the essential tools that the new developer might need. Now, in this case, sometime last year, a security researcher was able to get access to the core Homebrew repositories. They had commit access to these repositories, which means they could have made a change, and that change would have flown to anybody that ran Homebrew update or recently installed Homebrew, and they would have gotten their malicious formulae. Now, the takeaway really here is that there was no way for the end user to protect themselves. So, really what we are asking is the tools that package manager should be able to detect if a package has been tampered and block installing such packages. In addition, there should be a way for package authors to stamp the identity on a package, and on the other end, for a package consumer to be able to validate this identity and confirm that this package truly came from who it says it came from. In a separate incident last year with NPM, the NPM account of the maintainer of a really popular package was compromised. The attacker went on to publish a malicious update that got into 4,500 accounts and got the credentials before it was taken down. Now, we should understand that this was not a fault on the NPMJS side. The attacker was able to get the password and username for the maintainer from a third-party leak and was able to reuse it to gain access to the maintainer's NPMJS account. So, the clear takeaway here is that username and password security is simply not enough. In a separate incident, an attacker uses a different vector to target package users. They published several packages that looked very similar to a very popular package which has over 2.5 million weekly downloads. These were all malicious packages that harvested environment variables from the victim's machine, and this attack went unnoticed for nearly two weeks. So, the key takeaway here is that popular repositories should be able to detect and block typosquadding attempts. In addition, there should be a way for users to define which publishers they really trust, and if a package does not meet that specification it should fail to install. Now, let's take a quick look at what does a typical NuGet workflow looks like and what are some of the best practices. So, let's say I have an awesome library that I want to share with the world, and the obvious next step there would be to create a NuGet package. The first best practice here would be to author sign it. Now, author signing it allows me to stamp my identity on the package, and on the consumption side, it allows the consumer to validate that this package really came from me. Next, I want to publish this package to, let's say, NuGet.org. Last year, we NuGet.org adopted MSA-based authentication which allows users to opt into 2FA. Now, setting up 2FA is really easy. It is literally three steps after you log into NuGet.org. Now, if you are an enterprise customer that uses Azure Active Directory to log in, you can ask your admin to enable 2FA and it's just that easy. As we have seen, enabling 2FA would have prevented several of these attacks that we looked at. The next step here really is to publish this package to NuGet.org. Now, when the package is being pushed to NuGet.org, we actively check if this package ID looks very similar to an existing package, and if it does, we block it from publishing at that point itself. After it passes that, and it does not look similar to an existing package, it undergoes several checks to see including virus scanning to ensure that this package is safe for consumption. Once it passes these checks, NuGet.org then signs it with its own repository signing cert. This way, you can be guaranteed that this package is truly coming from NuGet.org, even if you are using it through let's say, ADO using an upstream source or this package are rehosted on an internal feed, etc. Now, let's look, this was the publishing side of things. Let's see what we can do when you're consuming these packages. So, to begin, we have something called as client trust policies. This allows a package consumer to define policies that say I want to trust a package that has been authored by a certain author, or I only want to trust packages that come from a specific feed, let's say NuGet.org, and you can get more specific by saying only by these specific users or accounts. Let's take a look at a quick demo of how you can do this in your day-to-day flow. So, I have a solution that is NuGet.Cleaner. I've been working on this site project for a while now along with the new PM on the team, and I want to implement client trust policy. The first thing that I did here was add a new NuGet.config file, and this file is at the solution level. This is necessary to isolate the settings that I want to apply to this solution from the other solutions that I might have on my machine. So, the first thing I really want to do here is start validating signatures when the package is being installed. The way you would do that is by setting the signature validation mode to require. What this does is every time you install a NuGet package to any project in the solution, it will check that the package has a valid signature. Now, this happens at the time of a solution which means when the package is being extracted, it checks for this signature and then that point onwards that package assumed to be safe. Hence, it is a best practice to add a new global packages folder which is exclusive for the trusted packages that you want to use with the solution. The way you can do that is simply by using by adding a new key called global package folder and defining that folder. Now, let me quickly close the solution and reopen it so that the new config can take effect. All right. So, let's try to install some NuGet packages into this project. I'm going to try to install NuGet and sort of JSON and what do we have here? The package failed to install, although it has a valid signature. The reason for that is although we said we want to validate packages on the solution, but we did not really specify what packages or sources we trust. So, the next thing to do here is define NuGet as a trusted package source. The way you can do that is using the NuGet trusted signers command and you can say add a new trusted signer and say, NuGet at org and what this does is it'll go out to NuGet at org, figure out what is the right fingerprint for the packages that are signed by NuGet at org and insert in your config file. So, I'm going to once again reload my solution for the settings to take effect, and let's try to install that package again. As expected, this package now successfully gets installed. But what if you told me that I want to have more fine-grain control over which package owners on NuGet at org I trust? You can easily do that by specifying owners, and in this case, I'm going to say I trust James and K and I trust any package that is published by Microsoft. Awesome. So, let's reload our solution again, and we're going to try to install a bunch of more packages. So, I already installed Newton's out of Jason, I'm going to try to install this one for Microsoft. It should, since tells me license acceptance, goes through perfect. Now, what if I try to install a package which was say something like this, castle.core and as expected, it did not get installed because although it has a valid signature, it is not in our list of trusted signers. So, to summarize, client trust policies allows you, guarantees you package authenticity and integrity. That is, you're guaranteed that this package truly comes from who it says it comes from, and that the package has not been tampered with after it was created or published to NuGet at org. Now, in addition to client trust policies, you can also use lock file. What lock files do is they store the hash of the package content, and when a package is being restored, it validates this content hash against the package which is being restored. If the content hash does not match, it fails the restore and ensures that a bad package does not find its way into your NuGet dependency chain. To close the loop here, we have hosted a first-class and highly available symbol server on NuGet at org. This allows package owners to publish symbols using the new SnubKec format. The new SnubKec format uses a secure portable PDB for symbols. This also allows you to securely debug your NuGet packages. To make things easier, NuGet at org symbol server is listed as one of the default sources in Visual Studio. The added security here is that the DLLs in the NuGet have a checksum for the PDBs, which means that when you're debugging, the Visual Studio debugger will actually validate that this checksum matches and provides for a secure debugging experience. Now, let's take a sneak peek at what we have coming up in the next few releases. We will be integrating with GitHub token scanning service. What this means is, if you inadvertently checked in one of your NuGet at org API keys, within seconds, we will be able to detect that and disable that API key and also notify you. This will minimize the potential for malicious use of any API key that you might have leaked. Also, we are working on a new feature that helps you identify and flag vulnerable packages anywhere in your dependency graph. This includes any packages that you have taken, any packages, this includes your transit dependencies as well, and this will allow you to proactively update your dependencies to the ones that do not have these vulnerabilities and ship more secure code and applications to your customers. So for anything that we looked at today, you want to know more, go to AKMS slash secure NuGet, and we have links for everything there. That's all we have from the NuGet perspective. Thank you very much. All right. I'm excited to be here, Mika. Me too. So we have just come into frame because I was seeing some Spice Girls stuff over there, and they're like, hey Seth, the director's like, you need to go in there right now because you're not a very good singer when it comes to Spice Girls number one, and number two, you're kind of working and you're like, Seth, let's get in there. Yeah. Maybe do something. Yeah. All right. So what do we got here for a new post? Looks like we are done for today. Thank you so much for that. Thank you. The cool thing about NuGet, the cool thing about NuGet that I like is that it allows you to share code, but that's also the bad thing about NuGet. So it's good to have all of this infrastructure to make it a little safe. Do we got anything else in there? Yeah. Ryan actually is talking about securing NuGet packages that build super pumps about this session. Oh, so fantastic. So securing NuGet, like I said, the cool thing about NuGet is that you can share stuff, but that's also a bad thing about NuGet. Okay. So look, we are done in Studio A for today. Is that right? Yes. We are. Okay. So what we're going to do is we're going to now toss it over to Studio C. Yeah. So you can still ask questions on the.net.com hashtag, but I'm probably going to go home and have a dinner and then maybe go to sleep or something. Yeah. Take a nap, sink some more of Spice Girls. What we're going to do is we're going to have Jeff come in here because right now, come on over, dude. As he's coming, we're going to toss over to them. You had the party last night too, right? The after thing? We had the after party last night. Okay. Okay. Yeah. The after party party. Okay. And so were you wearing that? No. Okay. So now this time, we're going to have like an extra party. So it looks like you're muted right there. So why don't you talk into the small of my chest here? Hi, Seth. It's so good to see you. How are you, buddy? Over so that I'm not muted. Oh, now he's on. Am I on? Can you hear me now? Yeah. No, you were on before, but not with the mic. I see. We're professionals. So let's go show them where Studio C is. Let's go for a walk, Mika. All right, yeah, let's do it. So what we're going to do is we're going to go out here. I'll open the door for you because no one ever gets to see Channel 9 Studio the way I do. No. So this is Channel 9 Studio. Yeah. That was off. So let's come over here. Wait, come here. I want you to meet someone. Come here. Come here. Come here. Closer. Closer. This is Beth Massey. She is the one that did all of the work. We just stand around and look pretty. Here, look, speaking to this right here, tell... I love you all. Thank you so much. We had such a great event. And we adjusted the mic so that she could literally talk right into it. Right, do you see this? This was all planned by our wonderful studio people. So we're going to have you back up here. Let's come over here. Let's go for a walk. Let me show you all the buttons that they push. I'm going to try to push some buttons and if anything happens, just let me know. This is where all the buttons happen, right? This is the director here, Cameron, Cam Cam. Golnath tells us what to do and we do it mostly out of fear. So there is that. And so these buttons, she says, never to push, okay? This means that we're recording it, right? Come over here. So that way, at the end of 24 hours, all of this stuff, all of the hot little bits are going to be in your hands as well. So should we go back to studio C? Let's take a walk. Come on, let's come this way. I think it's about time. Let's come this way here. So you don't want to turn it back. Is that awkward for you? No, no, no. I got to come over here. So this is studio C. Obviously it says warning, anything in this room can be live stream. So we're going to go in here. So let's open it up. Here we go. Okay. Hello. Fantastic. Oh my goodness. This is now where the magic's going to happen afterwards. So why don't you come in here? Join us. Very good. Oh my gosh. So we're about to go live, live, 24 hours straight on Twitch, twitch.tv slash visual studio. If you're out there, if you're watching on Twitch, click that follow button. You want to follow along. You want to know what's happening because we've got lots more content after .NETCon for you. But we've got our crack team of hosts all throughout the day.  So they are telling me to wrap in my ear because I have a special earpiece that makes me get into really special places all the time. It's really fantastic. So we're going to wrap it up here with the .NETCon from Studio A. Oh, they tell me to keep talking. So Javier here is the other mastermind that you may not hear. Let me scoop down so you can talk into it. He does all of the website stuff and he brought up a beautiful dashboard. You're welcome. Hold on, let me get closer. You are welcome. Thank you. Thank you. So here's how technical we are. Look at this. Double keyboards. Two keyboards. Because I have to. This is a two keyboard show. It is a hunt. It's great. And there's more buttons over there. Look at all these buttons. There's buttons. Don't push this one. No, they told us that this will cause issues. It is not good. There's literally an X there that says don't push it. But I kind of want to push it. All right, no. So before we go, we'd like to say thank you for those that were with us in Studio A. We're going to turn it over to Studio C after a short little break but go to twitch.tvfrontslashvisualstudio to continue all of the joy and the fun. Make sure you tune in. See you there.


So just to give you an overview of what we're going to spend the next 25 minutes talking about. Our big question is thinking about how do you actually evaluate deeper learning? Forget about evaluating the things that are really easy to measure on a bubble test, but if you're really interested in deeper learning, how do you start to measure that in a meaningful way in a rich online learning environment, not in a curriculum unit, but actually an experiential learning system? So we're going to use vital signs as a case study for this. I'm going to start by talking about what our learning goals are for vital signs. What are the actions and the evidence that students take and students lead through their participation that we could use as raw material to mine to look at deeper learning? And then what we really want to do is have a discussion with you all, looking at how would you approach evaluating deeper learning in vital signs or in another similar learning environment? And how do we address this challenge? Because it's a common challenge that, from what I know, nobody's really cracked successfully. So vital signs, and I should say I'm from the Gulf of Maine Research Institute, and in Maine we have an extraordinary asset. There's a statewide Maine Learning Technology Initiative that's given a laptop computer to every seventh and eighth grade student and teacher, and about 60% of our high school students have them as well, as well as all the teachers. So we're leveraging that infrastructure and creating a statewide citizen science and science education program. So we have, it's an open system, anybody can participate who wants to. We have special resources and put a special focus on educators, both in formal education and outside of formal education. And then we're also working with scientists. We're working with about four dozen scientists in the state who are interested in the question that we're researching, which is invasive species, where are they and where aren't they? What kind of impacts are they having in their habitats? And that's a question that scientists really need help answering, because they can't be everywhere at once. But it turns out you have more invasive species problems, you have more people, and where you have more people, you tend to have more students. So it kind of matches up quite nicely. So what are our learning goals? Well we're really interested in seeing our participants gain scientific knowledge, scientific thinking, and developing skills with using evidence and using data to back up claims. We're interested in collaboration and communication skills. Those are things that show up on the Hewlett Deeper Learning Rubric, as well as in the framework for the next generation science learning standards. And they're not things that you can easily measure in a bubble test, but through participating in vital science we have lots of teamwork, there's lots of collaboration both between peers, but also among the peers, scientists, and the community at large. So there's an opportunity to start looking at communication and collaboration, so we want to do that. And then of course to look at content knowledge, and not just do you know some more facts and figures about invasive species, but are you learning principles about ecology, biology, and biodiversity, and even something about the science practices, which are giving new importance in the framework for next generation science standards, along with engineering practices. So again, the question is how do we measure these things in a rich, experiential, online learning environment? So to give you a sense of kind of what the vital science experience is like and what kind of resources we have that we could start using to measure, I'll just run through kind of a standard in quotes experience that a student or learner might have. Vital science is designed to be open and different educators, different learners, different people use it very differently. So there's no real one way that people use it, which is very much by design, but it is a particular challenge for starting to measure impact on learning. So the typical way that it happens is that participants start out with a question. And it might be a question that they have about their local community and invasive species. More likely they're going to turn to one of the field missions that we've developed with some of our participating scientists. And we've developed these for two reasons, one, because the scientists really wanted useful data, and two, because educators were having trouble coming up with investable questions. So they start over the field mission. They learn something about the species that are involved. They learn some background information about why is it important. And then they go outside and they actually look for that species. And they look to see if it's present or if it's not present. And they collect evidence to back up that claim. They collect pictures and they write evidence statements. They also make general observations and they get a GPS location, all of our data is geo-referenced. So they make a claim. They did find it or they didn't find it. And then they upload their observation and all that data onto our website. So this is kind of a zoom in of what one of these species observation looks like. There's a field note not required until we often see them. There's a sketch, again, not required. But we often see them and they're often really wonderful. And then they are required to have at least two pieces of supporting evidence but they can have up to three. And that's a pair of picture and a written statement. And a couple things happen as soon as an observation is posted. One, the species expert who's the designated person who's most interested in that species gets an email. And that's one of the ways that we've incentivized their participation is they get an automatic notification when there's a new observation of the species that they're interested in. And then the other thing that happens is that it's immediately public and anybody signed into the system, including that species expert, can make a comment. So we're starting to see these great conversations happening around some of the data and collaborations as well in those conversations. So just to dig in a little bit deeper, can you all read that or should I read it out loud? Let's just, you can read it. So here's a field note. It's fairly typical and there's a sketch as well. And what's fun is that this field note was posted and then the scientists who's the species expert actually picked up on the fact that they included a sketch and kind of validated that they had used it and the importance of using sketches and mentioned why he thinks it's important and then also pushes them with another question, like did you see any evidence of seeds or fruits? So this is kind of typical of the conversations that happened between student teams and our scientists. We also see conversations between student teams. So in this instance, there was a team, I think it was the invasive destroyer, I don't know which team it was. Anyway, they'd gone looking for Oriental Bitter-Sweet. They hadn't found it. And another team, Babby from another town, no connection in real life, saw this activation. They'd looked at the same species and found it. So they chimed in with the way that they've been able to recognize it. So we're starting to see students exercising their new knowledge and sharing it with each other, which is really exciting. We all know if you want to learn something, you teach it, right? So those are some of the ways that students are communicating and beginning to collaborate with each other. We also have a place on the website where they can post final projects. So some of the things that we've seen on the laptops that all the computers have is iComic and they've made some really incredible comics that bring the species to life and their interactions with their environment, interactions with other species into life in really fun ways. And even posting videos. So in this picture, you see a young woman demonstrating the use of a Lake Aquatic Species sampling tool in a video, so she's sharing what she learned. So the other thing that we're interested in is content knowledge and thinking about how do we actually look at this across the whole system and thinking maybe there are some words that we can start looking for inside the field notes or inside those evidence statements that we could use as an indication of shifting vocabulary or more scientific vocabulary that would start us thinking or start us being able to document some of the learning that's happening. You. Let's stay here for a second. So again, what we want to actually do with the bulk of this session is use vital signs as a case to get at this question of how if you were a member of the vital signs team and the Hewlett Foundation came up to you and said, wow, this looks totally awesome. You've created this incredible open learning environment where students and scientists are co-creating data and it seems like, you know, our gut instinct is that students are doing really interesting stuff and they're developing this sort of important understanding of scientific thinking and developing a connection with the main coast. But how can we say whether or not they're actually doing any learning in what ways? And not even, I want to say measure, but I want to say quantify because not all the data we might gather to do this would be quantitative. But how would you go about doing this? So I've been, I'm not going to tell you almost anything at all about my research, but I've been looking at deeper learning in wikis. So students who use things like PB works and wiki spaces as collaborative learning platforms and trying to assess whether or not you can evaluate the degree to which students develop skills like collaboration or expert thinking in those kinds of learning environments. And so what I want to do with my couple of minutes is just give you a sense of sort of a way to think about this data as an entry point to think about, so how are you going to do this measuring together? Well, we're going to do it together for a while and then Sarah has to actually do it. With your help. With your help. So I've started to think about this kind of data as scalable, real-time, individual behavior and learning data. So vital science is tracking continuously to the second. Every interaction the teachers and students and scientists are happening in this learning environment. And it's capturing it in such a way that we can look very, very closely at an individual case or we can aggregate that data. We can scale up that data and we can look at hundreds or thousands of cases simultaneously. So we can do really rich detailed qualitative work and we can also do broader statistical work. So we're calling this data scribble data. With the idea that actually what we're capturing is that all the little scribbles that students and teachers make in the process of their daily activities of learning. And to me, I think there's incredible promise for essentially replacing summative assessments with the process of sort of constantly, formatively evaluating the things that folks are doing. So to kind of prime your thinking, here are some of the ways that I think about potentially in a more generic sense sort of exploring this kind of data. So this is a state space diagram where the number of cases is on the Y axis and the duration of kind of data collection is captured is on the X axis. So if you're sort of coming at this from the field of education, technology, learning sciences, most of the work that you're doing is probably down around here. So you're doing design research experiments where you're tweaking what's happening in a couple of classes. You're tweaking what's happening either in the delivery of the content, you're tweaking the actual learning environment, you're doing observational research where you're following a group of learners for a while and seeing what they're up to. There are lots of folks in this particular group who are interested in looking at hundreds or thousands or millions of cases over years and saying, okay, let's see what kind of statistical data, maybe without even actually looking directly at the content, can we get some sense of the learning activity that's happening? I was at a DML conference where they were doing things similar to this and they were like hooking kids up to wires and seeing how their skin temperature was changing and their heart rate was changing second to second and what their eyes were tracking on the screen. So really looking like in the case of one kid, data over microseconds to see how they're thinking. That's not quite, you have to do it a little bit more to do that. But this map is meant to sort of prime your thinking that the data here is incredibly rich because there's thousands of cases but we also have this incredible depth to it. We have this incredible real-time history attached to it which makes us think that we ought to be able to measure learning in ways that we haven't imagined thus far. So again, the, well, I'm actually gonna, let me run back to one slide because I think it'll be helpful to put this up again. You know, what these folks are trying to figure out is to see, are these students making gains in scientific and evidence-based reasoning? Are they developing collaboration and communication skills? Are they developing content knowledge? And how do they go about measuring things? So here's kind of where we need your help and how we wanna spend the next, the sort of second half of this presentation, the next 15 minutes in conversation. You know, if Sarah tapped you to be one of the advisors for this committee that's gotta put together this research program to evaluate deeper learning, you know, what advice would you have for her? What approaches would you take to evaluate learning? What relevant examples do you know about? What other people are you aware of that are doing similar kinds of things? You know, who should they be talking to? What kinds of, you know, are you working on a program that has similar needs and has similar challenges and how are you thinking about addressing them? So I think what we'll do for the next 15 minutes or so is open it up to you all. Let me, and I'll stand up here and keep some notes and you can probably call hands and talk with people. But yeah, it's folks idea, you know, get them out. Sure. Are you attempting or have you been trying to capture activity-stricted data and using that tentative specification that's out there right now to, you know, face the sub-eurology that you did this, simply more elaborate. And then once you do that, then you can do analytics on that based upon assessing the models that you've already made. You have to kind of look at how it's currently pulled that data in and see what that means in terms of what's going on through social media, through the interaction between What do you mean by an activity stream? We just thought it was all the same. Well, this versus back out right now it's been you're working here and I was saying ADL is working with it right now in terms of capturing basically online social behaviors and this has to be the actor. What you have done, you might have clicked this button or you may have, you think that's very, very basic. But then you can elaborate on that and start gathering and you launch an API of the tracks. You gather that data, tracks a little bit and it reports it back to something to be interpreted for being labeled. So you're capturing the system. And if I could build on that actually. So the activity stream spec was originally built as an open spec for social networking platforms. So it's used Google plus and but the work we're doing with the learning registry and that's where the ADL is part of that. We do have a new spec out that people can use taking that sort of syntax based way of assembling the actor and the action and the object of the action and everything for learning data and for learning kinds of activities and pedagogical contexts. And so I'll post that wherever that makes sense to post that and we would love to have people test it and contribute those data from your platforms to the learning registry open data set of these kinds of learning resource peri-data social mitigation. And besides learning registry ADL also has other projects we have a project in which is actually capturing that to report back to a learning record store. So that you can and it's another way of interacting, capturing data which is not necessarily peri-data. It's a little different but it's very similar and you can use such things in the activity theory different things but we'll look at that in the practice and see what it means. But one of the things that I'd love to see and then I'll let it up is where you're coming at from that research perspective and you have vocabularies maybe that you're using to code text where you're saying okay we're going to look for these terms and these terms equate to something about a level of learning or a level of communication or even how closely people are adopting the resource a level of commitment to the resource that they're using that kind of thing. Where you have those vocabularies already I would love to see some of that come into this discussion about how do we share these data openly so other people can see where that activity is happening across multiple platforms. So if I were to summarize what I'm getting from that and so what good learning activities would look like and then kind of develop a syntax for capturing activity stream data that would let you test whether or not students seem to be doing following desirable pathways. And it might go beyond the syntax in terms of actually APIs that plug into the various services that you're using whether it be a virtual environment or social media or what have you you want to try to track things. So you're moving all the way around So once the data is established and it goes into this whole discussion about open data skills Maybe you have something totally different and we have ideas that you do not resonate with that at all It's totally different It could be the exact same thing American Public University has done some interesting things with IBM's text analytics that we used to praise where they're looking for language Are students making arguments or presenting hypotheses you might search for the word because it's a really simple way to talk about you're telling a program to analyze lots and lots of words That goes towards machine grading The best is an expert human grader but if that's not scalable enough you can make human grading easier with rubrics so it's established in standards and allows less qualified people to create faster by using rubrics even peer-to-peer grading can be really effective looking for what other students think about this and thought about this stuff That is imperfect Keep going And thinking about would you a priori hypothesize what's some good text to look for what we'd be or would you grab a sample of 1% of posts start reading them and classify them and take all the really good ones and say within this set of really good ones is there common language that we're starting to see and then use that it's interesting to think about what should be data driven and theory driven so you might do some really grounded stuff to say what words count and then use that to drive your measures What other things would folks be interested in trying if you were a if Sarah hired you what would you sell some nature back to you This is kind of different than what I'm interested in using computational linguistics an algorithm that's one of the most machine learning intervention to look at and analyze again you're doing post-hoc analysis of data but you're doing it in a different way and it would prove you have to emergency there's work going on right now and it's open and dark it's things like that that are doing that for other reasons but it's very easy to be turned towards educational space as well towards educational things who are some of the people who are doing that how would I follow up on that okay don't find terrorists find smart kids identify the future heroes in Europe I actually work with a group a senior older who is using a program that they've written algorithm that computational linguistics would find students misconceptions in science against experts that's basically what he was just talking about so it's something that people are looking at and really wanting to get better because it gives the students immediate feedback on their ideas that they're writing about and then if you wanted to do more things for other people you could really be back to the students yeah I think that's a great point that almost anything we might come up with to assess learning for our own purposes like is this working in theory it would be great to think about the next step of feeding that back to educators feeding that back to students in real time and saying based on what we've developed which ways to figure out what good scientific thinking looks like and you're hitting it or you're not this is kind of coaching a group around it or whatever one thing I think is somewhat underutilized in evaluations portfolios an example of that a really good one I was sitting on the web radio where the students had been producing a bunch of just kind of sounded like almost random content interactions just broadcasting via this radio but the teachers had them organize it such that they had actually on a blog like a blog post of all of the time so that it was really easy to go back in and look through what individuals had done and so you know sometimes when you have a bunch of kind of random information sometimes it's helpful to create some kind of portfolio assessment that students actually help you organize what they can do and then it's really easy to take a rubric and have someone go through that material that's great yeah I think all those are fabulous ideas so you know we had some of them going in in terms of using some kind of language crawler I think that came up I think we have both students and educators developing rubrics for educators to use to evaluate student work and to see how their assessments of quality change over time to think about giving the community some kind of responsibility for doing that assessment so developing some kind of rating or starting system doing a peer review which I think is connected to that Sarah's got Bridget Barron in Stanford who does a lot of ethnographic research who's just got a grant to study vital signs and the first thing that they're gonna do are case studies they're gonna spend a ton of time with a small group of students and teachers and follow them really closely and do a bunch of ethnographic work and actually when you hear that earlier conversation about kind of all the analytic stuff that you want to do and the sort of sense that you need to have some kind of theory moving into it as to what quality it looks like to begin with you know it makes a lot of sense to have someone really spend a lot of time sort of looking at the learning process looks like getting a sense of the richer content and using that as a way to think about you know how you could extrapolate from that to do some of that yeah so I mean to me you know to me what's incredibly exciting about this is the opportunity to use lots of different approaches I mean I think in education technology broadly when we wanted to learn something in scale we've done surveys and we wanted to learn something in depth we've done case studies or design research and those have basically been the two things that we've done and this kind of data I think what's so exciting about it is the opportunity to mess around with those trade-offs between depth and breadth and get some you know having data that lets you do things that both are generalizable and have really high internal validity and get into a couple different angles Are there parting comments? Thanks for your time