Another day, another patch in NBA 2K22 and some people aren't happy, ladies and gentlemen. Let me tell you that. Some people are. It's mixed feelings in the community right now. I'm gonna keep y'all up to date on what's to know and what's not to know. Hey! Yeah, new to the channel, be sure to subscribe. At this point, it's just necessary. So resident Tryhard Power was up at 5 a.m. and caught the update. He posted on Twitter saying, four gigabyte current jam patch just dropped for 2K22, probably to fix badges. Okay, so new update, what's in the update? Well, we didn't know, actually, for a very long time. Which led some people to speculate. This tweet here says, bro, a 30 gigabyte update, did 2K add a new shirt again? But no, like about 12 hours later, 2K decided to clarify on Twitter saying, our latest patch on current gen is live with more fixes to improve the overall experience of NBA 2K22. And because we know it's NBA 2K and they never really tell us all the risks to know when the patch notes, people began to speculate in the replies. Badgeplug responded saying, paint defense seems to be buffed, anyone notice it? To which the Nadex responded, what else did they patch? I ain't reading all that. Oh, here Nadex, I got you, my brother. That's what I'm here for. That's my whole value. But no, once you get into the patch notes, you realize just how deep it is, like just let me scroll for a little bit. Like these patch notes go on forever. The way 2K organizes patch notes is crazy because you would think this was one big patch note. Nah, if you scroll down enough, they just compiled previous patch notes in the same file.  They actually surprisingly did get into detail about specific bugs in the gameplay, which they fixed. If you're curious about what those bugs are, they include things like, tightening collision detection to prevent ghost charging, blocking fouls during gameplay. There was a glitch with the rebirth and associated quest should once again be attainable for those users still looking to properly acquire it, which is probably the most important patch note. I've heard so many countless people tweet me saying that there's something wrong with the rebirth build. They can't get access to it. It's a glitch, 2K fix it. And then they begin to immediately get more vague by saying many more fixes included to improve the overall experience of MB2K22 across all game modes. They talk a lot about UI updates, which Jesus, there needed to be some. And UI is like the stuff that never makes headlines, but it just makes the experience more not frustrating. It doesn't even make it better, which we all need more of in 2K. Okay, and I get it. There's like a lot of back and forth. This person wants this. This person wants something else. One thing that everybody wants is great skin. Hey, this video right here is being brought to you by Qology. One time I was talking to a female, she went to my bathroom and she said, a detective agent, where's your face wash? Where's your moisturizer? And I thought, why would I have that? So I asked my friends, hey, do you guys have face wash and moisturizer? And they all did. So I felt like I was behind the curve. And for some reason, I never feel comfortable talking about it, but your skincare is really, really important. Never look at a celebrity and think to yourself, how do they look so young? It's because they take care of their skin. But unfortunately skincare can get expensive, and that's when Qology comes in. You go on the Qology website and you fill out a survey that tells them about your skin. You take some photos and they take a look at you. Based on the information you give them, they put together a customized formula for your skin. It's specific to you and your needs. With Qology, skincare becomes affordable. The first month is $5 and every next month is less than a dollar a day. I'm curious what the guys think.  I use hand soap and hand sanitizer. I was wrong to ask you.  Milk. You put milk on your face? Yes. You're not helping my case.  Don't do that. Don't put that on your face. They're gonna actually do it, so nobody gonna sue me. I don't recommend sanding your face. Me, though, me, I recommend Qology. Okay, so maybe those guys were the wrong people to ask, but the process is pretty simple. It goes like this. First, you apply the cleanser and then you apply the moisturizer.  So, God damn, it's start taking good care of your skin today. Click that top link in the description. It'll send you to Qology. You can fill out your information and you can get your customized skincare routine done for you. Something simple, man. Take care of yourself, ladies and gentlemen. They'll go a long way. Thank you to Qology for helping sponsor this video. They actually go on a list of different adjustments they've made to the courts, arenas, of course, to jerseys because the NBA just released their new city addition jerseys, I believe. And they also go on a slew of different players that they've adjusted the likeness of. Some of them are new face scans they added into the game, like Lindsey Allen. Don't know that person, but Scotty Barnes, Toronto Raptor native. So yeah, we all know Scotty Barnes or you should. He's a beast leading the Raptors in points, you know what I'm saying? He's a rookie, talk about it. I mean, so it's like, don't mind me or anything like that, man. And again, these are fixes that everybody can get behind but now we get into the gameplay stuff. Reduce the frequency of stumbles after dunks as requested by the community. It's funny how in some of the more like, I guess technically controversial notes, they let you know like, oh, this was like requested. We didn't even really want to do it. It was the community that requested it. And then the big one, the one that power referred to at the beginning of this video adjusted the logic for several shooting badges, including chef, limitless spot up, difficult shots, circus threes, rhythm shooter, catch and shooter, set shooter and stop and pop.  I have yet to experiment, although I might hop on stream, I twitch link in the description if y'all want to catch that. Sorry, am I showing off my hand too much? Just really proud of my pedicure guys. Pedicure is on your feet, I just lied to you. Players can now dunk more frequently in certain situations even when not using the skilled dunk mechanic because obviously it's seen some more criticism. It's on or off with the skilled dunk stuff. Some people hate it, some people think that it's tolerable. Some people actually like it. I would argue a lot of these gameplay adjustments because you don't need to get into all of them are good. Address the community reported issue where the AI could get stuck attempting several fake passes in a row so there they go again with the community suggestions. Which is cool, I guess it's nice that 2K is keeping a pulse on what's being said. Every once in a while you see a clip go viral on Twitter or on Reddit of just the craziest things happening in 2K. And it's natural, it's a video game, it's not gonna be perfect but the fact that it's being addressed is good. They go on to make like some quality of life stability fixes across both the city and the rec center. And they severely need it. Like the latency I'm playing on 2K is atrocious. It's not something that should belong in a triple A video game. It's a shame that it does. Fix the issue where people were having difficulty squatting up in the Gatorade training facility. Fix the issue with invites and voice chat. Fix the issue with the stats not being displayed properly on the program. They also talk a lot about fixes to my career and the quests on current gen. I haven't played the quest system on current gen only next gen. So I can't tell you how horrible it's been but I know on the next gen system the issues were so tragic that there were times where you couldn't even continue the story mode until an update came out. So all of this stuff is like well appreciated. I'm glad unlike the last year of 2K, if you guys remember correctly, where they just abandoned the game. They just disappeared. The devs just like walked off. Like in the middle of their job they just left to work on this game. So if the same thing happened with this game then it's like, all right, well then every game is just going to be incomplete. But it seems like they're trying to form it into a complete game. Although we could all sit here and argue that it should have been done at launch. But NBA 2K players are so split on the changes being made to the game. Of course everybody was really cautious when the game launched because in terms of like its sliders and the way it played, it was in a good spot. There was two main issues people had. One was on ball steals were way too easy and two was you're hitting way too many whites. So even if you didn't release the ball properly it still dropped. And since then things have evolved because of course now 2K has the ability to change things on a whim. They don't actually have to tell you they're changing something. And even as we sit here and go through an overview of the patch notes we have no clue whether or not these are all the things that were changed. I haven't said a single thing so far. There's nothing in the patch notes about paint defense but Badgeplug is saying paint defense seems to be buffed. So what we're seeing now is a lot of tweets and posts and comment sections on Reddit and on YouTube all over the place saying things like. 2K changed something for sure.  And if you guys I wonder my I think it's my most recent video. I went on there to play the Halloween. I was trick or treating gathering my rewards. I was feeling myself. I hop in a game and it just feels different. It turned me off. And it didn't turn me off because the game was being updated. Update the game. Yes. It's the fact that I was not informed about those updates. I get most people don't care. Like y'all might like hop on a video like this to know what's going on.  I get it. But we shouldn't be in the dark about changes and we just we have to discover them on the go. That's crazy. Badgeplug says I don't care. 2K 22 was the top three 2K before the patch last week. Now I don't even know where to put it. Revert the patch. This patch was horrible. I don't care what you think. He follows up. Anyone with a brain cell can tell things changed that we weren't told about. I've never been so sure of a slider gameplay change we weren't told about. So someone says what changed feels the same to me. Badgeplug clarified. He said I'll give a few. The on the ball steel nerf lets people literally run through you. Yeah. You still get steals but way less that are deserving which also makes the left right cheese better. Shooting is horrible. Now paint defense was cruised. I'm delayed every game and more which is something that luck also said on Twitter. He said he was also feeling delayed while playing 2K 22 which I would argue I always feel delayed playing 2K 22 to go from a game that's like well optimized like Valorant and you play on 15 ping or rainbow six and I play on 31 ping or even games that are poor like Call of Duty where I play on like 60 ping to go to playing on like 150 on 2K just always feels atrocious to me dog. The idea of the seasons is fantastic. It means that you get more content more. What I'm starting to realize might be the case though is that the content for the seasons was just developed at launch and then they're just holding it behind a gate until like you know the season's over and it's a new season and then they give you all this new stuff that was developed at launch. It's a little different when a game like Rainbow Six comes out with their seasonal content because they're actually developing it as the season's going on. So it's new content. This is technically just content that like we didn't have access to but that being said even if they are gating content and giving it to us as time goes on it is making the process of like oh wow there's something new there's a reason for me to hop on and play. So I would argue the addition of seasons has its positives. Like D-Man Pirates we saying for real though seasons being added to 2K is a W. We had to get season two started off right with a clip here of well here we have some gameplay on the rec Taz Alleyoop. Oh, oh, okay. Chamoma says seasons are literally one of the greatest things 2K could have added to the game. They definitely got a return in the future but then again it is split. Here comes its critics. Chalk says you guys keep saying that the level 40 grind is too easy because people are hitting it fast. You understand that legend took around 2000 plus games. Hitting 40 each time takes around 500 games. It's adding up to basically the same amount. People's criticism is that a lot of the excitement behind grinding in NBA 2K was seeing the race, right? In 2K 17 was the most popular example where Nate X and Orlando and Chicago were just battling it out on one of the Xbox one on the PlayStation, I believe. The point was is that beef right there that drama gathered so much excitement that people tuned in every time they leveled up to a new ah, ah, ah. So just having missing that because now everything is staggered by seasons to reach legend on NBA 2K 22 should I say takes I think they said four seasons of you hitting level 40. Although someone will still hit it first, the race is staggered. Think of it more like a 400 meter relay instead of one long 400 meter race. It is not enough time to build a tension, I guess. It's not even really like a relay because everybody starts at the same point every single season. Besides the point, I personally as a person who's not sitting there hitting level 40 every single season because I have too much things to do, I'm actually in favor of seasons. It gives me a reason to continue to want to come back especially like deep into the year, man. In July, bro, yes, please give me a good reason to hop onto the game with some new content and new season something. I also noticed, by the way, I don't know if y'all noticed that, but in terms of the animations, like the intro animations when you get into a park game or the park celebrations, we have lost 90% of these animations, guys. The animations we used to have in previous years, it was such a plethora. There were so many hundreds, you didn't even know how to choose which one you really wanted. It kind of feels like they took away some animations so that they can give us the same ones we had in previous years as seasonal content because there's 40 different levels in multiple different seasons so they need something to give us for each one of those levels. And on top of that criticism, a lot of people have been saying that reaching level 40 is either too easy or too hard. It's too hard for the casual person who doesn't spend a lot of time on the game to hit, but it's too easy that people that do spend time on the game are hitting it in a day like Hank the Tank. Yes, Hank the Tank hit it within 26 hours of the season being out. Granted, he probably had like double XP and it was like day one so it was quadruple XP but 26 hours, that's impressive. But then you kind of have to fall back and remember that like, first of all, there's two different games that we only get to play for one year before the next potentially two different games come out. What I think is happening, and this might be like a future documentary is people are just experiencing a lot of franchise fatigue. People that have been in the community since like 16, 17, y'all played the game dawg. Like it's a basketball game at the end of the day. Like there's only so much that can be done.  But there's a lot to like about 2K22 and some people are just sick, they're just tired of it. And what I think is gonna happen because believe it or not, I was in the Call of Duty community before I ever came over to the NBA 2K community. Same thing happened over there. People fell off because they already experienced it. They did everything they could do in the community. And then some new people came in and the community got lit and active again. I believe that's what's gonna happen here again. I'm curious how y'all feel about seasons, these updates and just the current state of 2K22 is the game fun? Is it enjoyable? Is it boring? Is it dry? Why you think so? Leave a comment down below. I'm curious. I know there's a lot of people in the community that watch me so just let me know, man. I will be reading the comments. If you don't wanna do that, there's a video right here. Oh, I get it. You have something to do, you don't have to click it. All right, catch you guys in the next one.


Hello and welcome to this presentation of the paper DLLA, Deep Learning Leakage Assessment, a Modern Roadmap for SCA evaluations. My name is Thor Mos and this is a joint work with my colleagues Felix Weegner and Amir Muradi from the Rohr University Buchen. This work proposes a new strategy for black box leakage assessment. In simple words, leakage assessment is a technique to determine whether input dependent information can be detected in side channel measurements of a device under test. Typically, if I can learn information about the processed inputs from a side channel trace, then indeed some kind of side channel leakage is present. This does not necessarily mean that the device is vulnerable to attacks, but it's usually desirable that no detectable input dependent information is leaked to potential adversaries. The most common approach to perform leakage assessment is to try to distinguish leakage distributions for different input classes using a statistical hypothesis test. The most commonly used input classes are fixed versus random, where one group for fixed input is collected and one group for many different random inputs. And then fixed versus fixed, where two groups for two distinct fixed inputs are collected. Then either the Welsh T test or the Pearson's G-square test are used to evaluate the null hypothesis, which states that both groups are drawn from the same population and therefore indistinguishable. This method has been applied for many years in the community and different extensions to the tests and the methodology exist to tackle specific problems. But there are still a number of aspects where the technique works less than ideal. First of all, side channel measurements usually consist of hundreds or thousands of sample points or even more, but the classical tests are normally applied to each sample point individually instead of on the full trace at once. Therefore, side channel leakages that are spread over multiple points, for example multivariate or horizontal leakages, may go undetected. Also, it is known that the separation of statistical orders in the Welsh T test can lead to false negatives when the noise level is low and the leakage is of higher order or when the leakage is distributed over multiple statistical moments. We have thought about techniques that could potentially avoid some of these drawbacks. And we noticed that leakage assessment is essentially a statistical classification problem. If we can generate software and algorithm or a machine that determines whether a leakage trace belongs to group 0 or group 1 with a better success rate than randomly guessing, then indeed input-dependent information is included in the traces. And to us, this seemed like a pretty straightforward task for a neural network. Again, the potential benefits in comparison to the classical methods would be that the network would receive each trace in full length and therefore can make a decision with respect to a full trace instead of one point only. And also that the classifier is not limited to simple univariate first-order leakage but inherently captures all leakages that normally occur. Based on this idea, we have created a new leakage assessment methodology which we called Deep Learning Leakage Assessment in short DLLA. The procedure looks like this. First, we record a set of side-channel measurements while supplying the device under test with a randomly interleaved sequence of two distinct fixed inputs. Then we split the whole set into a training set and a validation set and train the neural network on the training set. Afterwards, we use the trained network to classify the traces in the validation set while hiding the true labels from the network. If this is done, we can compare the classifications with the true labels and count the number of correct classifications. When we have that number, we can calculate the probability that the correct classifications could have been achieved just by chance. If this probability is smaller than it shows in threshold, for example the typical 10 to the power of minus 5 threshold in leakage assessment, then we can reject the null hypothesis. The number of traces for the detection is then the cardinality of the training set only because this set was sufficient to extract and learn a generalizable feature from the traces. In the validation period, the network is not updated so it cannot absorb any additional information from the validation set. But of course the number of traces needed for the whole evaluation is the combined cardinality of training and validation set. Here is the formula to calculate the probability that SM correct classifications are achieved by a randomly guessing binary classifier where M is the cardinality of the validation set and V is the validation accuracy. To get an intuition for the required validation accuracy to confidently reject the null hypothesis for different sizes of the validation set, we have listed a few examples here. For a set of only 76 traces, at least 57 of them need to be classified correctly which corresponds to a 75% validation accuracy. For a set of 45600 validation traces, for example, the validation accuracy of 51% is already sufficient to reject the null hypothesis. For even larger validation sets, the required validation accuracy shrinks further and further. In that regard, it becomes clear that DLAA does not require the trained classifiers to be very good at the classification of a single trace. It is only important that it classifies the traces better than a random guess over a large number of traces. This is the Python code that defines the multi-layer perceptron in Keras which we have used throughout this work. It is an extremely simple network with four fully connected layers and two output neurons for the classification. We have kept it very simple on purpose to make it suitable for a wide range of side channel traces and different forms of side channel leakage. The second network we have used for our case studies is a convolutional neural network. Its description in Python code is shown on this slide. Here we actually apply a couple of parameters and will evaluate the robustness of its detection performance across eight different hyperparameter combinations. We have tested our new leakage assessment strategy in a total of nine different case studies using side channel data from different implementations and platforms. We always compare DLAA to the t-test and the g-square test in order to judge the quality of the deep learning-based leakage detection. The first seven case studies are all based on hardware implementations measured on an FPGA. For some of them, the traces are aligned. For others, they are misaligned. Some circuits are protected by a masking countermeasures. Some others are not. Some of the traces that contain univariate leakage and some others include only multivariate leakage. In addition to the FPGA-based case studies, there is also one case study based on measurements from a custom ASIC and another one where a software implementation is executed on an ARM Cortex-M0 microcontroller. In summary, we have tested our method on a large variety of realistic target implementations and compared its success to the classical methods. The one thing that all case studies have in common is the underlying cryptographic primitive, namely the present-AT ultra lightweight block cipher. This slide shows the architecture of the unprotected serialized present hardware implementation that has been analyzed in the two following examples. On the top of this slide, we see a sample trace recorded by measuring the voltage drop over a 1 ohm shunt resistor in the VDD path of the power supply of a Spartan 6 FPGA using a digital sampling oscilloscope. On the bottom, there are four figures showing the results of a univariate first-order T-test and a G-square test. They also show the progress of the highest confidence values over the number of traces. Clearly, a significant amount of leakage is present and can be detected with confidence after less than 100 traces. The maximum confidence expressed as a minus log 10 value is slightly above 80. Using deep learning leakage assessment on the same trace set, we achieve the results shown here. Since the DLLA procedure does not produce one confidence value per sample point but instead only one in total, we have to use a different method to extract spatial information about the leakage in the trace. This can be done with a so-called sensitivity analysis based on input activation gradients. This method locates the points of interest by quantifying how much they contributed to the leakage function learned by the neural network. More details on that can be found in the paper. On the right side, the confidence values over the number of training traces are depicted. Obviously, the confidence values achieved are much, much higher than for the classical methods. However, for the sake of completeness, we have to mention that the validation set is not counted in this figure and that the total number of traces required for the evaluation is sometimes higher than for the traditional methods in these simple case studies with unprotected implementations. The next case study we want to take a look at is based on the same present 80 implementation as before. But this time, the clock of the circuit was randomized by an LFSR circuit, leading to a misalignment of the traces and a large amount of noise. We can see that the traditional methods require up to 5,000 traces now to detect the leakage. Using DLLA for the same analysis, the detection succeeds with about 150 training traces. However, the acquired confidence is also reduced by a large margin and the identification of the points of interest might not be as sharp as for the t-test. Here we see a serialized 3-share threshold implementation of the present 80 cipher, which is the underlying hardware implementation for the next case study. The S-box of present needs to be decomposed into two quadratic functions before a TI with 3 shares can be realized. On the top of this slide, a sample trace is shown, which has been recorded during the execution of the first one and a half rounds of the present threshold implementation. Below there are again t-test and g-square test results showing the detection of higher order leakage. For the t-test, only the third order is shown as the statistical confidence is the highest for this moment, but the results for first and second order can also be found in the paper. Since the leakage of the implementation is partially found in the second and partially in the third statistical moment, it is no surprise that the g-square test outperforms the t-test here. We also apply DLLA to the same traces. But unlike before, we do not plot the confidence values over the number of traces anymore because such an analysis requires the analyst to train a classifier many times using different sizes of the training set. Although this allows a nicer comparison to the classical method, it's computationally expensive and rather unnecessary. Therefore, we have first chosen a fixed training set of 3 million traces and plot the confidence value over the number of epochs to show that a much higher confidence can be achieved by DLLA than by the classical methods with the same number of traces. We can also see an important conceptual differences between leakage location using sensitivity analysis on a trained network compared to the classical approaches. Parts of the trace that contribute to leakage but correlate heavily with other parts contributing to the leakage might not be learned and used for distinguishing the two groups because there is no intrinsic incentive for the neural net to learn redundant information. Thus we can see that the leakage is pinpointed only at the point of the strongest leakage here and not in each point of the trace where there is actually leakage located. Here we have used a significantly smaller training set to demonstrate that using DLLA it's also possible to confidently detect leakage using a smaller amount of traces than the T and G square test require. The confidence threshold is overcome while training on only 500,000 traces although just barely. The final case study we are going through in this presentation is also based on a present threshold implementation. But here we ensured that all six component functions G1, G2, G3, F1, F2, F3 together with the respective registers are clocked sequentially and not in parallel. We did this by gating their respective inputs with AND gates which are controlled by a finite state machine. Therefore no univariate leakage should be detectable but combining multiple points in the trace should still allow to distinguish the two fixed groups from each other. We can see here that indeed neither the univariate T test nor the univariate G2 test detect any leakage using up to 50 million traces. However, when using a multivariate T test by combining the respective points with the correct offsets we can get a confident detection after roughly 45 million traces. Yet this is a process that cannot easily be automated for any implementation as it requires exact knowledge about the underlying implementation. Thus it is not perfectly suited for a black box leakage assessment. Using DLLA we do not require any knowledge or information about the underlying implementation. We simply train a classifier on 20 million raw traces and the neural network learns the multivariate leakage automatically. When we evaluate the classifier on further 5 million traces we obtain a high confidence to reject the null hypothesis. These cases of multivariate higher order leakage are the most relevant and interesting application scenarios for DLLA since the classical hypothesis test cannot easily be extended to capture such leakages universally. Of course, the ability of the networks to automatically find and learn different leakage patterns does not come for free but at the price of a significant runtime overhead just like in most applications of deep learning. Here we have listed some numbers regarding the runtime on our machines. In short, if you want to analyze the same number of traces and train the classifier between 30 and 50 epochs the DLLA procedure will have a runtime that is about 100 times larger even when using the GPU support. Yet the absolute runtimes are still reasonable for a realistic number of traces and might spare you a lot of manual work for the more complex scenarios. Finally, we have also tested the CNN network which was introduced earlier for its ability to learn the different forms of leakage in the case studies from the different platforms. Also in this case we have tested whether small changes to its parameters have destructive effects on its detection performance. However, we can see that although changes are noticeable the confidence values achieved are typically in the same range. The largest variation is seen for case study 4 here. To conclude this talk, we have shown that leakage assessment using neural networks is feasible and worth the extra runtime for complex detection scenarios. We demonstrated that very simple networks can deliver quite universal detection performance even when faced with multivariate higher order leakages. Also we have seen that the amount of traces required for detection is typically lower in DLLA compared to classical detection while also providing a higher confidence. Finally, since DLLA produces one confidence value per set of traces instead of one value per sample point the risk of false negatives is lower if the same threshold is chosen. With that I would like to end this talk. Thank you for your attention. If there are any questions feel free to ask them during the live session at chess 2021 on September 16th.


Buenos dias. Good morning. Three decades ago, Canada, Mexico, and the United States made history. With NAFTA, we three Amigos established the world's largest free trade zone. We created millions of jobs. We enabled an extraordinary economic expansion and showed that by breaking down barriers, we could create opportunities for people. Former Canadian Prime Minister Ryan Mulroney, who negotiated the deal, reflected back on what NAFTA has meant for Canada. The benefits have been spread over our entire economy. It helped modernize our economy and got Canadians to be much more outward thinking and self-confident that they could trade with the best in the world and win. That's what free and fair trade does. It opens doors. It expands horizons. It creates the best possible outcomes for people and for businesses. Now, let's be clear, it wasn't just Canada that won. NAFTA was never a zero-sum game. It was always win, win, win. With it, we unleashed flows of goods and services between our economies to unprecedented levels. We quadrupled trade on this continent from the Pacific to the Atlantic, from the Arctic Ocean to the Gulf of Mexico. Over the past three decades, our economy has changed a lot. We've gone from fax machines to smartphones in almost every pocket. That has created whole new sectors and changed the way we do business. We've weathered crises, booms and busts. And most recently, we faced a global pandemic that wrought economic turmoil on a scale not seen since the Great Depression. Entire sectors shut down virtually overnight. Tourism, hospitality, culture. The economy reoriented itself to keep people safe. People worked from home. Businesses underwent massive changes, and our healthcare sector withstood and continues to withstand incredible pressure with workers who continue to be heroes day in and day out. It took its toll whether you were a student, a worker, a business owner, or a parent. That one year in 2020 added a few years to all of our lives. But as people pulled together, we saw the best in one another. And since then, all three of our countries have undergone strong recoveries thanks in part to our strong trading relationships. Free and fair trade has made us resilient and continues to foster extraordinary growth on our continent. Let me give you a few numbers. On our shared 22 million square kilometers, home to nearly half a billion people, we do over a trillion dollars of trade with each other every year. Combined, NAFTA has a higher GDP than the entire European Union. Our integrated continental economy is an advantage unlike any other. We must never take that for granted. A few years ago, you may remember, there were those who attempted to throw the certainty of our trading relationship into disarray. Motivated by protectionist, isolationist, and nativist politics, they were willing to put millions of jobs at stake in each of our countries. Our historic trade deal was in peril, so we reopened it to defend it. In the negotiations, the U.S. repeatedly tried to play off Canada and Mexico against each other. But Canada always believed that our greatest strength was in all three parties negotiating in unison. We understood that North American free trade was about good and fair integration across the continent. Business leaders, investors, labor groups, and workers understood this too. We all held fast to our belief that in the face of economic uncertainty, growth doesn't come by turning inwards and putting up walls. It comes from welcoming new opportunities, opening up to workers and investors, making economic activity easier, not harder. So we rolled up our sleeves, we protected NAFTA and even improved it. More protections for our environment, new opportunities for small businesses, stronger standards for workers, because trade ultimately has to be about people. It's about a nickel miner in Sudbury, a steel worker in Pittsburgh, a carpark manufacturer in Chihuahua, who all have good jobs because our auto-making supply chains are working well. It's about a farmer here in Mexico having easy access to Canadian consumers and vice versa. With food prices going up around the world, it's worth pointing out that NAFTA has helped lower food prices. Without our free trade, billions in import fees will be passed on to consumers. By integrating our economies, we're making them more dynamic, supporting workers, creating jobs, and helping tackle the cost of living. The goal of the NAFTA has been to build an equitable and well-integrated economy on the continent's scale. Faced with economic uncertainty, Canada has always believed that growth doesn't take over, by putting up walls, by putting up walls. Growth comes by welcoming new opportunities, by opening up to workers and investors, and by making economic activities easier, not more difficult. We have protected the NAFTA, and we have improved it. With new opportunities for small companies, better standards for workers, better protection for our environment. This is how we see food prices going up in the world. Without the NAFTA, the situation would be even worse. Without our free trade, billions in import fees would be passed on to consumers. By integrating our economies, we're making them more dynamic, supporting workers, creating jobs, and helping reduce the cost of living. There's a lot of uncertainty now and ahead of us. We're seeing Putin's illegal and brutal invasion of Ukraine that is worsening the global food and energy crisis. People around the world are suffering from hunger and cold right now because of it. We're seeing the devastation of climate change and increasing costs of natural disasters. We're seeing democracy backslide globally and arise in authoritarian leaders. We're seeing fears exploited and young people, like those who attend this university, wondering what kind of world they're going to inherit. These are historic challenges before us, and the choices we make today will ripple across generations. So let's think like people did back when they signed the original NAFTA. They couldn't know all the changes and challenges we would face, but they knew that growing our economies and deepening our ties would give us all the stability and certainty we needed to weather any storm. Not only that, they knew that an integrated economy put us in the best possible position to grab any and all opportunities, including those they couldn't even imagine yet. Now just look at what's happening now. Our global work to tackle climate change is bringing about the greatest economic transformation since the Industrial Revolution. That means booming new sectors and the modernization of traditional sectors. And it's happening at an incredible pace and scale across the continent. And all of it supports and creates good jobs. Let me give you some examples of what that means in Canada. Over the last two years, we've made strategic investments with partners like Arcelor Middle and Rio Tinto that will see Canada produce even more of the cleanest steel and aluminum in the world. This is creating a brighter future for our workers and for the next generations of workers. We're investing in wind, solar, hydrogen and carbon capture technologies so we can produce cleaner energy. Last year, we attracted investments from around the world for electric vehicle manufacturing and critical minerals that have secured around 17,000 jobs. Just last month, GM opened Canada's first ever full scale electric vehicle manufacturing plant re-energizing a community and giving our auto workers even more confidence in their future and in their kids' futures. And we launched a critical minerals strategy because Canada is rich with the resources needed for electric vehicles, clean energy and digital technology. Canada's best in class resource workers are an essential part of the EV auto manufacturing supply chain here in North America. Our workers in our natural resources, the best in the world, are essential. And critical minerals are not only for electric vehicles. Portable computers and smart phones also need batteries. Clean energy sources like hydroelectricity, solar panels and wind turbines need critical minerals. One thing is certain. Canada is ready to be the reliable supplier of clean energy and technology, and that zero world will need. And we'll do it with our North American partners, both businesses and governments. The economy of the future will also be shaped by new forefronts in technology and innovation. It's something that I know the students and scholars at Centro are right at the center of. They're helping shape the innovation economy and the creative economy. And I'm sure there'll be leaders in new sectors we haven't even thought of yet. We need to keep building an economy that works for all North Americans. It creates good middle class jobs of all types and that young people can see their bright futures in. Collaboration is key. It gives our creators, entrepreneurs, farmers and manufacturers access to new markets. It fosters competition, benefitting people with lower prices and more choice. It builds supply chains that are both efficient and resilient. It makes our economies more vibrant and it creates stable economic cooperation among friends, friends who share a belief in equality, democracy and rules based trade. As former Mexican President Ernesto Zedillo who implemented NAFTA said, trade by being mutually beneficial gives each party a stake in the well-being of the other. We can never underestimate just how valuable our reliable and stable trading relationships are. They give confidence to investors, to workers and to the people who want to build big things on this continent. Of course, like all friends, we'll have our disagreements from time to time. But overall, things are working very well and Canadians see that. That's because we've worked hard to make sure that the benefits of economic growth are felt across the economy. And that's really important. During economic growth, we must pay attention to the advantages of this growth. Of course, some sectors will benefit from trade more than others. That's why we must reinvest the benefits of this growth in people. We must invest in education, in healthcare, in social programs that help vulnerable people. We must orient our prosperity to emerging industries in order to create new jobs and opportunities. We must grow our average class in order to make our economy more resilient and more stable. We need to grow our middle class so we can have a more resilient and stable economy. This isn't just good for people. It's good for businesses. When the benefits of growth are tangibly felt in our communities by the citizens who vote for the direction of their country, businesses thrive. A more stable society is one where people with good jobs trust that those good jobs are going to stick around. And businesses, in return, benefit from the generations of workers who bring knowledge, dedication, and pride to their jobs. Businesses are implanted within communities and when people feel that loyalty and share in the benefits of growth, they are loyal in return. A more fair and optimistic society is one where bottom lines are stronger. But of course, when the world is uncertain, it's human nature to look inwards to feel the pull of protectionist urges to want to hunker down and let the storm blow past. But as governments and as leaders, we need to be wise enough to recognize that giving in to those voices is not the way to overcome our challenges. Putting up trade barriers is putting up barriers to growth, and that doesn't help anyone. We three countries, of all the ingredients we need, to be successful. And by working together, we can continue to unleash the enormous potential in our people and in our resources. Canada is a trading nation. This is a truism that goes back hundreds of years to the earliest interactions between Indigenous peoples and Europeans, when trappers traded food, the furs for food, and other goods. And it remains true today. We have free trade agreements with 51 countries and are the only G7 country to have a free trade agreement with every other G7 country. We have investment treaties with all 28 EU member states. And simply put, Canadian businesses have preferential access to two thirds of the global economy. It's easy to see why so many countries want to do more business and count. We have the most highly educated workforce in the OECD, a sound financial system, favourable tax rates, world-class universities, and a highly competitive research and development environment. On top of that, Canada has universal healthcare, affordable childcare, vibrant cities, and ever-increasing areas of protected nature. And importantly, strong democratic institutions. We are a reliable partner with an abundance of talent, a very attractive investment climate, and a great quality of life. And since 1993, trade between Canada and Mexico has increased more than ninefold. And recent data shows that in the first three quarters of last year, Canada was the second largest source of new foreign direct investment in Mexico ahead of Spain. There's huge potential for growth between our countries. So let us continue this momentum. Let us keep doing what leaders did a generation ago. Hold fast to our belief in open trade and collaboration. Having quoted both the Canadian Prime Minister and a Mexican president, I'll give you one more quote from Bill Clinton in 1993 when NAFTA legislation passed in the U.S. We are on the verge of a global economic expansion that has sparked by the fact that the United States at this critical moment decided we would compete, not retreat. Let's keep forging ahead and pursue more growth for our economies. Growth that includes everyone. Growth that creates good jobs and strengthens the middle class. Growth that means our air and water is clean. Growth that is shared amongst friends and makes Mexicans, Americans, and Canadians even stronger and more optimistic about the great future waiting for us. Thank you all for being here today. Gracias.


Hey, this is Brett, and this, 2022 Ford F-250 Crew Cab Short Box Lariat Ultimate Tremor Package is stock number 13596Z. I am here at Summit Automotive in Fond Lac, Wisconsin, you're new and used heavy-duty truck headquarters. This 2022 Ford F-250 has the 7.3-liter V8 Godzilla engine, puts out 430 horsepower, and it's paired up with the 10-speed automatic transmission. I'm going to go all the way around in this video, inside, underneath, start it up, and take a look under the hood to give you the most accurate representation possible. This truck has been fully-safetyed and inspected by our service shop. Has a fresh oil and filter change, all the fluids have been checked and topped off for the state of Wisconsin inspection process, and it's 100% ready to go. Iconic silver is the color, and I shoot all my videos in 4K, so if you have HD capabilities, turn them on now, because it is your best way to check out the quality condition options and cleanliness of the truck before seen in person. As you go down the driver's side here, you can see just how nice that boxes, cab, and doors are in fantastic condition as well. Didn't see any major dents or dings on there. It's really good, as does the front fender. If you like the video, you can subscribe to my YouTube channel, which is youtube.com. So I have Summit Auto. Click the bell notifications to get updates on the videos I do each and every day here at Summit Automotive. This one comes with a really nice set of fuel-painted alloy wheels, and it has Nitto Ridge Grappler 37x12.5R18 LT tires, and I would say that they have just about all the tread left on them, 90%, maybe a little bit more. The Rancho shocks there, these are special for the tremor package. It actually gives you a two-inch lift from the factory. You can see just how clean that frame and underbody is. Like new, coming around the front, bumper is in great shape. No dents or dings in there. It does have the factory fog lights and the chrome-trimmed grille. The hood is in fantastic condition as well. And the passenger side front fender, absolutely perfect too. Passenger side front wheel, no major scuffs or scrapes. And iconic silvers, got just a little bit of metal flake to it. It's more like a pearl coat, and really a metallic, but either way it looks really good. And as you go down this side of this 2022 Ford F-250, once again take note of how clean the body is, how reflective and mirror-like that paint is. I take these HD videos so if you are far away or even if you're close by and you just cannot make the trip down, but you're still interested in purchasing the vehicle, you can see the truck, hear the truck, and have confidence in the vehicle that you're looking at. Before you even get here, so when you do get here, there's absolutely no surprises. You can make a smart and informed buying decision from wherever you're at. If this video helps you do that, your salesman knows you saw the video, it was helpful and that Brett sent you. Back tires have just as much tread as the front tires, and the frame and underbody is in very, very nice condition. It does have a Magnaflow exhaust on it, so it sounds really good as well. And lower rockers and cab corners and underbody looks great. It does have power running boards, which we'll check out in just a second.  Full towing package, which includes a receiver hitch, 4-pin and 7-pin wiring, and the tailgate is in really nice condition as well. So it comes with a Lomax fold-up hard tonneau cover, and that's in really nice condition. Has the power drop-down tailgate. You get a spray and bed liner, as well as a bed mat, and you do get LED bed lighting back here. I don't know if this one has any prep packages. No, this one does not have a prep package. Anyways, very nice and clean back here, and that spray and bed liner goes onto the tailgate. You do get the tailgate step assist, and that shuts nice and solidly. Powerful disclosure, this back wheel is in nice shape, and once again, down the rest of the side of the truck, very nice and clean. It does have the door code entry if you ever wanted to figure out what your code is for your 17 to 22 Super Duty in the upper right-hand part of the screen. I got a video showing you how to find that. Blind spot monitoring, directional signals, these power fold-in, power telescope out. Really nice mirrors. Sometimes you get the tan leather bucket seats, really nice shape, no rips or tears on there, and you get the factory all-other floor mats and the heavy-duty rubber floor, so kind of a rare package with the Lariat Ultimate and the rubber floors. Power pedals, auto headlamps, those are LED side lights on the mirrors. Power windows, locks, and mirrors, and once again, these mirrors do power fold-in. I always like showing both sides so that you know that both sides are working properly, and then they do telescope out and in, and out, and in. Back seats are just as clean as the front seats, no rips or tears back here. It does have the latch-child safety system for any child car seats you may have. Power sliding rear window with the built-in rear defrost. These seats do fold up. There's a collapsible storage area back here. You can lock that in place. You also get a 110-volt, 400-lop-backs plug-in, a USB and USB-C plug-in back here, and a 12-volt power point. All-other format back here as well. And you do get child safety locks on the back doors. Inside of the doors all look fantastic. Get the tire and loading information sticker right there, as well as the VIN sticker, and you can see those power running boards are working nicely. Up inside, check the miles, radio, everything that this truck has to offer on the interior. You get the built-forward, tough animation right there. 10,155 miles, you do get the digital spinometer, a nice digital instrument cluster, compass display as well, comes with the leather-wrapped steering wheel, Bluetooth and audio controls on the right, cruise controls and information center controls on the left. Here's your shifter with the optional tap shift. This one comes with the sync, I believe it's a 12-inch radio. You get the factory navigation system right there. There's Highway 41, so that is working nicely. And you get AM, FM, and SiriusXM radio capabilities. You also have all your different settings on here. You have your apps with Apple CarPlay, Android Auto, and all that good stuff. This is also where your backup camera shows up, and you can see that that is working nicely. You also get the B&O Premium sound, stability control, trail control, and then lane departure warning. Down here is your turned-out four-wheel drive factory brake controller, dual climate controls, heated and cooled seat buttons, your wireless cell phone charge pad, USB, USB-C, and then you get two cupholders, but you can make that four cupholders just like that. That's my only magic trick today. 12-volt power point and 110-volt 400-watt max plug-in. Passenger side format and seat are absolutely perfect. This truck's never been smoked in. And you get the second real fixed-glass power sunroof in front, VistaRoof. Power sliding rear window button, and sunroof and sunshade controls and map lights. We'll start it up and take a look under the hood and also check out that Magnaflow exhaust. Starts right up, no check engine lights or anything like that, and one thing to note, this thing does have a heated steering wheel, so that's a really nice feature as well. I would personally like to thank you for checking out the video today, and hopefully from this HD video you've been able to verify the quality and condition of the truck inside and out. Very nice ride. All the lights are working nicely up top. And under the hood we have the 7.3 liter V8 gas-coated Godzilla engine, 430 horsepower engine base, very clean runs, very smooth once again. This truck has been fully-safetyed and inspected by our service shop as a fresh oil and filter change, all the fluids have been checked and topped off, and this truck is 100% ready to go. There's your emission sticker, and probably one of the nicest F-250s we have on our lot right now. Really good looking, and to see more pictures of this truck, or one of our other 550 new and used cars, trucks, SUVs, minivans, Wranglers, halftons, three-quarter tons, one-tons. You name it, we've got to go to that website right there, summitauto.com. Some pictures and descriptions of every single vehicle from two locations, all at summitauto.com. And if you'd like to check out more HD videos, you can go to youtube.com.com. Remember to like, subscribe, and share on this video, and all the videos that you see there, click the bell notifications. You'll get updates on the videos I do each and every day here at Summit Automotive. In fact, any second you see a link to subscribe to my YouTube channel on the upper left to watch more F-250 videos like this one on the upper right, a link to this vehicle on our website, and a link to one of our latest YouTube videos on the lower right. Make those checks out. We're super excited to help you with this ultra clean like new 2022 Ford F-250 crew cab short box, Lariat Ultimate Tremor Package in iconic silver metallic. Thanks again for checking out the video. Remember to like, subscribe, and share on the YouTube channel. I really appreciate it. Thanks again.


Kia ora koutou everyone. Thank you, Fi. I'm very nervous about this presentation. This is an original piece of work and I've done a lot of talking internally at Te Papa around it and I've started sharing it a little bit wider and it's really interesting, the feedback. I'm going to go through it pretty quickly. It's a reasonably decent sized model and you probably will have lots of questions until we won't get through it in half an hour, but I'll do my best. OK. Not working good. 1.8 million. 1.8 million visitors in 2015. That's a big number. So it must mean that we're pretty successful. 12 million is an even bigger number. 12 million people passing through the Wellington Railway Station a year. So that must mean that the Wellington Railway Station is more successful than Te Papa. It's pretty important. I mean there's no denying it. 500,000 visits to collections online. So less successful than the building. 400,000 visits to Rainbow's End. Is that comparable? Are we competing with that kind of thing? It's a similar kind of number. 700,000 sky subscribers. So collections online is less successful than sky by the numbers. 250 likes is the highest we had in Instagram for a post. It's a picture of the Sunday markets. Nothing to do with Te Papa at all. 520,000 visits. Is this something we should be comparing ourselves to? Visits to Citizens Advice Bureau in a year. So numbers are important, but they're not the most important thing that are out there. We need to find a way to differentiate ourselves from the numbers that are pouring through the alternative museums and the other things. And so we need to start telling different stories. But numbers are important because we need them to understand market share and reach and plan our budgeting and all of these sorts of things. But it's only the beginning of what we can actually do to measure our success. We do wonder and awe and emotion and those sorts of things actually fit in to what we do and how we measure. What does the funnel of engagement look like beyond the numbers? What is the value of what we offer and what is the impact of what we offer for our audiences, not for us, for our audiences. So this is an attempt to move away from big dumb numbers. So there's a bit of a theme there.  Traditionally, Te Papa has talked a lot about feet through the door. That's our main metric. Paid views is something that we report to the Government for some reason. And dollars earned, which is really important. We have to have money and we have to gain that kind of stuff, but these are the things that we've traditionally measured as success. But as the National Museum and Art Gallery, shouldn't we be doing more? And we thought, yes. So a couple of years ago, when we had a digital directorate, we were setting up a number of tools and frameworks that made sure that we were building the right things for the right people in the right way, for the right amount of money. We had something called the Digital Product Development Framework, which I'll talk to soon. We had a customised lean canvas to make sure that we understood the problem, the audience, the goals that we were attempting to do. We used design sprints to make sure we were focused. We had a really good focus on user needs and how we tried to identify problems to solve. But a massive gap that we had was a consistent and meaningful way to identify success beyond numbers. This is a new version of the Digital Product Development Framework. You can... See, you start with an idea. You write up some stuff for a lean business canvas, maybe a short business casing like that. You make a decision when you put it up against other products that might go through. Do some solution validation, which is a short, intense, maybe design sprint or something like that so that you can try and come up with a problem or a prototype that you can say, we think this is the right direction to be going in. Then you get a goal and no go, so you get the rest of the money to actually go off and build something. Then you launch and then you go into high care so that you look after this product for a while very actively. You move into sustain and grow, which means that you don't just walk away from products once they're launched. You make sure that they're being looked after and maintained. You look after your communities. You look after these products and you may decide to retire it or you may decide to move it on to the next iteration. Just a nice life cycle to make sure that we're covering all the parts of digital product management that we need to. In this area here is where we notice the real problem of not having a good way to understand what success of fixing a problem looks like beyond numbers. This is the modified lean canvas that we use. The lean canvas is incredibly useful for very quickly checking all your biases and making sure that you have a problem that you're trying to fix and that you understand your audiences. It's modified from a start-up lean canvas. One of the things that we did was we modified instead of revenue, which of course is my start-up thrill in goal, is we had value and success metrics, but we didn't really have a good way of actually filling in that box. So that's all that stuff. The way that we started was using Dave McClure's Pirate Metrics, and they're called Pirate Metrics because they are because they're acquisition, activation, retention, revenue and referral. And they were good, and they were better than what we had before. They were a model that kind of worked and worked for a lot of people, but we're not driven by revenue most of the time.  We do kind of have an ad hoc way of doing things when we are coming with exhibition visitor goals. We'll come up with what we attempt to achieve in the long term, but it's usually pretty inconsistent and there's not an actual framework behind it. Google has a model called HEART, that's Happiness Engagement, Activation, Retention and Task Completion. That's more UX-focused, so at least it's based around the visitor, not the business, but again, still not quite right. Task Completion, we don't just want people to complete tasks.  That's Morris Hargraves McIntyre, which is a cultural segment company and they have an engagement spectrum that talks about the traditional ways that museums have done stuff in the past, which is labels in cases and that kind of thing, at one end of the spectrum, all the way through to co-creation and participatory design and things like that in exhibitions. So there's a spectrum there. MK Hayley is an imaginary at Disney and she's got a model that I'll talk to in a second. And what I found was a really simple common flow spectrum. It's not a one-to-one relationship across all of these models at all, but there is a common flow. How to move people from initial attraction. Most of them start with, we need to get people. Obviously you have to get people before you can do anything. Through to something more. Depending on the model, what that something more was varied. At the beginning, always the user commitment was pretty shallow, but at the end there was a more deep commitment from the person who was engaging with you. Time frame started off from medium to long term and value exchanges minimum effect through to maximum impact. Looks a little bit like this. I've also included the tupapa changing hearts, changing minds, changing lives because it kind of happened to fit in there as well. Changing lives is if that's our goal, how do we get down there. Again, the spectrum is kind of similar. Start off with the lighter stuff, end up with the heavier stuff. I've since removed the MHM engagement spectrum and the mapping of this because it's more of a how to get impact rather than what is the impact. Every time I share this with someone new, they go, have you thought about model? I go, no. I didn't know about it. I couldn't find it. What usually happens is it doesn't prove that I'm wrong. It actually gives more evidence that there's something there. A recent one was the Kirkpatrick model which is a model for assessing organisational training. It starts with, do we get people coming to training? Are they engaging? Are they learning? And then are they actually using that learning back at the organisation and there is the impact from that learning? And as of yesterday, we've now got the Ethan Zuckerman when you look at the symbolic impactful light through to heavy. So we're only talking about the X-axis here. The thin and the thick again is more about how rather than what. So I've added those two in here and you can see again if there's flow from lighter symbolic acquisition, grabbing people with that sort of thing through to the heavier the idea of changing lives, of tar success, of having an action of results impacting the business, of impactful engagement. So I started developing a new spectrum for digital products that started with the lightest engagement and moved through to more successful impacts. Came out with a high level of attention and motion action. Pretty basic, pretty straightforward. And nine steps underneath the user's scoring system. Lots of testing, lots of talking, some yelling, lots of feedback. I'm looking at Rob here for the yelling bit. And lots of discussion around what we found was, people liked it but it wasn't quite right. They didn't trust it, it was one of those things that I saw and they found too many ways to pick it apart. So I tried again. An attempt to be much more in a sweet spot. So it's now a five-step model with a 1 to 10 ranking for further clarity and more tangible scoring. And it really, I think, focuses more on the idea of having qualitative and quantitative together. And using the quantitative to support pulling the qualitative measures up higher. And it provides, really importantly, a structure for storytelling. So these are the five steps. The tension, we get someone. Reaction, they react to whatever it is that we've provided for them. Connection is they find a reason to actually want to engage with this. There's a connection between them and the product or the story or the experience. Insight, they learn something. Action, they actually go away and do something. There's some kind of something that actually happens. And it looks like this. So we've got the 1 to 10. The five are broken down into 10 more tangible, scorable spots. And that looks incredibly boring and incredibly dry. It's when you actually put the real data in that it starts to make a lot more sense. I know you can't read that so I'll pull out a few of the highlights. Most of this is real data. Some of it is anecdotal. But it's based on experience and some of the data that we're gathering. So attention. We're good at attention. Tepapa is very good at attention. That's getting people. So that is getting people to click on our social media. It's getting people to read a label. It's getting people to walk into an exhibition. It's getting people to click on a website. Immediate response, we put a lot of effort into this. This is where a lot of the design originally goes. This is getting people to actually enjoy something or feel an emotion or have a reaction. It might be sad. A lot of people come out of Gallipoli feeling sad. That's a great response that we might have wanted to do. You might have jumped with surprise. You might be repulsed by a spider. That is an important part of actually moving further down the spectrum. And then there's personal connection. So my grandfather was in the war. I wonder if his gun was like that. Thinking about the stuff down in Gallipoli. We're developing a new nature exhibition that will open early next year. And one of the things that we're talking about is river health and the amount of ills that are around now. I remember when I was in Wai Paoa growing up in the Wai Paoa River there were places you would know would be ills. Now there's hardly any. So that is the reaction or the emotional and personal connection that I can have that will pull me into wanting to go further down. Well, Tepapa has a journal on replay station game. That's why we have some of these things in the collection and we tell these stories because it is about the personal connection. Simple learning. We want people to learn basic stuff. So learning basic treaty facts. The idea of finding out learning that the two versions of the treaty were not the same. That's really important. It doesn't tell you a lot but it gives you the beginning of understanding sure, that's okay. Wow, look at all those kiwi feathers. It must take a lot of birds to make that kura wai, that cloak. I learned a word of te reo. I learned there was more than one kind of moa. I hadn't even really thought about that before. So really simple learning. We're good at that. A lot of museums and libraries and archives and all that kind of stuff, we're good at that. Then it starts to get a little bit harder. Contextual learning. Is where you actually start to question something about what you thought before.  But now, when we look at her story, I just had thought of her as a drag queen before. I hadn't thought about her as a hero of her society and that she was a business woman and all of these kinds of things. Start to question stuff a little bit. Seeing where your opinion fits in relation to other people's opinion, actually understanding that. To level five. Level six, applied personal learning. Actually starting to change the way that you think a little bit based on what you've learnt. Changed my mind about something, understood how the treaty fits in my job. Never really understood why it was relevant. I understand a bit more now. I had a talk with my family about war and we talked about what we thought, what was right, what was wrong. Able to see art in a way that I haven't before. Moving on, applied empathetic learning is actually starting to think about how that learning and what you've learnt affects other people. I'll read the labels or other things. I'll read the labels on seafood products more carefully now and I'll look for sustainable ways of fishing. I talked with my kids about the impact of pets on wildlife and what we thought was appropriate as a family and with our pets. Felt confident at work in a discussion about when to rail shouldn't be used. This person might feel more empowered to actually have that discussion. I'll look at how Māori are represented in popular culture more carefully now. We're not into action yet but we're starting to get people to empathise with others and share that empathy. Then we get into actual action. With the nature exhibition one of the things we want to do is get New Zealanders understanding the impact of pests on native wildlife and that sort of thing and actually doing something about it. We might sell pest traps down in the store and we want people to leave the exhibition be moved enough to go, I'm going to take action, I'm going to buy a pest trap, I'm going to install it. That is something that would be a significant success for the exhibition. Committed to using a few words of te reo in my daily life and actually doing it. Going to keep my cat in it now. Again, at night. That's an action that we would be an amazing benefit. Better earthquake proof, that water cylinder I guess. This is a real one where a couple of years ago we made the Minecraft shaker mod in earthquake simulator in Minecraft and a lot of things fall over if you don't stabilise them. We had kids playing the game and we had parents to go and do stuff to the house to make sure that it doesn't happen in real life. And so again, that's personal action. Made, wrote and created something. We don't record that kind of stuff enough as a measure of success of what we enable. So that's personal action. I'll come back to, I think I've finally decided to go back to school because that's a real one. Then moving on to nine which is group or community impact. What we've done has enabled somebody else to do something for others. Has created an impact on more than one person. So this is a real one. Created their own guide for an exhibition for others when the art galleries were closed when we were redeveloping toy. We put a guide on the website that says there's still art in the building. This is where it's scattered around and you can find it. A young man who was on the spectrum came in and he did that. He really enjoyed the guide because he was able to find his own way around. He's next to a lot of other people. He could do it all himself. He liked it so much he went home and made another one for his classmates. Started a stream restoration project in my street with my neighbours. That would be pretty impressive. Why aren't we designing for that kind of stuff more? This is a real one to use for Māori myth and legends component of our treasures of Tane's Zoo School programme at Orana Wildlife Park. Again it's making something but it's making something for other people that has ongoing educational benefit. That's from collections online as a result of our high resolution downloadable images. We can actually talk about national impact as to Papa. There's a number of things that we do that can enable national impact. We might provide some content that someone is able to turn into some really successful movie or something along those lines and they generate and come for New Zealand. Secondary and economic benefit in the money doesn't have to come to us but we've done something that's actually affected the economy. Aided research to find the original pink and white terraces. We're doing that again with the content particularly using our historic content to help digitally recreate and also locate the pink and white terraces. And affecting policy change at government. There's lots of ways that all of our organisations can do that kind of thing. These are all very general but what it shows is that there are real stories that show that we can move down this spectrum. But it is a funnel. We're not going to get everyone who comes in the door doing national impact. That's just silly. It's just a really indicative funnel. If we get a certain number of people into the exhibition, into a digital product, whatever this could be the user flow that we have of losing people as we move all the way through. We might only get 0.01% of that first number thinking about doing national impact. There's still one. One out of 10,000. This isn't going to apply to every product or anything along those lines but it just goes to show that when you think about the funnel or the design for it, there's a lot you can achieve. So some real examples. Artwall is a little kiosk where there's about 400 paintings and that sort of thing on there where you can scroll through. You can pick the one that you like. You can say where you like it, where you're from and you can put it up on the wall and everyone can see it and see why you liked it. It's just come down but it's in toy art. It's actually on display in the trade stand. We wanted to make sure that this wasn't a digital projection. So we actually put some measures against it. 500 user sessions a week, we thought based on the traffic that we have, some testing and observation that we've done, we thought that might be about right. We want the average session length greater than three and a half minutes. That means that people are actually moving through, they're looking at all of the stuff, making some decisions. This is still at level one though. We want three or more artworks that you can scroll through thumbnails. Click on one and go, no, not that one. Close it, keep scrolling. Again, that shows that people are really actually using the work. Immediate response. We want 50% of sessions resulting in artwork being successfully sent to the art wall. 50% of submissions include comments so there's an actual action. It doesn't matter what those comments are, we want 50% of the people to make a comment because that shows some kind of response. We want 20% of comments to indicate emotional response, draw, intake. This is all real data because the comments are the data. We want... I'll just jump ahead because I'm running out of time. We want 5% of comments to indicate new art confidence. So this is actually a target for it. We're not just going, this is a shiny thing. We need to design for these targets when we're doing it. We'll move on to another one really quickly. We were talking about making a digital discussion platform for Suffrage 125. This is the year to do it obviously. We thought this is something that maybe we could have National Retro on. You don't have to come to the building but we would have a building kiosk that did it as well. So based on the traffic to level 2 we thought, okay, we want 80% of people to level 2 to see it. We want 60,000 campaign homepage views online over the year and this is all based on the... We kind of know the numbers. We want 10,000 on-floor visitors to start a campaign view. 10,000 non-textual responses left. That's there because we know that not everybody is willing to do a textual response or put their voice to something that they may be a little bit nervous about so we need to design a way for them to have access into this conversation. 30% of surveyed visitors we want to have a new perspective on Suffrage beyond voting. We want 5,000 comments and it moves down to, if we just jump to say 7, 5% of commenters report having had a follow-up or ongoing discussion with others since visiting. 2% of commenters report having made volunteer mental or charity contribution as a result. 1% of commenters having made a change in their lives, e.g. asking for a pay raise they weren't going to applied for a job that they were too nervous to starting a sport that they thought was too rough or changed a home situation or chore. These sound lofty but the point is that they're supposed to be because now we have to design for these comments. We're going to do it. How do we get those kinds of comments, those kinds of impacts to happen? We have to design for it up front. Collections online has 500,000 visitors also a year. This is all real data. We gather lots and lots of data about use and reuse every day from people who are using it. We've got all this stuff about I liked it. I'm using it as a nice picture for my phone wallpaper.  That's great. That's our most popular task completion but it's only a level 4. Lots and lots of level 4s, so that's great. Lots of other things moving through but we've got lots of examples of people creating things as a result of using our collections and our images. We've got comments indicating teaching use and again the idea of reconstruction of a national site. We have gathered all of this data from people who are using it. This is the example where first of all I realised that this wasn't just a digital spectrum. This was about anything that we do. When nature closed the natural science exhibition that had been open for 20 years. There are people in Wellington who had grown up with that exhibition and had come in and seen the giant squirt, the birds, the wetas and the trees and all that kind of thing. That had hidden in the trees from the parents. We were shutting it down so we held a closing party and there was disco, there was DNA themed cocktails, all of this sort of stuff. You might think for that kind of stuff we would only design for, and we did to be honest, a four or a five. Simple learning, maybe contextual learning. There were curator floor talks about this is what this bird likes and all of that kind of thing. So you get access, what we didn't necessarily design for but is what we enable through the one-to-one access to curators and that kind of thing is we convinced an adult student to feel that her passion was worthy now and that she wasn't afraid to go back and do it. Why can't we do with this one-on-one stuff when we have people start out on the floor talking to our public? We've got a lot more opportunity for impact there. Giliplee is a really interesting one. This is all retrospective. One of the things I found interesting about this is when you think about personal action, most of the stuff that is in there is sort of level four, level five. I learnt about the war, I learnt about the Turks. There's some empathetic stuff here, so the idea of during the commemoration thinking about the Turkish side, not just the Allies side and the impact that it had on those as well. Everyone was people. But also the idea of potentially negative impact. Where do we go with negative impact? So I decided not to join the army. War is horrible. What if we convinced that they did need to join the army to protect the country? Is that okay? I don't have an answer to that. Everything is based on use of success, not on to Papa's success. Reaching the higher levels is difficult and it's difficult to measure and it requires new ways of thinking and I think it's one of the reasons why we haven't got there yet. Not every product will hit level 10 but this model asks how far can we go. We were happy designing Artwall to get to level four that fits well with what that product was. Obviously the different products are different goals. Allows for the shiny to sit alongside the deep very obviously. And the models definitely designed to be aspirational. The models also designed to be applied at product level, at exhibition level, at programme level and at organisational level. And what you can do is come up with things like this is how are we doing organisationally across our programme. Are we happy that we've got this much impact and we've got lots of volume down here? Are we happy that product C has got hardly any impact but it goes, sorry, hardly any visitors so the numbers are very low but it's getting all the way down to lots of personal action that might be through an education course. It's a design tool, a prioritisation tool, a testing tool, a success measurement tool. The idea is it should be used not just at the problems of validation stage it should also be used as a decision making tool it should be used when you're validating your solution any decision and point it should be used should be made as your building and testing it should be used as your in-high care when you've just launched it, how well is it doing and it should be used throughout its lifespan and it should be used to figure out is it still doing what we thought it would do. Measuring success is hard how do you measure the impact if it's five years after the visit and again this is one of the reasons why I think we haven't done it. We can use our channels, we have people Te Papi has amazing networks and so does most of the sector so we can use the Ministry of Culture and Heritage, Ministry of Education Schools, Universities, find out the stories that they're hearing about us and about the impact of visitors so art schools, how many people were influenced by, how many new students were influenced by Te Papa in choosing their school and use those channels we need to do more social media monitoring we need to do follow up surveys and repeat visitor interviews when people leave Te Papa they do an exit survey that says how did you enjoy your visit? Why don't we add into that how did you enjoy your previous visit and what impact did that have on you in the last six months or whatever we need to automate the earlier stages more so that we can put more effort into the later stages but we just need to be more proactive, we need to ask people and get their stories and not rely on the numbers it can be used in all kinds of things and there is this is a movement that is happening everywhere this is a major science funder in the UK and they want to know what economic and societal impact is as well not just academic so there are some weaknesses possibly not backed by enough no-one is that linear in their motivations it's a pretty simple flow but it's better than nothing different cultural groups and audience segments reacts to emotional engagement in different ways and there is research around that some levels are open to interpretation and it can be cheated so you need to keep an eye on it and it defies a complex need and yet at the same time it's too challenging for some people it exists, it's pretty simple it's designed for our sector we need to ask people more about what their impact is going to keep working on it going to make it easy to use it's not that simple at this stage I know it's at the beginning but we are going to start using it to map to Treasury's four capitals so when we report to the Government we've just been reporting numbers and that kind of thing the Government in general wants to know more about what real impact do you have not just on the finances we give you money and you tell us how many people came how are we actually starting to be good for the nation and as of yesterday it's like why can't we even just map to the UN's Sustainable Goals because this is the kind of stuff that we're working to as part of our work sorry that was bad


Howdy how's it going? My name's Davy Chappie, and I have made many characters throughout my years of tabletop roleplaying, and one of the most annoying things in the world for me was spending my time working out a character, coming up with a cool concept that I could bring to the table, and then I roll for stats and everything goes down the drain. The truth is, I'm not the best at rolling dice. When I was a kid, my group and I were constantly making new characters for campaigns that never happened, and it always felt really shitty to just find out that all of my stats were immediately inferior to somebody else's because I made the poor decision of tossing my squares the wrong way. Or maybe I'd roll decent, but other people around me rolled poorly, and suddenly now I feel shitty because for no reason at all, I was put in a higher position and I feel guilty about it. But then 5E came in, riding on a golden horse, and upon its back, it carried the solution to all my problems, the point buy system. Finally, a way to create characters without having the party members be unfairly lopsided right out the gate, and so I took point buy, I ran back to my fellow roleplayers, and I showed them the new religion, cast rolling for stats into the fire, destroy it, and my fellow roleplayers turned back to me and said no. Further inquiries as to why everybody was against point buy, and also why we play our games on a steep cliff just above an active volcano, led to pretty much the same few answers. It's not what I know, it's too complicated, I like randomness more, stop raking into my house and asking me questions. For the first thing, I get it, change is weird, nobody likes change, that's why I use a credit card. But if the only thing stopping you from using a better system is that you weren't indoctrinated into it from the very beginning, then what are you doing watching this video?  And to the next point, people think point buy is complicated, but it really just complicates itself by 5E being bad at explaining it. Here, point buy sets all your stats at 8, and then it gives you 27 points to put wherever you want. Increasing a stat costs a point, except when you raise it to 14 and 15, which costs 2 points, and then it caps out at 15 so that you can't engorge yourself on one stat. After you're finished with point buy, then you add your racial bonuses, and that's it. Now you know how to use point buy, you're welcome. Some people say that they don't want to do math, and I say, tough titties, the game is math. And if you want to be lazy, just go online and use a point buy calculator like I do. But if that still doesn't sway you, if the lustful call of gambling with your plastic sugar cubes is too alluring, if you just can't get high without that level of randomness, then go ahead, your party can do whatever it wants, forget about me. But I also know that a lot of people aren't even giving point buy a chance because of all the purists out there who think that rolling for stats is somehow the only real way to play Dungeons and Dragons. Fun fact, 5E was made for point buy. You want to know how I know that? The Fighter and the Rogue. In an effort to balance out these two classes with the rest of the choices, Wizards of the Coast added more ability score increases to the Rogue and way more ABI's to the Fighter in order to show that they were more marshally and or mentally capable than the other classes. The point of the Fighter is to have higher base stats as a means of keeping up with the guy that can cast firebearers out of his hat and it screws with the Fighter's balance when you de-level the playing field. With rolling for stats, if the Fighter rolls really high, then they just end up picking feats all day because the decision-making process, where they have to make a choice as to whether to take a feat or an ABI, is completely taken away. But that's not so bad.  You want to randomize your stats?  Or, better yet, go ahead, roll for stats and then, if they're too high, you can lower some of them so that they'll fit in with point-by. Or, if you rolled too low, you can spend the difference in points however you want. It's still random, it's still weird, but it doesn't break the game balance before the game has even started because rolling an 18 for stats and then getting a plus 2 from your racial bonus means that you get a 20 at level 1 and having a 20 in an ability score is supposed to represent that you are the apex at a given thing that nobody outside of inhuman beings are capable of being better than you at a given talent. Starting at level 1 and already having a 20 in something is like rolling up a new character who killed a dragon in his backstory. You just want to be the best at the game and you don't particularly care if it makes sense. Even Jarlaxle Bainry, the DND King of Charisma, only has a 19 and he's a G. But, in the end, what it all comes down to is personal preference. I can talk all day and all night about how much better point-by is than rolling for stats. I can have all my players use point-by so that I can indoctrinate a new generation of roll players who will instinctively oppose rolling for stats on principle, but really, much like rolling for health, it all comes down to whether you want to take a gamble or not. If you want, let people make the choice to either buy or roll their stats, and hey, maybe it can convince your teammates to allow the temptation of point-by to sway them into giving it a try sometime. But, that'll about do it. I hope you enjoyed this video. I know it was a little shorter, but be sure to share it with all your tabletop friends and maybe support me on Patreon so that I can spread the good word of point-by to all who will hear my whines. But yeah. Davy out.