I was a communist for the FBI. Starring Dana Andrews and an exciting tale of danger and espionage. I was a communist for the FBI. From the actual records and authentic experiences of Matt Sevetik, come many of the incidents on this unusual story. Here is our star, Dana Andrews, as Matt Sevetik, who, for nine fantastic years, lived as a communist for the FBI. Matt Sevetik, you can read that name in the official reports of the FBI. Nine years worth of reports of communist conspiracy against the United States. Life-long friends fit the word commie at you, and sometimes you wonder if it's worth it. This is one of the stories that answers your question. In a moment, listen to Dana Andrews as Matt Sevetik, Undercover Man. Now, here is Dana Andrews as Matt Sevetik, Undercover Man. This story from the confidential file is marked, Red Clouds on the Good Earth. It had been a long, hot summer with the usual news headlines, hottest days since 1896. On this particular night, I was in my small hotel room, typing a report to the FBI. The window was shut tight. You never knew when the party might have planted someone close by. Even the sound of a typewriter could arouse suspicions. Comrade Sevetik might have to explain. It was my cell chief, Comrade Belisky. Comrade Sevetik, you will report to the dancing school at eight o'clock. Tonight? Look, I'm beat. This is a special meeting. Belisky, I've pounded the sidewalk from one end of town to the other, collecting dues all day. I tell you, I'm all in. Comrade Sevetik, I suggest you read Comrade Willis' excellent article on personal emotions in this week's issue of political activities. The meeting is at eight o'clock, be there. Drop dead. That's the first happy thought I've had all day. Comrades, we have received special orders. More infiltration must be done in the rural districts. We must build the party rapidly and increasingly among small farmers and agricultural workers. And most important, we must recruit the youth and assure them that the Communist Party of the United States will fight their great enemy, the monopolists, until they are beaten by the... I listened to Belisky with my third ear while I looked around Madame Irina's dancing school. Something new in Russian ballet, huh? But like the rest of them, I turned my attention back to Belisky as he snapped his orders. I call your attention to this map. It isn't the northern section of the state. There are many small farmers of foreign extraction. Comrades have already been placed on some of these farms as farmhands. Up here is ballet center where the annual harvest festival will be held in ten days. This is our focal point of operation. Comrades Johnson, Kadov, and Sonja Danecek, you will report to Comrade Storm for instructions. Comrade Svedik... Yes, Comrade Belisky? You will work directly with me. Please wait. Now, all of you, we must start recruiting among the farmers immediately. Any names of potential members you can get, please turn in as quickly as possible. That is all. Comrade Svedik, let me show you on the map the territory you are to cover. You will start from here, Lindenville, and work your way right up through to ballet center. I see. You'll start by planting the idea of organized youth clubs where special study can be given to specific problems. Distribute copies of our magazine, political activities, perhaps even solicit articles from the young people. What happens at ballet center? At the harvest festival, we'll contribute to the entertainment by showing motion pictures. Our propaganda films, of course. Our educational films. I'll be there to act as narrator and drive home strategic facts in the film. Any questions, Svedik? Yes. What's my cover-up for being there? We've secured a job for you as a traveling salesman for a farm supply and implement house. Comrade arranged it. This will bring you into contact with the farmers. Oh, incidentally, there's a farmer at ballet center, Mark Pavolsky. He is quite a leader among the young people. Colovate this boy. Try and stop at Pavolsky's house if you can. It'll be convenient. Any further questions? No, I guess not. Very well. Pick up your sample case first thing in the morning and leave. Good night, comrade. Good night. Oh, comrade Belisky. Who pays for transportation?  They always furnish cars for their salesmen. A good joke on their reactionary boss. Someone should only know, eh? Yeah, someone should only know. But you won't be traveling along. Why not? Why should you? Well, there's no reason except that I suppose I'm what you might call a lone wolf. Yes, I've noticed that. And I don't like it. May I remind you that even wolves fight in collective packs? I'm going with you also as a salesman. Any objection? No, no, no, it'll be a pleasure. We'll cover the whole thing like a rug. Right down to Farmer Pawalski. Can I, comrade? I'm phoning from a drugstore. I just left the meeting. What's up? Infiltration of the farmers. I'm leaving on a trip. What's the bitch?  At least that's what they say. But I've got to hunch there's more I haven't been told about because... Well, so long, honey. I didn't get a chance to tell the FBI agent my destination because a glance over my shoulder had told me that Comrade Maliski was moving close to the phone booth. Oh, hello, Maliski. Good day. I was just going to phone you. Isn't it a small world? What about? I thought we'd plan our route tonight so we can get an early start in the morning. Tonight? Well, as a matter of fact, I just made a date. A very red-headed date. But of course I can break it. Why break it? Maybe share the friend for me. Comrade Maliski, personal pleasures before party business? May I remind you of Comrade Willett's excellent article on personal emotions in this week's issue of political activities? Oh, of course, of course. You won't mention this. No, but what's that kind of talk, Maliski? Others are not so generous. Now, let's get something to eat and talk over our plan. I knew Maliski's suggestion about a double date was an act. The party's best stool pigeon and responsible for at least two deaths are, pardon me, suicides in the Communist Party. Yes, Maliski was very good at ferreting out traders and I had a terrible hunch that he wasn't coming with me just because he liked my company. Lendonville. Communist clouds gathering over a peaceful community. Communist seeds of agitation had been dropped into rich American soil by the party members who had been working on the farms. Now it was time for the killing. Comrade Maliski was in his glory as he made like little Joe and addressed groups of farmers, kids mostly, at secret meeting places. It is in the hands of you young people, the farmers of tomorrow, to decide the fate of the soil. Even now the capitalists are speeding up a drive toward war. West Falls. You can join in the great strength ring of the world peace forces which will weaken the present world capitalist system until your soil in peace and plenty or stand idly by and let them take away your plow and hand you a gun. Is that what you want? No! I met party workers I'd never known about. My list of names was growing. I'd have a pretty big report for the FBI when I got back. Corrections lead if I got back. With Comrade Maliski sticking to me like a mustard plaster, I still hadn't found a safe moment to make contact with the FBI. Valley Center was our next stop and I kept wondering what would happen there. How many more miles to Valley Center, man? About 32. We did all right with those kids in Jasonville, didn't we? Yeah, we did great. They ate it up. Driven into war, it's pretty good. Every word of pearl, Comrade. Thank you. Uh, Zabeliski, how about this plan to show propaganda movies at Valley Center? Sounds kind of weak to me. I mean, we gotta dream up something stronger. We have. Oh, what gives? During the movie at a given signal some of our party members planted in the audience will rise in protest to pick their picture up. That will be our chance to jump up and demand our rights as Americans to see and hear for ourselves. Then the riot. Violence, arrests, and you recruits who sympathize will join our party. Exactly. It'll work. It always does. Hey, slow down. We're no good to the party dead. I'm just trying to make Valley Center before dark. You're not scared, are you? I was scared plenty. This riot act was a small detail Beliski hadn't told me. Why? Just how far didn't he trust me? More than ever, I'd have to watch my step. But this planned violence at Valley Center, I knew what could happen. I'd seen it done before. Somehow, somewhere, I had to lose Beliski long enough to make contact with the FBI. Back to Dana Andrews starring in I Was a Communist for the FBI and the second act of our story. When we reached Valley Center, we could feel the excitement in the air over the coming festival. We headed straight for the general store displayed our samples and established our excuse for being in town. Luck was with us, if you can call it luck, because Mark Powalski was in the store too. It didn't take much effort to maneuver an invitation to stay at the Powalski Farm. There was just one thing wrong. Comrade Beliski declared himself in on the invitation, and the odds against my losing him long enough to make that phone call to the FBI were getting too big for comfort. Well, Mr. Powalski, you're very kind to invite us to stay here at your farm for the festival. I hope it isn't too much trouble. Trouble? Of course not. When I meet man named Servetek at the general store, in my own country, I say this man and his friend stops only at Powalski Farm. And tonight, my daughter Paula cooks for us only old-style dishes. You like that, eh? Yeah, that's great. Mr. Powalski, I have to make a long-distance call. I'll pay for it. Sure, sure. The phone is in kitchen. Thanks a lot. Oh, here's your friend Mr. Beliski back with our journey. I must go tell Paula to make supper. Mr. Beliski, I never thought I'd like that. Pop, Mr. Beliski's been giving me some figures on the consumers market. Sure? Oh, Mr. Servetek, you want to make long-distance call now? No, I'll do it later. There is Paula. Paula! Paula! Who are you going to phone in long-distance, Vedek? I'm just going to report that we've reached here. Report our location, you know? No, I don't know since I'm in charge. Hey, Pop, come here a minute, will you? I want to ask you a question. A question? Yes. Did you ever think of the tremendous cost of farm produce to the consumer? The profit's all along the line, but the farm only grosses a few pennies on them. So what kind of a deal is that? Johnny, you don't understand. There are many problems before the consumer gets the product. Transportation, canning, labor cost, advertising, this is big business. I wouldn't want to change places. Why? What's so great about being a small farmer all your life? That, Mr. Beliski, is something only a farmer could understand. Excuse me, Mr. Pavalski, I just... I know, I know. It's only too much talk I hear lately about the profits. It is dangerous. I'd better go to the barn now. Oh, hi, Pop. What's your hurry? Wait, Pop, I'll go with you. Paula, we've got company. Oh, Mr. Vedek, this is my sister Paula. Someone should have prepared me for Paula. About shoulder height, my shoulder that is, blue eyes and blonde hair that proved nature was right once in a while. Kamrat Beliski made some inane smirking remark about the traveling salesman and the farmer's daughter. And I made an inner non-proletariat prayer that the Pavalski kids would say no thank you to the particular brand of poison we were selling. I guess we're all here, Mr. Vedek. Okay. Hey, kids, this is Mr. Vedek and Mr. Beliski. They wanted to meet you and I think you ought to hear what they have to say. Johnny, you'd better close the barn door. Let's keep this private, okay? Oh, sure, Mr. Beliski. I'll take over. Okay, but go easy. You never know about these kids. Uh, Mr. Beliski and I have been traveling around the country and we've seen a lot of things that concern the small farmer. Like what, Mr. Vedek? Well, for example, what would you say is the biggest problem on your farm? Well, that's easy. The cost of irrigation. But what can we do about that? Alone, nothing. But the monopolists who are squeezing the agricultural workers to the wall with their ever-increasing high rates can be beaten by the fine efforts of all the workers, both industrial and agricultural. Industrial? How could the factory worker fit into our picture? I don't understand. I'll answer that one, Paulin. Right now, workers all over the country are standing by ready to strike if necessary to bring about better conditions for the American farm. Don't ever mention the word strike to paw.  And commies, as he calls them, have horns. What does he know about the Communist Party of the United States? Or, for that matter, what do you know about it? Well, just what Pa says about it. Oh, what we read in the papers. Only here say, and yet you judge it. Is this the American way? Do you read in the papers that the drive to war is impoverishing the workers and farmers of this country? Do they tell you that the young farmer of tomorrow... Wait a minute. I think I heard a car. Why must this meeting be so secret? Because people brand the truth communism. And we bring you the truth. Quiet, Belisky. Come on in. Oh, you're having a club meeting. Don't let me interrupt. I've been visiting a patient down the road and I'm afraid my poor old car is thirsty again. Oh, that radiator. I'll get some water. No, no. I can find my way to the pump myself. Thank you, Johnny. But, uh, I don't think I know these gentlemen. I'm Matt Severick, Dr. O'Neill. This is Mr. Belisky. We were just passing through. I see. And I trust you find our young people interesting. But please don't let me interrupt. I can get the water myself. I knew Dr. O'Neill was suspicious of us. Belisky knew it too. That's why he gave me the high sign to follow the doctor upside. Which was just what I wanted. I had a hunch it was a long shot, but it might work. Dr. O'Neill, I'm over here at the pump. May I give you a hand? Oh, thank you, but I have enough water now. Then I'll carry the pail over to your car. All right. Thank you, Mr. Severick. We're very proud of Johnny Puvowsky and the other boys and girls. They've done some good progressive work on their farms. Yes, they're fine kids. All right. Dr. And I know that nothing will ever change them. I like to believe that of all our young Americans. So do I, doctor. So do I. I'm glad to hear you say that, Mr. Severick. Oh, well, here's my car. Let me put the water in for you. Careful of that radiator cap. It gets very hot. Don't worry. Licky radiators are my specialty. I've never had a car yet that didn't have one. There we go. Should I have the thing fixed? I know. There are many more things these days that need to be fixed. My little car seems unimportant. Oh, thank you, Mr. Severick. Oh, Dr. O'Neill. All right. I've got a favor to ask you. Of course. What is it? Will you make a long distance call for me? But Mr. Puvowsky has a telephone. Well, I can't explain. Well, suppose I say, trust me, doctor. I'll make your call, Mr. Severick. Thank you. Here I'll write it down for you. This is the number of the city. That's for Stevens. Tell him Severick must have more samples sent to Valley Center before tomorrow night. Say it's a hot line. He'll understand. Very well. Oh, and here's some money to cover the phone call. Please take it. Thank you. I will. My labor is skilled but not highly paid. Goodbye, Mr. Severick. Why? It was the big night. Park calling contest, square dancing, a midway with games and prizes. For a brief moment, I was just a guy out with a girl, Paula, having a ball at the country fair. Test your strength, Mr. Ring the bell and win a baby doll for your bear. OK. Paula didn't know how much strength I'd need to carry me through the night and to take me away when it was over because pretty girls and nice people were not in the car for Matt Severick. Not now. I'd be rioting in the movie tent soon and hysteria. I'd had no contact from the FBI and it seemed that I was strictly on my own. The propaganda movie was on the screen and Belisky was narrating. Here are the farms of the present under the Soviet Union. See the fields of plenty. I looked around the audience. I knew which of the party members scattered around were ready to start a protest against the film. That would be the signal for Belisky and the rest of us to give the pitch that this was proof that Americans were no longer free to see the truth. And I still didn't know if Dr. O'Neill got the call through to Steven. I could only hope about ten minutes more and the fireworks would start. I heard the tenseness creep into Belisky's voice. Under the Soviet Union and here my friends are actual pictures of just a few of the many small farms all over the United States that are being forced out of existence by capitalism. Suddenly in the middle of Belisky's narration something happened. Someone had started a riot all right, but they weren't communist party members. Who tipped off the cops? How should I know? Who tipped off the cops? I don't know, you don't think I had it? I never trusted you, you did it. You're crazy!    I did. You, Johnny? Yes, me.  Because you got it coming to you. You must think the kids around here worked. Well, we're not. You called it yourself last night. When you said we were the farmers of tomorrow for your information, John, we intend to defend that future. Come on, Sveta, let's go. What happened just now, the fight between us? You won't report it. No, I won't. You'll have enough explaining to do about how your plans got fouled up. The party won't like this. I know, I know.  Sveta, you better go back and see how many arrests they made. I'll go get the car. All right, I'll meet you on Main Street. Mr. Sveta, Mr. Sveta! I've been trying to find you. Oh, hello, Dr. Anil. I'm very sorry, but I couldn't get your call through. Mr. Stevens was out of town. I've been wanting to return your money. Here you are. Thanks, doctor. Don't worry about the call. It didn't matter, after all. Hey, may I give you a lift? No, thanks a lot. I'll go on by myself. Just as you say. You take one last deep breath of the fresh country air. The earth feels good beneath your feet. The good earth. You look back at Valley Center and wish you could chew the fat with Johnny or visit with Mr. Pawalski or walk into the stars with Paula. But forget it. You're an undercover man.  You won't alone. Dana Andrews will return in just a moment. This is Dana Andrews with a word about the story you've just heard. In this story, as in all others, names, dates, and places are fictitious to protect innocent persons. Many of these stories are based on incidents in the life of Matt Sveta who worked undercover for the FBI. Next week, another fantastic adventure. Join us, won't you? Thank you.


Hi, my name is Julia Silge and I'm a data scientist and software engineer at RStudio. And I want to thank our medicine so much for having me here today to talk about data visualization as it's used in real world machine learning. So data visualization, this hugely important tool for all kinds of data practitioners, not just those who are building machine learning systems. And a lot of what we're going to talk about today is true because data visualization is so effective at surfacing information to human beings in this visual channel that we have. So when it comes to data, these visual ways of perceiving information form how we think about the work we're doing, how we understand the problems that we're solving and the models that we're building, and how we decide. And I especially want to focus on data visualization as a support or aid for decision making because building models or machine learning systems is this complicated, involved task. And I want to explicitly to frame the plots and graphs that we make the way Tamara Munzer does here. This purpose is so that we as machine learning practitioners are more effective. When it comes to models and machine learning, more effective can have a couple of different meanings. It can mean our models are more accurate. It can mean they're more practically appropriate. It can mean they're more fair. And we'll talk about these and how they work and how we use data visualizations to do these. I work on an open source framework in R for modeling and machine learning called tidy models. And a lot of the examples I'll be showing today use tidy models code. I'm glad to get to show what I work on and build, but a lot of what we'll talk about and walk through today isn't specific to tidy models or even are really we're focusing on data visualization practices for real world folks engaged in building models. So let's start by asking this question, when do data practitioners build data visualizations? Asking this question requires us to think about the process of model building to build a mental model of model building, if you will. So I'd like to put out this sort of schematic here as what might happen during a typical model building process from when a practitioner first sees some new data to having a working model. The model process starts with exploratory data analysis, EDA. This is a systematic iterative process of exploring data to investigate it. Next comes feature engineering. This is this task of transforming our original variables to a more useful feature for our model. Then we have model fitting and tuning. Notice that I've represented this with lots of little skinny bars to represent that we typically for with breakfast, when we're using best practices, we typically do this using resampled data sets. And we often we do we don't know a priori what kind of model we're going to use or like what will work best so we've not don't try one model. We try multiple different kinds of models. Next comes model evaluation. This is when we measure how well a model performs using some metric that is appropriate to that specific kind of model and problem that we're working on. Are we done? No, we're probably not done. We're probably not done at that point because this whole process is usually iterative. We take what we learned during that first process and we go back and we do a little bit of more exploratory data analysis.  We go back to the maybe the two models that look like they were going to turn out best and we fit and tune we tune and fit them again using the resampled data and then we know we evaluate again. Also notice that this whole process is ending with some goal or output that might be communicating the results of the model or might be deploying the model to some kind of production system. I represented this here with another pretty thin bar but picture this really as transitioning to a whole other set of tasks. In this talk we're going to keep the scope of what we're talking about on this iterative process of developing a model, developing the model itself. So machine learning practitioners can of course make plots at any point during this process but I want to make the argument that there are two phases of this modeling process during which practitioners make data visualization the most. The first of these is exploratory data analysis. I said before that EDA is this iterative phase of investigation, question generation and refinement and exploring. This is a phase that maybe some of you may be saying wait a minute is this really part of machine learning but we're going to walk through examples and an argument that where we see that these investigations and the understanding gained during EDA have a huge impact on models. The second part where people make a lot of plots is during model evaluation. This is a part of the modeling process where a practitioner has trained multiple models and is assessing, measuring, comparing how well they performed. The reason why is that these two phases of the machine learning process are the points of the most important human decisions. These are decisions where validation of them can be difficult.  And visualizations are used as an aid or support to carry out these decisions, these tasks more effectively just as we talked about at the beginning with that quote from the Tamara Munzer data visualization book. So let's dig deeper into this and start with exploratory data analysis. And let's ask this question, why are these plots built? So at a very high level, these plots are built to understand the data. Understand the data or the input to your models. So when a practitioner starts out on a modeling project, this is one of these first substantive tasks being faced. The reason we as practitioners need to have this understanding of the data is because there's about to be a whole bunch of decisions to make. These data visualizations let practitioners ask and then refine specific questions related to our modeling goals. Like what kind of model is appropriate? This leads into the feature engineering phase that we talked about before. Like what kind of pre-processing do I need to do? And all of these kinds of questions are ones that machine learning practitioners ask, refine and answer by making plots during the EDA process. Let's look at a couple of examples. Some of these plots take the form of very familiar classic statistical graphics like this scatter plot. This plot uses a data set called the Palmer Penguins. Penguins are observed in an arctica near Palmer Station. And it maps each observation to a point on the plot with aesthetics of the plot like position, X and Y, size, and color representing things that have been measured about these penguins. So exploratory graphics like these are important for modeling practitioners because they help us understand the type and amount of data we have available. And they highlight, they visually highlight relationships in the data that we can expect to use in feature engineering or models. So since machine learning practitioners use visualization during EDA to make so many different kinds of plots, it's important that the tools that we use for visualization facilitate fluent iteration through lots of different kinds of graphs. So I have personal experience with decades of various scientific plotting tools. And I think that learning about and internalizing the concept of the grammar of graphics is a huge step forward. And this being able to have this kind of efficient iteration and adopting a system that applies this grammar gets you just a huge step forward in efficiency in this task of asking, refining and answering questions. To be able to get ready to train models. So ggplot2 is probably the best known system that implements this grammar of graphics and almost like all the plots that I'm gonna show you in this talk are plots I've made myself over the course of training models using ggplot2. So investigations during EDA inform decisions about what kind of models to train and also inform decisions made in feature engineering. So in tiny models, we capture data pre-processing and feature engineering in the concept of a pre-processing recipe that has steps. So you choose your variables, like your ingredients. You define steps and prepare them using training data and then you can apply them to any dataset, other dataset like a testing data or new data. But the specific feature engineering steps are typically chosen because of what we learn during EDA. So let's look at another example dataset. This is a dataset of houses in Ames, Iowa. We have a lot of information about them, like how big they are, where they're located, what was their last selling price. So this plot shows the sale price on the y-axis and the latitude on the x-axis. There's a big hole there where Iowa State University is and there's some unique and complex structure in price as you move across the city. One option we have for modeling that kind of structure is to use splines. We can use splines to build new features for modeling, but we have to decide how many spline terms to use. Two doesn't look like it captures the real structure very well. Five looks better, I would believe 10, 10 looks pretty good. 20 seems okay, but I'm starting to feel a little skeptical. At 50, I no longer believe this. And at 100, this is just silly. What's great about data visualization during the process of EDA is that we as machine learning practitioners can see and absorb this and use it to make decisions. These decisions then may end up looking like this if you use tidy models with a feature engineering recipe defined with steps that transform and prepare our data to get it ready for modeling. Whether that's dealing with factor levels or creating indicator variables or setting those 10 spline terms. So whether it's for engineering new features, choosing an appropriate model or understanding data quality, data visualization during EDA has this huge impact on what model practitioners do later on. And in some ways it's the foundation on which the rest of the model process is built. It is often possible to estimate the best answer or value or option for some of these choices that we've talked about using empirical methods. But it's nearly impossible to remove entirely the element of human judgment from these choices, or at least it's a bad idea. Machine learning algorithms are astounding. They're incredibly capable of learning patterns from large amounts of data. Vigorous adoption of EDA as part of the machine learning process is one's safeguard against ending up in a situation where you have to ask yourself whether it was a good idea to just deep fry your data. So now let's move on and talk about model evaluation. The phase where we have trained and possibly tuned models. So we have predictions to compare to true values, and we can assess how our models are performing. Let's again ask this question, why are these models built? At a very high level, these are built to understand the models or the output of our process. And here we're assessing model performance. This is the point at which we have to reckon with questions about success and failure, and ask, did this model work? The process of model evaluation also allows us to interrogate subgroups. And this is typically where discussions of model fairness comes in. A lot of the current work on machine learning bias and fairness focuses on how models perform differently for different groups. Perhaps for different protected classes that may not have been included directly in the model, but we can still see differences because certain predictors are proxies. Model evaluation is also concerned with understanding with why a model is making the predictions it does. For all of these questions, practitioners heavily use data visualization. For a lot of the same reasons that it's used during EDA, which are largely the same reasons that anyone ever uses visualization. It's that this visual channel is such an effective way for us to absorb a lot of information quickly. There are some pretty unique things to note about plots used for model evaluation. So let's walk through that a little bit. So one is that many of these plots for model evaluations involve graphical norms or idioms. So this plot shows how two model evaluation metrics, RMSC and R-squared change with the regularization penalty. Like how regularized a model is for a lasso regression model. So someone who is familiar with these kinds of models can look at this plot and can quickly recognize whether or not it worked. Whether or not it needed a lot of regularization or a little bit. And maybe looking at the R-squared, see how well or how not well in this case. This model fit this data. So it's like a shorthand or an idiom that packs a lot of information for a familiar audience. This plot is similar in that sense. This is a receiver operator characteristic curve, an ROC curve. Well, it's ten of them actually. For a model trained on that on Palmer Penguin data. And it's a visual representation of how well a binary classification model is doing. So the more up in the left hand corner, the curves are the better the model is at distinguishing between the two classes. The axes have the true positive rate and the false positive rate on them. And ideally we want the model to do a great job at both of those, at all thresholds. So this is another example of a model evaluation visual idiom that is familiar to modeling practitioners that can be used to compare models to assess a model and so forth. These kinds of visual idioms are a great fit for automatic plotting methods. When we talked about EDA, I said, you need to use a really flexible plotting framework that can be used to create any kind of plot really fluently. And let's adopt a grammar that lets you learn the grammar and then you can make any kind of plot that you want to in a flexible way. A lot of model evaluation plots are the same every single time you make them. And you can be better served by using shortcuts that are built into modeling frameworks built like by my coworkers and me or the equivalent of whatever modeling framework you use instead of copying and pasting the same code for an ROC curve every single time. So I've focused so far on model evaluation plots that show us as practitioners how models are performing. In terms of metrics like RMSE or accuracy or similar, but model evaluation activities can also encompass model explanations. And visualization is a key tool for explaining models as well. So model explanations fall into two main categories. The first are global explanations, where we evaluate what features are important in a model's predictions overall aggregated for the whole model. This plot is from that San Francisco trees example. And the length of the bars here correspond to how important that feature is in predicting, using a random forest model, whether the tree is maintained by the department of public works or not. Globally, for the whole model overall. So having a private caretaker is the most important and then we see that spatial information. Then the year the tree was planted and so forth. The other main kind of model explanation is feature importance at the individual observation level. So one framework I like using for this is Daleks. There's support for tidy models in Dal extra. But you can use it with lots of different kinds of models. And so here we create an explainer for a random forest predicting price for homes back in Ames, Iowa. And we're gonna look at what explains the price of this specific duplex that I pulled out. So not the model overall, but just for this duplex. So there's multiple ways to do local feature importance. This particular plot shows an example of shapely additive explanations for the price of this duplex. So we can see it has a price lower than the mean and the things that contribute to that the most are how old it is, how small it is, that it's a duplex and not a single family home and so forth. We can also build up to global explanations from individual observation level explanations, even to see and highlight the model's behavior for subgroups here. Like comparing a duplex to a single family home and whatnot. So for both global and observation level model explanations, these plots are used by the creators of these machine learning systems to evaluate and understand them. So at this stage of the modeling process, we're again at a point of important human decision and model explanation together with model performance can be used to make choices. So through this whole explanation of data visualization and the machine learning process, we focus on plots made by system designers and operators for themselves as a tool to make better modeling choices. So these are plots made by practitioners for practitioners. So often data visualizations center it as a communication tool and it absolutely 100% is. Like a lot of the plots that I've shown here today could be used to tell a stakeholder about a model or how it works. But we can center some of data visualizations other functions, like its ability to be used by system designers to understand a system more fully and do a better job of designing it. But whatever a reason a plot is made when we're building machine learning system, some plots are created to help us understand the data that we start with the input for models that we create. Investing in data visualization to understand the data is like laying a strong foundation. Some plots are created to help us understand the models that we create, the output. And careful visualization practices in model evaluation sets us up for success in models that are as accurate, as appropriate, and as reliable as possible. And with that, I'll say thank you so much. I want to be sure to thank my teammates on the tidy models team, as well as the folks who organize the tidy Tuesday project. This is a weekly social data project that most of the data sets that I shared today are from.


Hi guys, welcome back to my YouTube channel, this is Daniel Rossell and today I am making a novel video, first ever Android screen share, I'm just going to get rid of this because I'm one step ahead if I do that and this is crucial information if you're an Android fan so as a Linux fan it should go without saying that I'm also an Android fan and this is my current Android device, I'm actually recording this on my actual phone not on a emulator or anything like that and you know one thing that you may not have in your life is a one touch pizza button, a pizza SOS button for when you need to order pizza and these situations present themselves in you never know when you will need to order pizza. So you can see my home screen so this is an app called Apex and it's really good and it's just a replacement for the stock launcher in Android so instead of the stock so this is my home screen in Android and you can see I've tricked some settings out, I've turned off text, I just have icons and I have all my key apps here like Reddit and YouTube, SoundCloud for remixes, Google Maps for finding my way around the world etc etc etc so what I'm going to do is I'm going to I've added my local pizza store and the first step in this important process is adding a one touch dialing app for the pizza store so this this if you have Google contacts in addition to stock contacts for Android you have two so these guys in the middle here are Google contacts and this is stock contact so I'm going to go for the stock widget and excuse these all in Hebrew and I'm going to just go for the pizza store PI Pizza, Yerushalayim that says Jerusalem pizza in English and then I'm going to go so you have you can choose the shortcut I'm going to go for the direct dial shortcut which will call the pizza store in Jerusalem that I order from frequently and click okay and now it's added a widget so I'm just going to go and it's it's being put here for some reason so I'm just going to go and move him back and the next step is like this this this this is your pizza SOS button but why do a job halfway I say so you would want really a pizza icon on this and because apex lets you put custom icons on you can there's no reason not to do this so I've just called up a pizza icon and go I've gone for download image and save the image it took me a while to find a pizza icon without pepperoni as a Jewish kosher keeper I don't eat pepperoni so here's my icon and what I can do here is I can actually change the icon so click on edit so this again is apex apex it's really good I really like it and I click on change icon and then I can go for a select picture and then I can replace that default icon with my pizza icon and save and now watch the magic unfold so I'm browsing the internet and I'm like you know what I need pizza and but I'm lazy because typically when one orders pizza one is feeling lazy so I don't want the hassle of googling or even going into my contact so I just go literally boom watch this one one click pizza being orders I'm just called the pizza shop thanks for watching my first ever Android tutorial on the important subject of creating a one click shortcut for ordering pizza I hope this was of interest and check out the rest of the videos on my YouTube channel I've no idea what I've gone into being here something more videos more Android vids on the way I'm more I'm definitely more of a Linux guy or more I'm much more of a desktop guy but there will be some more Android videos coming about things you can do with your phone until next time


Okay Connor, go deep. Okay. Keep going. Keep going. A little further.  Adam does movies at 9.30 going to be talking Christmas films. I have 25 of them for you. Check it out. It's going to be awesome. Dad?


TLO what's poppin We are on kick KIC cake calm. We are not. Oh, we don't even do well I'll be splitting my time between quick kick and twitch, so We're not on twitch either, but you can leave a like comment subscribe turn on your post notification Because let's continue to watch. I mean let's continue to grow the family from Chicago to the UK and today is Sunday All right, so you know what we got man Before we get to it man, this is the live channel if you miss any of the lives on either one of the platforms It'll be here. Don't forget. We do have merch and we do got patreon Monday through Friday Yeah, get me somebody just cop the sweater appreciate you Let's get into this quintessentially British memes 73 Sounds slight on a Sunday God did All right I told Mark will come let's start. Let's go sign land up. Bring on.    Talking to that dog like that. This is his house. You hear me? That's what you get.      I was terrified. We used to bully the bullies man We we don't we don't we don't participate in bullying man.  So maybe best mate pick the gay lads one down with That's logical That is logical because you and your best mate and then the dude that's already You know what I'm saying He's the one out of the three I've got a maté level.   Oh my everybody's back there Crazy part about this is y'all got stick shift cars, so it take real effort Well, the bro is in there pressing three pedals to get this done No way No way we need to see that what happened. Yeah, though. We're showing some footage I mean it is unbelievable that everything is baked bean themed in there including am I right was that a baked bean coffin? Ah Yes, because you know, um, obviously, you know, I'm getting on in my old age now. You could say my Bit of a has been but yes I thought I put a cue on my future status as captain beanie with a big bean coffin that you can well see So you're gonna keep that are you? Oh?  It's a comedy skit It's a real dude mr. Beanie. I wonder what his diet consists of straight like Heinz bean sandwiches and so as you well know my local legends, you know a beanie done that and initially Oh, I forgot to mention the queen had a state funeral Captain beanie is gonna have a mistake funeral Didn't get it Beanie is gonna have a mistake funeral Yeah You know and here I am thinking I was gonna marry a UK woman And this is what this is like He letting her speak to him like that She didn't call it that man a BIT CH Threaten to put RAT poisoning in his tea Like you know y'all love tea and he gonna drink that tea no matter what you put in it.  He's upset that he's late.  I Know you almost caught fire, but you need to be at work today We're short staff Looking for two geese as we were stoning that Okay, asking for a friend here as anyone actually got drunk to the point where they've come back home Four o'clock in the morning. Absolutely steaming walked upstairs to the bathroom Forgetting to turn on lights and then proceed to piss in the toilet, right? And then you're like, oh, it's so dark in here. I can't I can't see like why have I not turned on lights I might as well do it now.  I am a cat or my cat has seen me Go through a lot of stages of inebriation, but I have never did that my cat. First of all is a cat He would not let that happen the first site of participation his way. He's gone. So No Sure, mate, I'm not gonna lie. This is a new level of British that I have not reached yet It's the final boss of being British Sorry mate, um, I'm from around the area my dogs gone missing Oh, we should be able to take a look just in case you've seen it Yeah, oh, you're not joking. No, no, it's the family dog Jefferson.  I already knew you left me out. I'm so sorry darling. It's just that We needed to see a picture of the homeowner for reference That's not the door that the door is boarded up. That's just a door leaning against the wall This young mailman, she's just trying to peacefully do dishes like TLL even like how subscribe turning your post man. Happy happy days, man. I'll go


We take another example from the Hogg, McCain and Craigs books that is application number 7.4.6a part. The statement of this question let a random sample of size n taken from a distribution of the discrete type. Now we are considering a discrete distribution so that you can easily understand how we can handle the discrete type of random variables for the completeness. And the probability mass that is 1 over theta and x ranges from 1 to up to theta, 0 elsewhere where theta is an unknown positive integer. So what we need to show that, show that the largest observation say yn of a sample is complete sufficient statistic for theta. So for this discrete type of distribution we have to show that it is sufficient as well as it is a complete statistic for theta. So let us move further to solve this. So first we have to find the distribution of the largest observation for the discrete random variable. So in case of discrete random variable the distribution of the largest observation say y that is y of n is for the discrete type of distribution that is let us say y is equal to y and that can be written as probability of y less than or equal to y minus probability of y less than or equal to y minus 1. So that is the way how we can find the distribution of the largest observation for the discrete random variable. Let us say for the n observations that can be written as y less than or equal to y raise to power n minus probability y less than or equal to y minus 1 raise to power n. Now the distribution function of this that is y over theta that is very simple. So we just put the values of this one that is y over theta raise to power n minus y minus 1 over theta raise to power n. So we can write it as that is y raise to power n minus y minus 1 raise to power n over theta raise to power n. So the giant distribution of the sample probability mass function is that is probability of x is equal to x and that is equal to 1 over theta raise to power n for x 1, x 2 up to x n. Now by using the factorization theorem that the probability x is equal to x our probability of y is equal to y if it is equal to h of x independent of theta we say that we say that y is sufficient statistic for theta. So probability x is equal to x our probability of y is equal to y that is equal to that is actually 1 over theta raise to power n and that is actually y raise to power n minus y minus 1 raise to power n over theta raise to power n. So when we simplify it we get 1 over y raise to power n minus y minus 1 raise to power n. So this is free from theta so it means that it is free from theta so it means that y is equal to y n is sufficient statistic theta. Now to show that y is a complete so we have to find the expected value of g of y that is equal to 0 for all theta and this implies that g is equal to 0 almost purely. Now the expected value of g of y that is actually sum y is from 1 to theta that is g of y into its pdf that is y raise to power n minus y minus 1 raise to power n over theta raise to power n that is equal to 0 and this implies that actually if we put the value of g of 1 we get 1 if we put g is equal to 2 we get 2 raise to power n minus 1 and so on up to so on it will give us g of y that is equal to 0 because this is the first one and the second minus 1 will be cancelled out similarly when we do it for 3 the next factor 2 raise to power n will be 2 raise to power minus 2 raise to power n and in the end we will get net result that is g of y will be equal to 0 for all theta that is equal to 1, 2 up to so on so this reflects that y is complete statistic and that is what actually we want to prove.


In a previous video I showed you how to reinstall the stock firmware on a phone, in this case in particular a Motorola G-Stylus from 2020.  Ideally you can get a custom ROM like Lineage OS or OmniOS or there's many different custom ROMs out there for Android devices. Thing is sometimes it's difficult. You would think Motorola they let you unlock the boot loader, it should be pretty simple to find a custom ROM out there. I've looked so far I haven't seen a custom ROM for this Sophia P Motorola G-Stylus phone. Maybe some unofficial one out there I haven't come across it yet. It would be great if I would just compile my own custom ROM for it which is supposedly a doable thing but when I've read through the instructions of it it's just you gotta download so many files and it takes so long to do I just haven't gotten into it. But next best option, not a perfect option is you can clean up the stock ROM. You can disable at least some of the pre-installed apps. A lot of the pre-installed apps are on read only partitions so unless you modify those partitions which is an option you can't really remove them. But I'm going to show you a simple easy way to at least disable them. You don't need to root the device, you don't even need to unlock the boot loader. All you need to do is enable ADB connected to your computer and I'm going to show you the commands to run and in particular I've made a list of commands for this Motorola G-Stylus that uninstall disables a lot of the default applications and cleans up the phone quite a bit. In fact, I'm going to try to remove as much as I can and even the default home screen and keyboard which means we're going to have to install a new launcher and keyboard. I will keep the Motorola launcher installed and I'll explain why later. Again, this is not a perfect solution. Some of these APKs you're going to disable, other packages might re-enable them. And as I said, even though they're disabled or not installed for this user anymore technically they're still in the system. It's not the best option but if you're stuck with the default firm or the stock firmware on a device, pretty much any Android device, you can do this to most of the pre-installed apps. There's going to be some things that if you disable, the phone won't boot, in which case you just do another factory reset. Not a big deal, that's why you do this at the beginning once you have the stock firmware on there. But let's go ahead and I'm just going to show you the basic concept and then I'll show you the pay spin where I have a list of files for this particular phone where I remove most of the basically Google and Motorola applications that are pre-installed. There's still going to be some left but we'll talk about that at the end. So here are my notes, I called these notes Moto G Stylus Minifier. And again, if you look through here, I'm going to go over this but you can see that I'm basically just removing a whole bunch of things that have the name Google in it and Motorola. I did write a script once that went through and just removed anything that said Google or Motorola in the name and it removed them all and then when I rebooted the device it wouldn't boot. Again, just did a factory reset and reset all that because you're actually not uninstalling these things. Again, most of these things are on a read-only partition and so what we're actually doing is just disabling them. And I'll explain this a little bit more but let's go ahead and quickly look at the device here.  It is, this is the stock install on this Moto G and I have done nothing other than go through the basic startup screen where I just kind of click skip everything and then I enabled ADB debugging mode so I can hook to my computer here. But if we go into settings and we click on apps, you can see here that there's 41 apps installed which isn't too bad. I've seen some stock devices that have over 70 installed and also you'll notice in here there are things like that you don't see in your app drawer like Android Auto. If I go to my app drawer here, Android Auto doesn't show up in here. It's just running in the background but we're going to remove that as well. So let's go ahead. I'm going to maximize this and go back to my notes here. Again, since we're disabling everything, I'm going to install all the default keyboards, all the Google keyboards and I also want to put in my own launcher. So I'm going to download a simple keyboard and install it and then for a launcher, I like the lawn chair launcher. Now I'm going to be pulling it down from the asteroid repositories which is actually an older version of it. The newer versions aren't pushed to asteroid. I don't know if that's a licensing thing. The newer version does have some newer features but this is the easiest for me just to pull down with Wget. So that's what's in my notes but you can go to their website lawn chair and get newer versions. In fact, I say all this in the notes up here. So real quick, I am going to download this APK just to this empty directory I'm in. And then again, I have ADB installed and if I type in ADB install, I give it that APK name and it'll install it and something else I'm going to do that isn't in my notes. So that line we just did is in my notes. I actually like adding in dash G which says grants all permissions. Because as I mentioned in previous videos, I think the whole idea, the whole concept of app, apps having their own permissions rather than users is just a flawed idea. Doesn't make any sense and drives me crazy. So any app I install as an app, I have looked at and trusted. If I don't trust it, I'm not going to install it. And if I trust it, I'm going to grant it all the permissions. That's my own view. You don't have to put that G in there but I'm going to install it and grant it. Next, I'm going to do the same thing for the lawn chair. So lawn chair launcher, I'm going to download it and I'm going to install it with ADB install dash G and the name of the APK. So give it a second download. Sometimes the FDroid website runs a little slow but it should download here in a second. Almost there, connected. There we go. And usually it works a little bit faster but sometimes their website does run a little slow if you've ever searched items on it, you know that. But I'm going to do ADB install and again, say dash G because your launcher does need some permissions for widgets and drawing over applications and stuff like that. But I'm going to install that APK. She'll only take a second. And again, now if I turn on the screen on my phone here again, I can click my button here. It's going to ask, what do I want to use as my default launcher? I'm going to choose a lawn chair and this is lawn chair right here. It's a very nice setup. But now we just want to remove all these apps. And like I said, we're going to pretty much remove everything that you'll see here and you're going to actually watch them disappear as I install them. Now again, we're really just kind of labeling them but let me explain what I have going on here. I've gone through and I've selected these for this particular device. And what we're going to do first is we're going to do ADB, Sheldon PM for package manager. We're going to try to uninstall, for example here, Chrome, okay? Probably 99% of these, that's not going to work but that's the best option because that will actually uninstall it. But again, most of these are on a read-only partition. So what I have after that is I have the two pipe symbols which means if the previous failed command failed then run this next command. And what this is doing is ADB, Sheldon package manager uninstall for the default user. So user zero. So it's still the APK is going to be there but it's going to be disabled for the default user. So again, not ideal but this will clean up your stock image quite a bit. There's also a disable option but for most of these apps that's not going to work either so I just left that out. So now all I have to do is I'm going to copy these lines and again, there's going to be other APKs on this device that I didn't disable but this is the majority of them. So let me go ahead and turn on my screen so we can see this in action. Open up my app drawer. I'm just going to paste all that code and hit enter and we will see one by one as these icons all disappear. And again, there were 41 apps installed by default. I installed two, our new launcher and the simple keyboard because you kind of need a keyboard to do things. You can see, look, they're all just disappearing. It's beautiful, isn't it? And the only thing that's going to be left here is the simple keyboard and settings. So now if we go into settings and we click on apps you can see there's only eight installed. Let's see what they are. So this CQA test, I'm pretty sure is part of Android and is required for making phone calls. Same with this SIM toolkit. Although I'm not going to be using this as a phone right now. If you are, you're going to want to leave those in there. I believe this ISM is also important for that and then wireless emergency. So you could disable these and then the styles and wallpaper probably could have been removed. So you might also say, Chris, why did you leave the Motorola app launcher in there? I'll explain that in a moment.  I'm just going to say always for launcher because that's what I want to be my launcher. And you can see we have nothing, like not even the dialer and you can install your own dialer. You can install FDroid at this point or just download the packages and install it.  Let me go ahead and double click on this and it's just saying yes, I want this keyboard to be activated because you kind of need a keyboard if you're going to do anything on a device. And now I have the simple keyboard and I have my launcher, but my device is very, very clean. It also thinks that it's October 9th, 2021, which it's not actually January, but I haven't even hooked this device up to Wi-Fi yet. So that's the default date that it gives. I flash this one a couple of times. If I don't connect to the internet, it starts off on October of 2021. Okay, so yeah, we have here, this is just saying I'm hooked to my computer through ADB, but we have a fairly clean system. So why? Why did I leave the Motorola launcher in there? So again, if I go to settings here, I go to apps and we can see all and we have this one. I can come in here and I could disable this. In fact, let me go ahead and do that. I actually have a little script I think I may have talked about in previous videos, but if I type in Android uninstall, it's actually gonna connect through ADB, give me a list of packages. And I'm gonna type in moto and I'm just gonna type in launcher. So yeah, moto launcher three, I think this will remove that if I hit that. It basically does the same thing we just did. So I just removed that launcher. I'm not using it, right? So what does it matter? Well, this is why it matters. So let me open up like this and let me open up this. So I have two windows open. Watch when I click the square here. I also disable gestures because I hate it, but it doesn't matter whether I'm using gestures or this. If I click this square down here or do whatever gesture I do to display the apps that are open or previous history of apps, it doesn't work unless on this particular device, unless I leave the moto launcher installed, which is a little annoying. So I'll show you, let me go ahead and look at my notes here real quick.  This is kind of like a step by step on how to do what we just did for different packages. You can see if a package is already installed, we can run this command. So we'll do that. We'll come back here. And so let's make this full screen so you can read it. ADB shell, CMD package, install-existing. So if it exists, which this package is, you need the package name, I'm gonna click this. And now let's go back and we'll click this. So that's reinstalled now. So it's installed for user one because again, never really was removed.  Let's turn that back on. Now if I click the square down here, now I get my app switcher. So for this functionality work on this particular device, I need that. If anybody knows an app that will allow me to do that, it's free and open source that I can replace this launcher with, please let me know. But I've tried searching open source app switcher. I don't even know what you would call this. Yeah. And I'm sure there is something because obviously, maybe if I just look at like a lineage OS, I can see what they're using. You would think that'd be part of your launcher. So launcher in my case, but it's not. But yeah, so look, I have launcher, which is my launcher, which I don't know why the icon took so long to show up here, but I have that. Settings and my keyboard and everything else within reason is uninstalled. Again, there's still gonna be a few Motorola and Google stuff installed in the background, which is not ideal. But if you are stuck with the stock ROM, you can go through these steps to clean up device quite a bit, which is really nice, especially if I'll lock down device like an Amazon Fire tablet, for example. It best if you don't buy a device like that. But if you get it, because they're such good deals, you can at least do this. And again, in my case here, I really like Motorola devices because you can unlock them.  But if someone hasn't created custom ROM for it, you have to do it, which seems like a long tedious process that I just haven't gotten into myself. But if anyone knows a simple way to do it, please let me know. And now I'm just gonna talk for a little bit. The tutorial's pretty much over, but I just wanna say, I really wish Android was more like just desktop system where you have one image that can work on multiple pieces of hardware. I know that's not how it is. I wish it was. It seems to me like Android is Android. And the biggest issue is basically some of the drivers on the driver partition. And I think basically when you make a custom ROM, like you take Linny and Joes and you take it for a device, you basically compile it the same way and then you pull the drivers over and put them into a certain folder or partition for different devices. I wish I could just extract the image and copy over it and not have to compile stuff myself. But all the directions I read, you have to download. I wanna say it's like 50 gigs worth of source code. And then it takes, depending on your hardware, a day or two to compile. And I just, I don't have the patience for that sort of thing. I really wish that I did. I've started the process a few times and like 45 minutes and I'm like, I'm done. I'm not even a fraction of the way done. Anyway, so thanks for watching. I hope you find this useful. Again, I'll link to this in the description. So the code we went over was my called MotoGStyle Minifier. It's called that just because specifically a lot of these are Motorola apps that are on this device but a lot of them are probably on other Motorola devices, a lot of these Google ones. So this, you copy and paste this on pretty much any device and it'll probably remove a lot of junk even if it's not a Motorola device. It will just give you an error on the ones that aren't installed. If you did go to filmsbychris.com which is my website, there's a support section where you can go and you can donate to me which is awesome if you can do that. But I wanted to show you in here there's a few different options like GitHub, my scripts, my notes. If you click on notes, it's gonna bring you here which actually searches through all my PaySpin posts. And so if I was to type in Android APK, you can see right there MotoGStyle Minifier. So you can search through my code here, you click on that and it'll bring you to PaySpin. So check that out. I do thank you for watching. And as always, I hope that you have a great day.