Hello, welcome to SSUnitech. So, still decide and this is continuation of SSIS tutorial. So, today we gonna discuss about term extraction transformation. So, what is term extraction transformation? So, it is used to extract noun or noun phrases in transformation input columns and provide occurrence in output. Next is, we can use term extraction transformation to discover the content of a data set. For example, we have a text file that contains the information of email messages that may provide useful information about product or depend on your business type. So, here we need to take care one thing. It can only work with either dt underscore wstr or dt underscore ntext data type and it will accept these data type in input and provide in output as well. If we have any column with text data and that column doesn't contain dt underscore wstr or dt underscore ntext data types and we want to use this transformation. So, data conversion transformation will help us. So, let's move to our scenario. So, this is our scenario. As we can see, we have a database for training history and we have a table that contains the information of emails and we want to analyze demanding courses. So, here as we can see, we have a OLDB source because we have a SQL server table that contains the information of emails. That table contains wire cat data type. That's why we have to use data conversion to convert wire cat data type to n wire cat data type. Next is we have to use term extraction to count the occurrence and then we want to load in our OLDB destination. So, let's have a look of SQL server table. Go to on SSMS. As we can see, we have term extraction table. Now, we can execute and we can see we have total 14 rows and we can check the data type for this. So, we can find it has wire cat data type instead of n wire cat. So, this is wire cat. That's why we need to use data conversion to convert this wire cat data type to n wire cat data type. So, go to on ssdt to implement the package. So, here I want to add one more package. Then we need to rename this package with term extraction. So, this is our term extraction. Now, we have to use data flow task. So, we can drag and drop this data flow task. After that, we need to double click on this data flow task. So, here we need to configure source, transformation and destination. So, our source is SQL server table. So, we have to use OLDB source. Now, double click on this OLDB source. Then we need to select the connection manager. So, click on new. As we have already made the connection. So, I want to use that one. Click on OK. Then we need to select the table. We have term extraction table. So, we can select and go to on columns. So, we have mail description. Now, click on OK. So, we have done our source. So, it would return wire cat data type, but we want unicode character. So, we have to use data conversion. So, we can drag and drop data conversion and connect with source. Then double click on this data conversion. Then we can select. So, as we can see, it's going to return dt underscore text. So, we have to convert this dt underscore text to dt underscore n text. Now, click on OK. So, we have done our data conversion. Now, we need to use term extraction. This is our term extraction. So, we can drag and drop. Then connect with data conversion. Now, double click on this term extraction. So, here we can see we have three tabs. First term extraction, second exclusion, third advance. So, here we can select copy of mail description. Then, then we can see output column names. So, this is term and then score. So, term will return the courses. So, we can rename this with courses and this is a score. So, we can rename with total count. So, it will return the count of the courses. Then move to advance. Exclusion tab will be covered later. Now, we have term type. So, in term type, we have three options. First is noun, second noun phrases and third noun and noun phrases. So, we gonna use noun and noun phrases. Second is score type. So, in score type, as we can see we have frequency and tfidf. So, tfidf is a predefined formula. So, I am not going to discuss about this. I am only discuss about the frequency. So, I am going to select this radio button. So, here we can see parameters. So, what would be the parameter? First is frequency threshold. So, in frequency threshold, we can define the value. What would be the minimum occurrence? So, I am going to define two. So, if any noun or noun phrases will occur minimum two times, then it will return in our output. Second is maximum length of term. So, I am going to set with 20. Then, we can see we have an options. So, in this option, use case sensitive term extraction. So, I am not going to select this checkbox. Once we select this checkbox, then it will treat as a case sensitive. Now, click on OK. So, here we can see we are getting a warning. Rows send to error output if rows will be lost. So, I am going to ignore this failure. We will click on this configure error out. So, here we can see we have errors. So, we can ignore failure. Then again ignore failure. You don't need to worry about it. I will cover this part once we will understand about the error handling. So, now we want to load this data in our SQL Server table. So, we have to use OLEDB destination. Then, we need to connect with term extraction. Now, double click on this OLEDB destination. So, here we have already made the connection. So, we need to select the table where we want to load the data. So, we want to load the data in term extraction destination table. Now, go to on mapping. So, as you can see total count has been mapped successfully because input column name and destination column names are same. Courses we have to map manually. Now, click on OK. Before execute this package, let's have a look of table. As we can see we don't have any record right now. So, now we need to execute this package. Click on start. As we can see five rows should be inserted in our destination. So, we can stop this package and then go to on SSMS. Then we need to execute this selected statement. As we can see we are getting the courses and number of occurrence. SSIS four times, SSRS three times, .NET two times, Power BI two times and SSIS again two times. Next is exclusion. So, here we have an option to exclude nouns and noun phrases. Those are not required. So, we have a table and that is term extraction exclude table. So, we can execute. So, here we can specify the noun and noun phrases. So, I have specified .NET. I want to truncate this table. Truncate table and then table name. We don't have any data in this table. Now, go to on shtt. Then double click on this term extraction. So, go to on exclusion tab. So, here we can see use exclusion terms. So, we can select this checkbox. Then we can see it would ask for OLEDB connection manager. So, we cannot specify nouns and noun phrases except OLEDB source. So, we have already made the connection. So, go to on tables. What is the table? So, we have term extraction exclude table. Then we have to specify the column. So, we have only one column that is exclude. So, now click on OK. Now we need to execute our package. So, once we execute our package then .NET will be ignored. As we can see we are getting only 4 rows. Last time when we run this package then that was 5. So, we can stop this package, go to on ssms and execute our selected statement. So, as we can see we are getting 4 rows and .NET is not all level here. So, thank you so much for watching this video. If you like this video, please subscribe our channel to get many more videos. And don't forget to press the bell icon to get the notification of our newly uploaded videos. Thank you so much.


There you guys. This is one little patch of fat that's forward. I'll put red ones on the front so I can see. I was going to change the blade colours, but I just thought it'd be easier because it's already got its props on there. Just leave them the same. But I've got a problem straight away. So the battery says it's charged. I put it in. Switch it on. Switch on the controller. Okay, now I'll turn both of these things to the bottom right. Just hold it there. Okay, but this is the problem. See if you can spot it. See the problem? I see the problem. Tell me. I was really hoping I was going to get some flight time straight away. But it looks like I've got two choices here. I've got one. I can either send it back and go for the hassle of having to wait in again for another one to turn up. And from to say it's going to arrive tomorrow, but it will turn up yesterday when I'm not up because I'm not expecting it. And then today I'll get up extra early to be here and it doesn't get here until either now three o'clock. Or I can take it apart and void any warranty if there's any such thing with this and fix it, locate the problem. I imagine the controller board would be fine. The other ones are working and there's no indication of an error. It's probably just that this isn't connected correctly or could possibly be a faulty motor. All together, so I don't know. I took the camera off straight away because I don't really need it on. Just to be flying around inside here. But yeah, we got one propeller that was in turn. That doesn't make for very easy flying when it comes to these. There you go. This is for the sake of myself and giving a video to Amazon. I'm blunted. I'm sorry I keep touching. I don't know about it. It really annoys me. I'm going to come very self-aware about to stop that. It really annoys me. I just did it again. Right. I'm going to start slapping myself. Would you call that self-harm? Oh, never mind. Right, make sure. Bye. So as you can probably use, I decided to take it apart myself because I'm probably going to make myself quite familiar with these things. I might as well start as soon as I get out of the box. So I found the problem anyway. I'm going to switch the light back on. There's the wire here. In order for this to work, I'm pretty sure it wants to be soldered to here. I'm going to take a closer look. I'm going to try and see this. This is going to be very difficult now because one, this light isn't very good. But basically, this is the wire. This is the wire here. That should be soldered to there. And then I've got a funny thing. We're going to be all good. I just have to put it back together again, set it back up. And hopefully remember, yes, there are the red ones. They're on the red side for the propellers and they're on the blue side. That's all I'm going to remember. Put them back in the right way and we're good to go. So I'm going to get soldered now. I'm going to sort that out. And hopefully the next bit of video you're going to see is it taking off. You might not actually get to see that, but I'm sure I'll certainly show you it with all four rotors working. On here.  That's what it does, but you can see. The light isn't terrible in here. I have to work out. I can't remember before with my other light, my fluorescent tube. Was that as flickering? Oh, I tell you what, I've got sent to me the other day through the post. I've got sent a pen. This pen, if you do this. So there we go. Well, anyone who's familiar with these things, you'll know that the motors have to be soldered on there. So I'm going to do that and this thing will be working in, let's say, 20 minutes. Yay! Because all my batteries are charged. One four is charged. Those four are charged. So chances are I'll be going inside this very, very soon anyway to change these motors up. Right, back in a bit. Okay, so it's put bad together in. The screw's on in it, yep. There's a little bag of the screws. I'm just going to test it and see if the darn thing works. So I'm just going to pop you up in the holder for a second. I press the stop button. So let's try not to bash them off. Let's give it some power. The nice thing about this is if you take the camera off, so you've got some extra lightness and flight time across this, it acts as a stop for the battery when you plug it in. Let's just turn it on. Looks all right. Let's move that cup out of the way, just move that out of the way. Well, this is not me having nothing on it. So let's just down. I'm just going to push them both over to the bottom right and to the upstops. Now... Yay! But there's something definitely wrong though, because that was full throttle and we didn't lift off. So what I'm going to do now is just make sure I put these propellers on the way around, because obviously I didn't. Okay, let's put you on pause for a minute. Okay, so all I did was I just put that one there, that one there, that one there, that one there. And I think we're good to go. So I'm going to put the landing doobies back on.  Let's have a little go at this. Let's fly it. Well, it's a little bit rainy outside and a little bit wet. We've used the full battery and I haven't smashed it to pieces, which is good. We've tried to do some hedge fence cutting. Just a couple of tongs, that's what you can see on the edges of the blades there. That's all good. These are the first set of blades. I've got a whole packet here of spares. So we've got plenty.  Given that it's first fix, a strip down and fix. So I've got four more batteries that need to be used. I need to check what it meant by this thing just flashing loads. I presume that was the battery. I'll switch that off. I presume it was the battery I'm just trying to wipe off. Wipe off the rain with my petite shirt as you do. Hey, hey, hey. Brilliant. I'm going to get loads of footage of playing around with this. And it was great putting these on as well. So I know where the front and back is with having to read the top of this. That does make life a lot easier. Because this control here is left and right. I've never done this one before. I keep seeming to bring it down while I try and do it. So I chuck it straight to the floor. I'll get used to it. Brilliant. Brilliant. I'm going to cost me 31 quid. 29 quid or 31 quid.


Hi everybody it's Sam at Mixed Up Growl, thank you for watching my tutorial today. I've got this really nice faceted gift bag that I've made and I love it. So I've just played around with some score lines and made a few extra folds and yeah I've got this. So the top just opens up, I've put a little velcro fastening there.  So actually I could cut that off but anyway so it will be like so and then you just open it up and look you've got a really nice size there. It's the whole depth, really really good size so this will work for any occasion. Yeah really good I put the handle on there as well, again ignore that white bit and I've got all this lovely. You will have a triangle patterned paper in here as well but even on the back there it's just lovely so really really like this one. Very easy to make so let me bring in everything that you're going to need. So I'm doing black this time with this gorgeous paper which is from the same paper pack that I've used and that is from Craft Smith. It is an old pack so apologies. I will see if I can find any sellers. I bought it from a craft store here so it was discontinued and is this one and it is beautiful. My mum will be pleased because I got this for her when she visited so she has the same pack but you've got lots of foiled papers in this one and it is the same pack that I made that very large 12 by 12 fold flat gift bag and it was with this one here which was that big love kind of scrapbook page there. So yeah really really nice pack that one. Sorry to rub it in. Okay so you're going to need one piece of 12 by 12 card stock okay so you're going to score it two and seven eighths of an inch and five and three quarters then rotate the card so those two score lines are at the top and you're going to score at two and seven eighths of an inch just down to the second score line so that's the first one that's the second one then at five and three quarters down to the second score line eight and five eighths of an inch down to the second score line and then eleven and a half all the way down okay now also what you want to do on my scoreboard I've got these little black markers here and this indicates three inches six inches and nine inches so you want to basically put a little marker at the same score lines that we've just done here apart from that one actually because that goes all the way down but these three score lines that we just done we need to put a little marker at the same so at two and seven eighths of an inch I know that that's the little bit here just before this black marker so you just want to put a little kind of like mark in your card then you want to do another one at the five and three quarters so again there's my six inch mark if I just come down to then I know that that's where that one is and then this one is the eight and five eighths of an inch mark and I just need to come down three okay now if you don't have your scoreboard like that or you're not even using the scoreboard just with your ruler just line it up on your scoreboard here at the two and seven eighths of an inch one and then just go all the way down and put a little mark there okay like I said two and seven eighths of an inch five and three quarters and eight and five eighths of an inch okay then get rid of our scoreboard and then with your metal ruler so what you want to do now is rotate your cards you've got these square pieces at the bottom and we're going to create these crosses now that I've done and to do that if you start from the so here we've now got that tab on our left hand side okay and we've got two squares come up to the two squares and the left hand side of the top of that square so the top left hand side okay just here we're going to score from there across to that little mark that you would have made in your card okay so I'm going to go over mine again I'm going to use the smaller end of my stylus and just really embed those score lines okay then you want to go to the top right of that square and go up to the top here where that half inch score line is okay so again like so okay I'll put a little template on my blog as well for this then you're going to go from that same mark that we've just done this score line and we're going to go across to that next little marker on our card now remember every mark is above these three lines here okay if I hold it like that you can actually see what I'm doing okay so like I said we're going from here across the score and then to the top of the next square so you always go the left and then the right so we're going to go from the right across up to that one there so now we've done this cross and this cross then you just want to repeat that with the other two there now you can also just go from the top left of every square and go across and do all of your kind of left to right score lines so that one that one that one and that one and then just move your ruler that way and then go from the top right of every square and again go across so it's entirely up to you how you want to do it but basically you want to have one two three four five six seven eight squares at the bottom and then one two three four big crosses and then your half inch score line okay you can really see the lines there on that black card so hopefully that has all made sense for you okay so now what you need I'm going to bring back in my scoreboard to go through the measurements of all the pattern paper so you need let's grab these so these are the two and five yeah so these are two and five eighths just check they are the right size yeah they are so two and five eighths of an inch squared you want four pieces and these are going to be to decorate the four sides of the bottom square okay then you want four pieces of two and a half by two and five eighths slightly just not squared okay two and a half by two and five eighths and you want four pieces and these are going to make actually you need eight pieces because they are going to make our triangles here and I've just realized I've done mine all the same when actually I want different here so I'm going to cut some different card in a minute for that one so yes basically you want eight pieces like so and then if you want to do these different like I have so quite like that contrast although I've already cut these and I'm going to cut them into my other squares but you'll need four pieces of two and a half by six okay and that's going to make our big diamonds now if you have ever have paper packs and you don't have matching embellishments and things like that to go with them you can just fussy cut so all I've done here is fussy cut some of the paper to give me these and that's what I'm going to use to make this nice little piece on the top so that's all I did there is I fussy cut them from this paper okay so it's just a nice way there to to make embellishments decorations and stuff then for the handle you need one piece of eight and a half by half an inch okay to be honest if you want to make it longer you can you might not even want to put a handle on it's entirely up to you but that's what you need there then what we're going to do so our pieces that are square so our two and five eighths square they just stay as they are these slightly off-squared shaped ones so the two and a half by two and five eighths what you want to do is along the move my ruler out of the way um scoreboard sorry so let me just grab my ruler here so along the two and a half inch side we need to on one side so I'm going to just flip it over grab my pencil so along the two and a half inch side you just want to mark at one and a quarter okay like so then I'm going to bring in my trimmer and what we're going to do is we're going to cut from that middle little pencil mark down to each corner to create a triangle so I'm just lining it up in my cutter here my guillotine and just cut it like so and then I can go in the other side there making sure my points are all lined up and then I've got one of my triangles just get rid of that now you can go and cut each one like that if you want but what I find is is quicker really is line up along the two and a half inch side and you can kind of choose the way you want your pattern here so yeah line up the bottom two and a half with the two and a half because you want the height of the two and the five eighths and then just trace it with your scissors so just cut right along that line it's just a bit quicker than having to get the pencil and mark each one as long as you're lining it up along the two and a half inch bottom bit here you'll get your little triangles much much quicker there I need to cut some more of these because you will have eight of these remember like so okay so you will have eight I'm going to cut some more of them in a minute okay so I've just gone ahead and finished them so I've now got my eight triangles there I've got those four squares and then I've just gone and cut these again in a contrasting so I've actually done the same from the scrap that I had there because I do like that polka dot so what you want to do with these is along the bottom shortest side well actually you want to do it on all four sides but you just need to mark the halfway mark again you only need to do this on one so you've got two and a half you need to do one and a quarter and one and a quarter do this on the other side not on the printed side but it's okay I can rub it out and then along the six inch side you're just going to mark at three inches so there and okay and then again with my cutter which I need to clean that's some reason I've got a load of glue underneath it I need to take that off and then again you're just going to cut from point to point so I'm just lining up the pencil there and the middle there all on the edge of my blade all you can draw a line with your pencil and join them up and then just cut it freehand it's entirely up to you like I said I only do this once and then I use the one I've cut to just trace like I did before I just do find that's a bit quicker plus you tend to get it a bit more consistent as well so there you go so I've got one of my diamonds there so then I will just grab the other one sit it over the top get it bang on like so and a little bit over there there we go just make sure you hold it really tight between your finger and your thumb and just move it around there you see what I'm doing so just go over all of those four pieces or three now that you'll have left and just get those diamonds cut out okay so that's those four now all ready to go so you should have four big diamonds you should have eight triangles and you should have four squares okay then bring in your main card again and now what we want to do is burnish our score line so first of all I'm just going to burnish the main ones there and then that half inch one and then you can just pinch those ones in up to that score line like so again this is the do crafts colossal paper pack that I like using so this is a really nice card then for these ones if you just flip it over and basically just go along I like to do all the same first and then rotate it and just lift up the card each time so it's joining the tops of these squares nicely that one can go right over like so okay so now we've got them all burnish you can really see it taking shape so now what we want to do is stick all of these down so I find it easier to do when it's all nice and flat but there's a couple that you don't want to stick down until you have put it all together and that's the the very so basically you will have one here one here one here this one is going to be obviously half on this side and half on this piece so that one you will stick down last and that's it everything else actually you can stick down so then these are all going to go like so and then you should have a nice border made sure it's about a nice little like just under a quarter of an inch it's going to go like so these ones will go like that and then our base is these ones here we'll cut it all in a minute as well so make sure you keep it right in those squares and then the handle of it going last so actually the only one you don't need to stick down is that one so I'm going to get those all stuck down I'm going to use my wet glue but you can obviously use your tear tape just make sure you're not using a heavy water based glue because you don't want it to warp especially with a faceted style like this it would just end up looking a bit a bit odd so I'm just using my wet glue okay so they're all stuck down and as I was doing it I was thinking do you know this is too long it's it's going too narrow at the top and then I measured this and it's not six at all it's actually five and well yeah five and a quarter yeah five and a quarter so I will edit that into this video you don't want it to be two and a half by six you want it to be two and a half by five and a quarter yeah definitely okay um and you'll get that see it's like all the same at the top there got that nice purple whereas you look at my black it's gone a bit me being picky but I would go with the five and a quarter okay so anyway that's all stuck down now so if you just lift that up you can see and it's really going to come together now so what we want to do is just a few cut lines so on the bottom four here you want to cut up this tab and just take a little wedge out there and then do a little wedge on the top there okay and then go up each of these each time removing the score line either side it will just make the bottom of your gift box or gift bag much much neater okay so now we've done that you should have those four cut like so then along this side I'm going to add some of my red tape I'm not going to use my wet glue I'm going to use this instead I mean just want to run some glue all your tape along here make sure you hug the score line side more than anything you see that my tape doesn't completely cover it all but it doesn't matter as long as it's more on the score line side it's fine and then I'm just going to peel the backing off and flip it over now this one you can't just fold in half because obviously the design so you need to hold it and what I would say is with the squares at the bottom here and the score lines line that up first to get those squares nice and married together with the base like so I'm just going to squeeze them down so you can see there I've got that all nice and even and then this one here you can then just make sure the top is all nice and lined up and then the rest should just all meet together like so okay and then that last one is now going to stick in like that so I'm just going to get that one stuck in okay so that's now stuck in and it also covers up that score line which is really good okay so it almost looks like when people look at it they'll be like how have you put that together because I can't actually see any joins so now you've got the base so because I know that that is where it was joined that's going to be the back so this is my front so this will be the last one to go down so I'm just going to pop my back one up first I'm just going to run some glue around there then I'm going to pop one of the sides down and again other side you can see I've got nothing overhanging because we took off all that bulk and then the last one so this has got three layers to the base it's really reinforced it and made it very strong so again you could definitely put something heavier in this a miniature wine bottle would probably fit certainly heavy candles all kinds of heavy things and then that last one just perfectly sits over the bottom there you can see flip it over I'm just going to pop my ruler in there just to make sure it's all nicely secure and then what we want to do okay so when you if you fold it out like this okay and basically you want to pinch it in half on the sides just so it lies nice and flat just pinch it okay like so then open it up again and now those ones that you pinch just fold back in again and just line up your score lines like so down again just push that one back in okay so now we've also got nice dsp inside there which I think is a nice touch and yeah there's the closure so it's really easy that's the easiest way to do it you can add more score lines at the beginning if you want but you don't really need to it's very quick to just fold it flat now one thing I didn't mention is you also need a piece of two by one and you want to score along the two inch side at one inch I'm just going to fold this over and this is going to be my little fastening so if you don't want to handle I mean I'm having both but you could just have that on like so so what I'm going to do is stick it down first and then I'm going to build up my flowers on tops just so I can kind of see how they look so I put glue on one half put the piece that's got no glue on on the front is that the front is that the back I can't actually remember now no that's the that's the front for me so that way yeah and just pop it on the back there so the score line meets the top of the card make sure it's all in the middle squeeze it together and fold that over like so and just make sure that that is sitting in the center okay and then on that front piece I've got some of my belt pro here so I'm just going to take a pair so many I should organize these a bit better but so that on the inside and again fold that over like so okay and then with my flowers and I'll read the handle in a minute but basically I'm going to stick the large one right over the top I think and then have these ones kind of stuck yeah so let's have I'm using some foam adhesive here as well just to kind of lift them a little bit like so and then these kind of I'm going to have these leaves just kind of nestled underneath okay so you've got an instant embellishment there which matches perfectly and then that's going to stick but before I stick that down actually I'm going to put my handle on so this is the handle put your finger and thumb on each end so your thumbs are facing up and just bring it round towards you keeping your thumbs facing up the right way okay and then that's how you put a handle on so it's up to you which way you want it to be so let's do so something I'm going to flip it actually because it was that way yeah so it would be like that okay so I'm going to hold it together like so and just put a little bit of glue on each side and just stick that there for a second okay and there's the back it's much neater now because I haven't got that white piece there from what I've done before so that's how the back will look that's the front absolutely love this and then I'm just going to add some glue on my square and then just position my flower and there's the box so if I just lift that up now you can see you've got a really nice open up huge huge book set fill with treats and I just really really like this style I think it's lovely and then that folds down and you can hold it I'll see what the handle the handle is more decorative more than anything so there's something very heavy in there I mean it shouldn't come off at all but yeah I wouldn't put too much pressure on the handle but there you have it lovely lovely so there's the two so I hope you've enjoyed this faceted handbag tutorial from me today as always please give me a thumbs up and subscribe to my channel to see more thanks for watching bye


Hey folks, I'm in the midst of a series of episodes looking at different ways of working with and visualizing data related to climate change. In the past, we've looked at global climate change data, specifically looking at temperature anomalies. Now we're moving forward to looking at more localized weather data. I'm looking at data from a weather station in Ann Arbor, Michigan, which is just outside of where I live in Dexter, Michigan. In the last episode, I showed you how we can access the data files from NOAA, which is the National Oceanographic and Atmospheric Administration here in the United States, which keeps track of all the weather and climate data here. It also has data for people in other parts of the world. And so if you have your latitude and longitude, you can get the data for your neck of the woods, so to speak. And we developed this R script in the last episode that I'm calling localweather.r. If you want to get a copy of this script, as well as everything else that I have in this project, go down below in the description. There's a link to a blog post that will get you all the stuff you need from GitHub to get going. This is the script we created. If you want to customize it for your area, all you would need to change are these numbers here in latitude and longitude. And again, in the last episode, I showed you how you can very easily get those values from Google Maps. There is a Google Map R package that would allow you to immediately put in your actual longitude latitude, regardless of where you are. I'm not going to do that because it takes some steps that I'm just not willing to do, because I don't care that much, because I can use Google Maps. Anyway, you do you. But I think it'd be great for you if you're able to put in your latitude and longitude, just to personalize this a little bit more. As we saw in the last episode, we then used our local latitude, longitude to find the station that's closest to us that has data starting before 1960 and ending after 2020. We got that station, we then pull down the data from that station by creating the URL with the glue package, reading it in with read CSV to then get the temperature, precipitation and snow data. There's a few other data points that generally come with each of the weather stations, things like T min and T observed. I'm going to work with T max because that's probably the most extreme. So let me go ahead and run all this, we can have that local weather data loaded. So it takes a little bit of time to run because again, it's downloading some pretty big files. But once we have it here on our computer, we can of course look at local weather, and we see that we have a data frame with the date, the T max, the precipitation and the snow. What I'd like to look at in today's episode is whether there's any anomalous data in here, right? So whenever we're doing analysis, it's always good to make sure that data makes sense. And so I want to show you a couple of strategies for diagnosing problems in the data, as well as show you ways with quantitative data, at least that we can go about removing those outliers as we might call them. And so I'm going to show you two ways. So the first is to plot the data as a line plot. And the second is to plot it as a histogram. And so if I take local weather, and pipe that to ggplot, and on the x axis, I'm going to put the date. And on the y, I'm going to put each of these variables. So let's do T max. And then we'll do geom line. And I don't see anything too outside of the norm. However, one thing I noticed is that these temperatures are really high, right? Up to 400 degrees, something must be off. So if I come back to the directory where I got the local weather station data for, you'll recall that at the very bottom of this directory, there was a read me by station, we go ahead and open that up and see if there's any information about the precipitation, the snow or the temperature data in here to tell me what the units are, right? I just assumed that was Celsius and maybe centimeters or millimeters. I don't see anything in here. Actually, what I see here is C section three of the GHCN daily read me text file for an explanation of the element codes and their units. So let's go back to that. So that was the parent directory here. And down here we have this read me file. And again, if we scroll down to it said I think section three, right format of data files. And we can then see that it's got all these different values, everything. And here's the element. So elements is the element type, there are five core elements, as well as a number of additional elements. We're working with PRCP snow and T max. And we can see the units here, right? So precipitation is actually intense of millimeters, not millimeters. Snowfall is in millimeters. And then the maximum temperature is intense of degrees. So where the plot might say, you know, 400 degrees, that's really 40 degrees. So we'll want to go ahead and fix that. What we can do is come back to our local weather. And I'm going to go ahead and add to this mutate, where I will do T max equals T max divided by 10 to get it in degrees Celsius. Also my precipitation is intense of millimeters sold a PRCP equals PRCP divided by 10 to get that into millimeters. I'm also going to go ahead and rename T max PRCP and snow to all be in lowercase. So to do that, we can do rename all. And then what we're going to function will apply to all the columns is to lower. And so again, if I look at local weather, I see I have temperatures that are much more reasonable in all my column headings or lowercase. So I need to turn that to T max. And let's regenerate this line plot, we can see that there's a fair amount of variation. Again, that's these are daily temperatures. And so over the course of a year, you're going to have pretty widespread, right? So in Michigan, it's basically from about what, like 35 down to about minus 15 over the course of the year. Cool. All right. So that's one way of looking at it as to take your data and plot it across some variable, right? Just to see, is there anything that sticks out as being weird? Another way that we could perhaps do it would be to take local weather and pipe that to Gigi plot. And I want to make a histogram. And so here I'll do x equals T max, right? So the variable we're interested in, we'll plot on the x axis. And then using geom histogram, we can then plot the frequency of the number of observations we have in each temperature bin. This gives us a bit of a bimodal distribution, probably for winter months and summer months, it tells us that it's using bins equals 30. So to pick a better value, use bin width, could do bin width equals one to get a one degree bin width. And here you see something that's a little bit funny is that you get kind of this sawtooth shape right around like every five degrees, which makes me wonder if there's something about the temperature sensors that doesn't like five degrees, right? Because there's kind of a dip at zero, five, 10, 15, 20. That's a little bit weird. Anyway, that's why you'd perhaps want to use a larger bin width like two. This will then smooth out the plot a little bit. And if we do three, it smooths it out a little bit more. Of course, you could go up to like five. And then it's just going to look like a blob and not get a lot of good definitions. So let's go ahead with 2.5. And I think that looks okay. All right. So that works well for T max. Let's repeat this for our other variables. So I'll bring that down. And instead of T max, let's go ahead and do PRCP. And here with precipitation, we see that there are two dates that are much higher than all the others. And so I'm going to ask myself a few questions about these data points. And so I might say like, was this monsoon season, right? Did was there some freak storm that came through Michigan that we just don't remember? Well, you know, I could do a Google search for the actual date. I could get that date by doing something like local weather, slice max, n equals five on, let's say what PRCP. So we're kind of looking at the data to see if it makes sense, right? So this was September 5 of 1953. We had 1286 millimeters of rain. So for my American colleagues, that is 1286 divided by 10, to get to centimeters and divide that by 2.54 to get to inches, that is a 50 inch rainfall. That is like biblical proportions. I don't think that's real, right? 500 is going to be on the order of about 20 inches of rain. So that's not that's not logical, right? And so I think these other values of 115 millimeters, again, that'd be like 11 centimeters, maybe like six inches of rain, that seems like a lot also. But again, I'm inclined to trust that as being legit. Let me show you what happens if we use the histogram instead. So I'm going to go ahead and grab that code for the histogram. And again, instead of T max, we'll do PRCP. And I'll remove the bin width because, you know, bins of millimeters will be different than bins of Celsius. And here what you see is that most days have zero. And then it falls off pretty quickly. And there's probably something out here at the right side of the distribution, which we know, of course, right? That's that like, that's at 1286 is way out here. But there's only one. And so it's so small, you really can't see it. Perhaps we could do something like scale y continuous. And then we could do limit equals zero to let's do 50. And so here, then we see that there's something around 500, and something out here, right? And so for the most part, what we see is that there's a, you know, pretty consistent set of observations or precipitation data on the left side of the plot. But these things that like 500 and greater, probably aren't real. So let's do the same type of analysis. But now with the snow, so go ahead and take these three code chunks that I used with precipitation, and instead use snow. So put snow there, snow there, snow there. We'll go ahead and run this line plot. And here we see, there appears to be one outlier here in about like 1930, where again, we had like 1500 millimeters of snow. Again, that's just a gargantuan amount of snow. Even in Michigan, as we can see, that seems weird. So let's look at the five highest days of snowfall. And again, yeah, 1931, March 8, 1500 millimeters of rain, that seems just way high. As I understand, the precipitation column is a, when there's snow is basically like a rain equivalent of the snow where they take the snow they collect, they melt it, and then they see they got 11.4 millimeters of rain. So that's like four centimeters or about inch and a half of rain. I don't think 1500 millimeters of rain of snow will go into that much rain. So I think that's anomalous, right? Everything else here seems to be legit. Again, we could look at the histogram, and we'll see that there's a value way out here. Again, this is using scale y continuous with the limits of zero to 50. And so when you, when you use that limit zero to 50, then if the bar goes beyond 50, then with scale y continuous, it's removing those data, it basically truncates anything that doesn't fit in the plot. Anyway, but what the hell allows us to see is that this one is way out there on the right, and is a bit of an outlier. Again, my strategy for diagnosing an observation as being an anomaly or being an outlier that I don't want to include is to ask like, do the data make sense, right? Is this plausible? If I was collecting this data in my lab, I might ask like, do we have any evidence that the recorder, the data collecting device was on the fritz, right? Did we perhaps change how we were doing things? Historical data like this, we could use Google and go back to 1930 whatever and say like, were there any just like gigantic snowfalls or rainfalls back then? There weren't, right? And so those types of data support me in thinking that we can safely remove those samples, I can take two approaches. So first, I could use a filter to go ahead and remove anything that doesn't behave, or I can convert things to an NA and then drop all those NA values. So let me show you how I would do both. So let me start by showing you how I do the filter based approach. We'll do filter, and that'll do PRCP less than. So if any precipitation value is less than some threshold, then I want to keep it in my analysis. Again, if we go back and look at our plots for precipitation, we would then see that if we picked a value of say 200, then we would safely remove those anomalous values, right? So let's go ahead and do 200. And then if I came back down and ran the geomline on the corrected data, I now see that I no longer have those anomalous values, and that it's safely been cleaned out. Now let's come back and see how we could do it with the approach of converting the values to an NA, and then removing those NA values. So again, I will do a mutate on a snow. And I will then say if else, so if I'll statement, so if I'll snow is less than some value, then I want to replace that with snow, the actual observed value, if it's greater than that value, then I'm going to give NA underscore real. And so that NA underscore real is a NA, but it's a real. So if else requires that both the true and the false values be of the same type. So snow will be a double, and a real will be a double. So what's this threshold going to be? Well, again, if we come back to our plot, we see that 500 is a pretty good threshold. So I'll plug 500 in here, run that. And now if I come back down and run the line plot, I see that I've gotten rid of that anomalous value. So looking back at these two lines, you might say, why would I choose to do one approach over the other? Well, if I do the filter based approach, I'm going to remove a specific day and all that day's data from the data set, right? So I will also remove the temperature data from that data set for that anomalous day. Maybe you want to do that? Maybe you don't. With this mutate based approach, however, you're only changing the snow value for the days that are problematic. So for that reason, I would probably prefer to do the mutate based approach. So let me show you the code for that. I'm going to go ahead and comment this out. And I will add to my mutate to do prcp equals if else prcp less than 200, then return the prcp. Otherwise, well, let me show you what happens with NA if we've just put an NA but not the NA real, this will give you an opportunity to kind of see the error message you'll get. And so the, like I said, the error in mutate problem while computing prcp if else, caused by error and if else, false must be a double vector, not a logical vector. So an NA value, this is in the false spot. So this is the logical question. This is the value of true, this is the value of false. So NA is a logical prcp is a double, right? And so that's again, why we need to use that NA underscore real to return an NA that's a double rather than a logical. Anyway, so if we run that, we don't get any error messages, thankfully. And again, we can go back and we can look at our line plot for the precipitation, again, see no anomalies, do the same thing for the snow. And again, it looks very good. So the data that I have here doesn't lend itself very well to talking about categorical data that might be an outlier. So let me show you how we would do that. If we had an outlier, that was a categorical data point, right? So, for example, if I look at local weather, and let's pick a date, right? So let's pick a date like this 1891 date in October, right? So we could add to this the date column, and then we could do NA if so if the date column is some value, so that's going to be the Y value. So let's do 1891 hyphen 010 or 1009. And then again, local data, we see that we turned that specific value into an NA. And so again, if that date or that label that category was wrong for some reason, we could use the NA if to do that, right? Again, we could do the same type of thing with an if else statement, right? So we could then say date equals if else, date equals equals, let's do 1891 1005. So if it's that date, then we're going to return NA character, otherwise, we'll return the date. And I'm getting an error message. And it says problem while computing date equals if else, date equals that argument true is missing with no default. Ah, I've got a true and a false, but unfortunately, I have a parentheses in the middle, right? So that parentheses is excluding the true and false. So let's try that again. And that actually gives me an error message, because date is a type of date. And this is a character. So I think what we need to do instead is make this as dot date. And so this is a function that will convert a character into a date, of course, it's going to take an NA and make it an NA, there isn't a built in NA underscore date underscore variable. So that works. And again, if we look at local weather, we now see that sure enough, we have that NA value in there. Okay, so again, these are different strategies for identifying outliers data that we want to further investigate. Sometimes those outliers are real. But sometimes like the cases that we had here, I think are junk, right? There's probably something wrong with the sensor. I am doing this kind of empirically looking at the data and saying that looks weird. I suspect that when Noah and NASA when they're looking at all these weather stations around the world over many, many years, they probably have some algorithmic way of identifying a truly anomalous data, right? So like those cases where you have like a 20 inch rainfall in the middle of the desert, right? You know, perhaps what you might do would be to look at like a standard deviation and say like, is this a three standard deviation rain event or a four standard deviation rain event? And if so, then we're going to throw that out. Anyway, I think it probably takes a little bit more deeper thinking than what we're trying to do here where again, I'm just trying to show some principles of how to identify weird data, and then how to remove those data from your data set. One final thing to show you is that if you have NA values in your data set, like we do here, one way to go ahead and get rid of that whole row like you would with filter would be to do drop underscore NA. Then if we look at local weather, we see that those two days, October 5th and October 9th are now gone, that that drop NA removes those rows where we had an NA value. And of course, you could use drop NA in here. And you can give it a variable to remove those days where we had an NA value for a specific value, right? So if we did NA drop NA with PRCP, that's going to remove that row where we had an NA value for precipitation, rather than for the date, right? So now if I look at local weather, I still have those two rows, because it dropped the NA from the data set where NA was in the PRCP column. Okay, so again, a couple different ways of doing this, you do what works well for you. I think it would probably be good to keep notes, keep comments about why you're removing certain data points. Again, just for your own transparency, and reminding yourself, why did I pick 200? Why did I pick 500? Anyway, I would be interested in knowing what kind of anomalies do you see if you're trying this with data from wherever you live? Do you have a 500 millimeter or snowfall? Or do you have no precipitation or tons of precipitation? What are the weird things that you're seeing in your data? Let me know down below in the comments. Ultimately, though, do this with your own data, data that you're generating for your own research, whether that's in a laboratory or out in the field. I would love to know what kind of challenges do you run into when you're recording data on data analysis instruments, and what kind of anomalies do you find? There are other packages out there for kind of doing this type of work, but I ultimately find that this approach of using if else statements filtering is a lot easier than having to try to remember how to use some other package that will ultimately do this for me. You let me know what you think of that philosophy. All right, we'll see you next time for another episode of Code Club.


We're gonna get started But before I get started does anybody here play ultimate Does anybody know what ultimate is This is so sad So I've been playing ultimate for most of my life. It's a disc sport. I will say the terrible word frisbee So I've been playing for most of my life And I've been fascinated watching the sport grow because when I started playing like the typical responses The response I got here. Nobody knows what this sport is But now it's actually like on ESPN 3 and it's getting lots of public attention It's probably one of the fastest growing or potentially the fastest growing sport But just last week the U23 worlds was in London and India competed for the first time ever So Sushmita Azad from Bangalore said that being part of India U23 team means that I get to represent India in a sport That the mic that is the microcosm of the ideal world There are no referees no body contact and no heckling just highly competitive play Which only works because of the integrity of everyone playing and so I think that has a lot of parallels to The communities in free and open source software so I brought A few jQuery disks to give away after my talk You want one? Throw it to you Come see me after the talk So It seems appropriate that I'd be giving a talk about the community here at the first-ever jQuery conference in India because India actually accounts for 13% of sessions across all the jQuery foundation sites which makes it the second most active country following the US And for comparison the third most active country Germany only accounts for about four and a half percent of sessions So broken down by city Bangalore is actually the most active city not just within India But across the entire world and India accounts for six of the top ten cities We do Bangalore, Chennai, Hyderabad, New Delhi, Mumbai and Prunei Also, we have three projects going on right now in the Google summer of code and two of the three students are from India So it's pretty exciting to be part of this conference with such a vibrant community Even if it turns out that all you do is go to Stack Overflow and steal answers So As an open-source project jQuery UI benefits quite a bit from the contributions of the community and Despite the relatively small size of our code base


Item number SCP-1021 Object Class Safe Special Container Procedures No special containment procedures needed at this time. All instances of SCP-1021 are currently in use in various sites and facilities. For a complete list, see Addendum. Description SCP-1021 is a set of seven exit signs designed in art modern dating back to the 1940s. The anomalous effect of SCP-1021 will only manifest if it is placed directly over a door made of solid wood. Type and design may vary, but the door must be solid with no trappings, such as windows. Any person opening a door with SCP-1021 over it will find themselves exiting out of a corresponding exit of the ██████ cinema in ██████ Michigan. Each sign of SCP-1021 belongs to the specific door in the cinema, with the main entrance being the only door without a connection. SCP-1021 was discovered when a self-described architecture buff visited the ███████ cinema before its demolition in order to study its design and found a souvenir. According to the witness, the only thing left were the box of exit signs by the main entrance labeled Dump. Enabred by their old style design, he claims to liberate one for his own use at home, adorning his bathroom door the joke. Upon entering his bathroom, he found himself behind the ██████ cinema exiting from the delivery entrance. The subsequent internet postings were found by Foundation Computer Search algorithms, and an agent was dispatched to investigate. Upon learning the story and viewing its anomalous properties himself, the agent called for backup to the cinema. It was then purchased by the Foundation through the Scenic Cultural Places Association, an organization dedicated to buying and restoring old and unusual buildings. After administering a Class B amnesiac to the witness, all the exit signs were taken and, through study, were found to correspond to a door within the ██████ cinema. The cinema itself has no history of unusual occurrences, though a small security detail ought to remain for one month and secure the building. Due to its safe class nature and usefulness, all instances of SCP-1021 have been set up in various Foundation sites to act as an emergency exit, and the ██████ cinema has been renovated to act as an evacuation command area where Foundation staff may seek refuge. The evacuation points in all sites must be kept in the clear area guarded by level-3 security staff in order to prevent its misuse of the method of smuggling SCPs out of their home sites. Addendum ████, List of SCP-1021 Assigned Locations and Corresponding Door 1021-1, Site-19, Exterior Exit of Theater-2 1021-2, Site-4, Exterior Exit of Theater-3 1021-3, Site-11, Exterior Exit of Theater-1 1021-4, SCP-S Guardian, Delivery Entrance 1021-5, SCP-S Defender, Lobby Emergency Exit 1021-6, Dr. Bright's Closet, Unassigned, Request made through Dr. Corey, Alleyway Employee Entrance 1021-7, Access Denied Access Denied