histogram as you know, it's a little graph and there is You know a how many a's how many b's and how many c's and there's a histogram So there's this many of that and this many of that and these are like buckets These are frequencies and this is how many times it happens So a histogram, but we're going to do this thing where we're going to take count people's names And we're going to kind of count how many that we see But the interesting thing that we're going to solve just like many of the things in the computer Is we can't just sort of look at the data. We've got to look at the data iteratively one piece of data At a time, so I'm going to give you a little problem Okay, I'm going to show you a series of names one at a time And I want you to count for each name make a little bucket and then keep counting how many things for each of the different names Okay, you'll notice that you have to start with one and then you move across So just watch this and tell me How many How many what's the most common name of the set of names? I'm about to show you and uh, how many do we see? So how many what was the most common name and how many times did you see it? That's the question Now here comes the reveal So for humans, it's so much easier for you to just look at this and you think how did my brain look at that? And you're like, okay, what is pretty common? Oh Maybe Maybe Chen is common. Oh Chen Chen Chen No, maybe Jen is common to one two three four. Yeah that anybody else have mark wad's got three C seven and so you'll notice how our minds is come without computers. We just sort of like bounce Branch and bound we have hypotheses and then we decide it's Jen That's it and there's four of them Now How did your brain think about this as we were going through them one at a time? Well, my guess is you if you really had to do this a lot you would make a little picture like this And then what you would do is if you saw a new name You know x y z you'd add it to the list and give it a tick mark of one And then if you saw like c seven again, you'd give that a tick mark And if you saw x y z again, you'd make a tick mark And then you'd make you'd keep adding to these tick marks, right? And that's how you would do it and you wouldn't like many of the things we do in a loop You wouldn't really know what the most common was one until the end And then you'd sort of take a look at these numbers and you say, okay, that's the most most common number And then you'd you'd be done, but you have to watch them one at a time. You can't just bounce around And so That's how we're going to use dictionaries to achieve that Again instinctively as humans we just look at the stuff But if you add a million things you probably want to write a python program and use dictionaries And so this is the idea and there's two basic things that happen One is the first time you see a name you gotta say is this name there already If it's there already you really just want to add one to it, right? That's the adding of a tick and or you want to see for the first time You know blah blah blah blah and give it a one and so you can use the name As the key and then one is the value and then first time you see chen you stick one in there And so at this point inside the dictionary sort of dynamically adding as soon as it sees a new name It adds another slot in here But then if you see the same name again like chen again Then you end up with a one add one to it and so it's two and so at that point chen is two And so you can see how you can both extend the dictionary by encountering a new name or Adding when you see a name that you've already seen before The problem with dictionaries is like everything in python There are rules about what you can and can't do and one of the I think kind of frustrating things about dictionaries Is that you can't just look for a key that doesn't exist? So this is a fresh brand new dictionary We do a constructor there and we print out sub csev and boom it blows up and that's bad But we can solve this by the in operator the in operator we've used in the for loop So you use it in lists we use it in strings So that is a question. It's a saying is csev in ccc. Well, this is this empty one And so it is no it is not csev is not in ccc And so not using this in operator we can avoid the trace back We can say if it's not there put it in if it is there add one to it and that leads us to This bit of code Okay, and that is the kind of code that we're going to build a history. This is going to histogram code, okay And so this is going to have name is our iterator names. Sorry. I made them singular and plural. That's that's nice But so name is going to be csev chen csev gen. Normally we'll be reading this from a file But for now we keep it on keep it easy We're going to go through this and we're going to have counts as our dictionary So that starts out empty and we're going to do a simple if then else every time through the loop If the name we're looking at is not in the dictionary already is the key Then set it to be one If it's not go get the old value Counts sub name and then add one to it and stick it back in So this is This line right here is new adding a new thing and this line right here is adding Some things to existing things and you do this long enough you start with an empty one and you do this long enough At the very end it will print out The histogram that you're looking for histogram you're looking for and so you say oh We've seen csev twice gen once and chen twice and so that's the idea and so this can run a million times if you want now this notion of checking to see if a key exists and doing one thing if it doesn't exist and doing another thing if it does exist is Such a common practice that the dictionary Object has this method called get that'll that collapses these four lines into one line And so the idea is you're going to do one thing if it's in there and you're going to retrieve the current thing Otherwise, you're going to pick a default value in this case. We'll pick one. I mean pick zero This is like the default right meaning what is not there And if you say counts now counts as a dictionary dot get that's like string dot upper That's a method you give it a key and then a default And if the key exists you get back what's in the key if the key doesn't exist you get the default Okay, so and with no trace back this works. So so the best way to think about this is Those four lines are equal to that one line because x is either going to be whatever was in there before if it exists Or it's going to be zero Now the nice thing about zero is the next thing we're going to do is we're going to add one to it So that that's going to get us to one so collapsing that loop that we saw before Collapsing that loop we can we can make it just a one line loop and this will become An idiom this will become something that you will get used to and you will Use over and over and over again and after a while right now you're looking at it. Boy, boy That's a lot of syntax and semicolons and whatever After a while you just type this and not even think about it. It's an idiom It's basically included in this idiom is how to both create new entries and dictionaries and update existing entries By adding them adding one to them so Everything else and this is the same name is going to go through these five values and we're going to say count sub name equals Counts dot get name comma zero plus one And so if for example, this already has a one in it Then this is going to be one plus one becomes two if it's not it's going to be zero plus one equals two And so this is the idea of if new set it to one not zero set it to one Because the first time you see something the count should be one not zero So that's why we make this default now the get can be used for anything It just so happens that zero is a common default because it's really common that we're using this to basically make a histogram Right a little histogram of a b c Right and so we need to make a d in the but then the histogram has to start at one so that's basically the simplified counting with get and You know, there's a lot of things that we're going to do inside of Python that do have to do with frequencies and how many times certain things happened and this pattern Is a really good pattern to absolutely know So now what we're going to do is we're going to Switch from just looping through strings instead loop through files And we're going to it's going to take a little bit of work because we have to open the file And we'll bring a lot of things together at this point So here would be another task and that is here's a bunch of text from the book And uh, you can just split this into words and count and find out what the most common Common word is and how many times it uh How many times it occurs so go ahead and try to do this for a second feel free to pause Actually, don't bother pausing. This is too hard. We should write a program for this It's not it's not easy humans. Don't like this. It makes you concentrate Um, and so here is a counting pattern where we're going to take a line and then later we'll read this in a file and so We're this is just an adaptation improvement of the previous thing. So we're going to start with an empty dictionary We're going to ask for a line of text and read it in and then we're going to use split So remember the list of words. Well, what we're going to get here is a list of words We'll print it out and we'll run this counting. This is the this is a little loop For every word in whatever this was we're going to do this idiom of Add either adding a new entry or adding one to an existing entry and then printing that out So let's take a look at what we get there So if we run this We can give it some text and I've got this this would be all one line And then it splits it into words and you see that these words here are split split split split I mean that's strings and splits remember strings and lists And split and so now the counting is going to go through this list the clown ran after the and it's going to build a histogram the Clown, you know one clown the up up up of these things are going to go up Right, that's this histogram and then when it's all said and done We end up with the histogram and so counts is the dictionary that ends up with a histogram And we can side by inspection see oh the is the most common word and there are seven of those right So if we sort of take a look at this we start out we make a dictionary we read in a line of text the text goes in We um and then we split that and we print the words out. So these are the words Right, then we have a for loop that's going to loop through all those things and then produce a dictionary And when we print the dictionary out, that's what we're going to get and the seven, okay So that's one line of text That's how you walk across the words in a line of text after you split the line into separate words So now we're going to look at ways that you can loop through dictionaries We just produced a loop that can build a dictionary But now we're going to going to look at a dictionary and so we'll start with a very very simple example And then we'll work to a slightly more complex example. So here's a dictionary just the constant chuck is one freds 42 and jans 100 And so we're going to use a definite loop with the four four key and counts now It doesn't have to remain key but key is then is a good name because these are these are keys and values k v k v keys and values I just mentally think of this as keys and values and keys and values So this iteration variable is going to work walk the keys. It's not going to walk the values It's going to walk the keys chuck fred jant not necessarily in that particular order as you see it goes jant chuck fred Because just because I typed it in in this order. It's not like a list. It doesn't stay in that order It might move around a little bit as we add data to it or as we set the data up And so you can in the loop you can get the key and so that's what prints out the chuck jant chuck fred But then you can also get the corresponding count for each one of these by just pulling it out of the Pulling it out of the array I mean pulling it out of the dictionary right and so we can pull out the corresponding value And so we print out jant 100 chuck 1 fred 2 and that runs this loop three times So if you just use the in and you give a dictionary here Remember all the different things we've been able to put there on the end of a for loop And dictionary is another thing we can put on and we get a list of keys Now there's a couple of methods that allow us to get the keys And so we have you know, we can say turn this into a list and we get a list of the keys So this is a dictionary the same dictionary we get a list of the keys You can also get a list of the keys by using the keys method So let's take this dictionary jjj and give me all the keys which gives me a list Which is kind of the same thing and then we can ask for the values And they give me just then the values extracted out of this dictionary. So that's nice Um now the one thing is is that while I said you can't predict the order if if in two statements You ask for the keys and then the values they at least come out in the same order Even though you can't necessarily predict the order that they come out they come out in the same order And then there is a third Thing that we can do and that is list of ask for the items and we can say give me the items and that gives us a list this is our first really kind of composite combined data structure where it is a list a three item list Zero one two and inside that there is what are called two tuples jan maps to 100 chuck maps to one fred maps to 42 Coming up next we're going to have a whole chapter on that And so just take a look at that for the moment and we will come back to that in some detail later This whole items idea that gives us back a list of key value pairs Because it's not just a list of keys or a list of values It's actually a list of key value pairs allows us to write in python a very clever and elegant loop What we can do is actually this items gives us back each Item in the list has a key and a value and we can actually take two iteration variables for a comma bbb This is two iteration variables. And if you're coming from another programming language, this is super cool And it's a python only feature. I've never seen another language that's capable of doing something this simple and that elegantly So what this basically does it says we're going to simultaneously advance these two iteration variables So this is going to be the key in the value the k and the v Key in the value is going to be chuck one then it's then they're both going to advance fred 42 Jan 100 and so that means in this simple loop if we just print them out We're going to get the key value pairs of course in the order And so it's sort of a and bbb simultaneously walk down These key value pairs and so that's really pretty and it makes for a very succinct loop It's the syntax is a little sort of disquieting when you first see it But it's a super elegant thing and you just have to say items if you If you don't say items you just get the keys if you say items you get the key value pairs and You have to have two iteration variables If you don't have two iteration variables and use items it'll complain and say what are you doing? I'm giving you two things and you don't have two variables to receive them So two iteration variables and items are basically related Now we're going to take a look and this is code that I showed you perhaps many weeks ago About I said this is a little story about how to read a file and count all the words in the file And now we're back to it and at this point you should understand every single character of this program Every single concept of the program You should literally stare at this and look at it code it play with it until you absolutely understand it So let's take a look Again, I showed you this weeks ago So we're going to ask for a file name Then we're going to open the file name Then we're going to make an empty dictionary again This is all stuff you've done before and then we're going to have an iteration variable That's going to go through the lines in the file Right, so line is going to go line line line Then we are going to split that line each line into words chop chop chop chop So that's words is the list of the words in one line We're inside of a loop that's going to go through all the lines And then what we're going to do is we're going to write the have the word iteration iterate through each word in the line And then we're going to do is take each word in the line I'm going to do this histogram Right, so we're going this this is going to run not only just for every line But for every word in every line So we have a nested loop for every line then we split it and then we go across the line So it's almost like a typewriter where we go That's what we're doing So it's like the outer loop is going down down down the lines And the inner loop is going across across across the words And eventually we are going to see in this middle in this last line Every single word in the file and we're going to do the counts get word plus one Which is our magic histogram making line that if you don't remember what that is Go back a couple of slides. I just talked about it At this point in the code and it's important to be able to draw these lines at this point in the code You have the histogram and it's in the variable counts Now we want to find the largest one Now we have written list we have written loops that can find the largest in a list But now we want to find the largest value in the key value pairs of a dictionary So we're going we're going to start with the we're going to know what the largest count is And the largest word of the has that count and we're going to set them both to none Because we're going to prime our loop we have to prime our loop and we're going to say to none And so then we're going to write one of these cool things that says for word come account So word and count are going to go through the key value pairs because we've got items here So it's going to go through the key value pairs loop through each key Whatever it was there could be a million words in here We're going to go through every one and what we're going to do is we're going to make sure that key big count Is the current largest count we've seen so far and if it's none Well, then we haven't seen anything or the current the count We just read is greater than the the big count so far We are going to jump in and this is sort of like oh, this is a new new personal best count for this particular data set And so we're going to remember the word in big word and we're going to remember the count in big count So this is just a max loop It's a maximum loop with the extra thing that we're recording in addition to what count is the largest What the word that was associated with that count recording it So again, this is a starting part of the loop We're going to do some work and then when we exit the bottom of this Big word is going to be the word that is the most common and big count is the number of times And so if we run a file we say oh in that file too is the most common word and it's 16 times If we run the clown file Well, the is the most common word in seven And so this now is can and this could have a very large file and give you the most common word and so that is Sort of a really good application of dictionaries. So dictionaries are the most powerful. Well Of the they're the most powerful collection. We've seen so far Um, it is good to see both lists and dictionaries to understand what quest Collections are they are things Inside of python that can handle more than one item inside of it and we'll learn about another collection about tuples in a second Just Understand the get method because that leads to very compact code Um understanding there are various ways to iterate through dictionaries and so we've learned a lot but in the next Section we will learn even more and put these together and do some sorting and do some other stuff And really start to see the real power of dictionaries Uh, this is i'm going to do some coding. It's related to the Dictionaries chapter chapter nine and we're going to do some word counting. That's uh, it's basically right out of the slides for But i'm going to just write the code in front of you rather than uh, have you look at it in the book? So what we're going to do is i've got my text editor up here And uh, let me start by making a new folder new folder For my chapter nine exercise And then i'm going to go and make an untitled file. That was some of the previous one And i'll do what i always do print Hello and save it and save it here into exercise 09 and x09.py So now i have a folder that's in my py4e folder Uh, and that happens to be in my desktop py4e Is my folder on my desktop And now i have all of these subfolders cd ex08 ls is dur on windows ls oops i gotta go up one ex09 ls So i've got that file right there now i'm gonna want to read some files and so i'm gonna bring some files down Uh, a couple of files Um python for everybody code 3 intro dot txt. So i've got this url And i'm going to save it save page as and it's really important that i save it in the same folder as i'm going to write my code Just so that when i open this file, it knows where it's at. So i've saved that one and i'm going to also Take this clown text. I'll use this Uh to make my life simple So i have a real short thing that i can show you how it works And so now if i go back to my terminal I see i've got exercise o9 python intro dot txt and clown dot txt. Okay So Let's go back to my text editor and get started Uh, i will prompt for the file name Uh input Enter file Con space Now i'm going to do something if the length of the f Name that i just read is less than one i'm going to say f name equals clown txt i do this So that i can just hit enter and it defaults to clown dot txt if i want to give it a different name I can so this if i just hit enter at this prompt then this will give me a string that's zero length So if it's less than one i'll just assume that so let me open that handle equals open f name And let's read through it for line in Handle and we'll strip it Line equals line dot r strip to take the white space off the right hand side and then we're going to save print line Again, I i'm not just doing this I really when I write code. I just saved it when I write code I do these kind of stuff all the time just for my own sanity checking And so now i'm going to run python 3 p x09.py Just to test that I want to hit enter now and it's going to assume hopefully clown dot txt if it all goes well and yep it read It read one line. Okay, so that part's working. I'll just leave that print statement in The next thing I want to do is kind of a classic Thing where we're going to go read a bunch of lines and then go horizontally across those lines and words So i'm going to split that wds equals line dot split and print Wds So i'll print that And i'm going to save it and test it. I I really Love to test things over and over. There's the actual line this file clown dot txt only has one line And it breaks it into words and so I have those words. Let's just run it again With um intro dot txt So this will have a lot of lines line line line line lots of lines Every line has it prints out the line and then prints out the words that we split it into Okay, so now I kind of one of the things that I do here Is I want to believe now I sort of can believe everything from here up Like oh, it's going to open the file. It's going to read through the lines I'm going to split them into words and so then I'll just kind of behind it. I'll just say, okay, I'll just I'll just Comment that out now. I need it another for loop for W in Wds now words is a python list It has some number of words in it zero or twelve or whatever was on the line And now i'm going to print out the word Okay And so now it will go through that horizontally. I'll now just do clown dot txt So that you see I i'm not printing the line out That's the words that have been parsed from the split from the line And now we've got this loop now one of the things that's interesting is just to make sure that you're you're going through all the words And I I like a print statement here to know that w is going to successfully take on literally All the words of this file. So if I comment this print statement out and I run it again Clown dot txt that for loop starting from here is every word in that file which happens to only be one line But now if I do the same thing for enter dot txt It's just going to go through the words and in sense by nesting these two loops We're going to hit all the lines and that's a lot of stuff But it hit all of the lines all the words and away we go Okay, so here's where a dictionary comes in I'm going to make a Variable called a di for dictionary and I'm going to say give me a dictionary Now di ct is not something you can choose that's saying make that's that's defining the type of dictionary di is a variable that I chose okay, so The key thing to this dictionary is we're going to make a counter and we're going to use w the word absorb elegant whatever And we're going to use that as the index So the simple thing to do is to say if w is in um di Then we can say w's. I mean The dictionary sub the word which is our key and the key value store of the end of the dictionary Is equal to the value that we had before in that area d sub w plus one And if it's not in there else di sub w Equals one and I'm and I'm I'm going to print Print new So every time we see a new word it's going to say new and I'm going to also then print w and The current value of the counter for w as it's going through Now notice how far in I'm indented this is all part of this inner loop So this is the loop that's going to run every single word Okay, and I'm going to run this first with clown So it runs slowly okay, so it We saw the was new It's and the count is one clown is new Count is one ran as new the count is one After is new the count is one now we saw the again, but now we made the count be two Let's print here. I'll say existing So you kind of see it now in the print. I'm printing this. Let's make it even a little more verbose print w And then I will make it so it prints the it prints the word before and the count after and then whether it's existing or new So we're put a lot of print statements in print statements are cheap Okay, so now we see the word the it's the first time we see it and we set it to one We see the clown the first time we see it. We set it to one. We see ran new one Later on later on we see the it's already in so existing means It was already in the dictionary w as a key was already in the dictionary Okay, and so that's why we added one to it. So the old value was one and then we added d i sub the equals d i sub d i sub the equals d i sub the plus one w is the string the That's what that what that string is Okay, and so So we've made it all the way through and you see is the in this one line occurred ultimately seven times So now I want to print out the contents of this dictionary At the very end of both loops. So I got it de-indent twice And so that will give us the counts And so this is what we get when it's all said and done, you know the happened seven times But it just worked through it was way through Okay So you got that now This is a pretty verbose way of doing this But I did it sort of the slow way to show that there are two situations If it's already there you increment it and if it's not there you set it to one Effectively inserting it right so you insert it and set it to one with this d i sub the equals one Okay, but let's get a little less verbose here get rid of some of these print statements because we kind of covered all that Um Get rid of this line and go back to printing w and di w at the end We'll leave that one in so what I want to do is I want to look at this bit of code right here this if w in di ls The We do this so much with dictionaries that there is an easy mechanism to Do this that combines these four lines into a single kind of contraction And so I'm going to do this I'm going to print Let's put two stars out then the word and di dot yet of the word comma negative 99 Okay, and so this will this this di dot get of the word is the important part The way it is is this is a dictionary Dot get says In its first parameters the key to look up which is word like the or fell or clown or whatever And 99 is the default value that we get if the key doesn't exist So this is an effect an if then else right This little di dot get w 99 negative 99 is you know If it's in there do one thing if it's not in there do something else Okay, so so let me show you how this works and you'll see that the 99 will happen when Okay, so the first time we see the get returns 99 All right, so let's move it over here the first time we see the The is not in the dictionary So this di dot get of the word the in the dictionary Gives us back the negative 99 Okay, and this still is working and so the is one clown is whatever but away we go Okay, let's do it this way. Let me comment this out. Let me comment this one out and run it again So it's a little clearer what's going on Okay So the first time we see the The is not in the dictionary the first time we see clown and we know it's negative 99 negative 99 But here we asked for it and the is one because We've seen it before And so that's just this get mechanism allows us To get the new value or get a value out if the key exists and specify a a um Default if it's not there. So I'm going to go old count Equals ti dot get W comment 90 comma zero So instead of using 99 here. I'm going to just get rid of all this Is what I'm saying is look up in this dictionary get is a function. That's part of all dictionaries Look up using the key w which is the and if I don't get it Give me back zero and so I'm going to say print word comma old comma Old count And now what I can say Whatever the old count is it's either the value that was in there or zero and now I can say new count Equals old count And now let's see new count and I can say Dictionary sub word is equal to new count So instead I'm going to get rid of this if then else then This is basically saying Look up the old count that we have if you don't find one uses zero We'll print that out and then I'm going to say Afterwards I'm print the new count now and so So we'll print the old count Here are some of these blanks print the old count And you can see the old count with the because the doesn't exist was zero the new ones one Clowns old is zero new is one clowns old ran old zero But now we get to the its old count was one and now its new count is two Okay So by using this get and saying if we don't find it we'll assume the count is zero that makes a lot of sense, right? You know If not there the count is zero if the Key is not there the count is zero Okay, so that's what this line does If get the get the value from under the key be uh associated with the key or Give me zero back and then I can take that old number and just add one to it and then stick it back in Now this is ultimately not how we tend to do it. Okay, we tend to Blend this all into one big long statement di sub w equals This part plus one Okay, so that says get the old value From this key or zero and then add one to it because that really combines All of these lines into a single line. Okay So i'm going to delete them now And now we've combined this all into one. What effectively is an idiom? retrieve create update counter All in one line. I'll still I'll still print out in this case. I'll just say di sub w And then we'll see the counter. Okay And so now I'll run this We don't it's it's we have a new but now we see it the second time it's two And so we see car the first time we see that the second time we see car I mean the third time we see car the second time and away we go Okay, and so that's pretty straightforward and so it really kind of typo there so let's just get rid of that And run it with the clown stuff and we get the right data there and Let's run it with intro dot txt And there we go. Okay, and so it's it's tearing out a bunch of words and giving us a dictionary So that was a lot of work to get to this line 16 that has the dictionary in it Now we want to find the The most common word And so we're going to loop through this dictionary and part of it is like once we printed this dictionary out and we verified that it's right Don't worry too much about the code up here, right? As a matter of fact, I can take out some of these print statements And we can kind of trust all this And so now we're going to work on this Okay, now we want to find the most common word now. This is like a maximum loop. So if you recall um We have a whole set of key value pairs communicate goes to two is to two Skills is three. So we have these key value pairs and we're going to loop through and look for the maximum now in a dictionary We can loop through the key value pairs with the following syntax for You know, I would call this these variables k and v for key and value but uh, yeah in The dictionary's name dot items And items is a method inside of all dictionaries that says give me The key value pairs and we need two iteration variables So this is like an assignment statement for k and v k and v take on the successive values for the keys The key and the value, okay? So if I just now print k comma v And I'll take this print statement out And then run the code On oops what I forgot to oh, I fell back into my python two days Any parentheses for my print? So there's clown and it just prints it out and it's kind of the same thing except it's pretty where we're putting each one on a line Okay, so the k the v is the value. So we're looking for the largest value. Oops So the thing is we know that the values are are always uh numbers that are greater than one. So i'm gonna i'm gonna do kind of a A quickie maximum loop largest Equals negative one now in previous times we've seen that this is a bad assumption But because we know these are counters that are always positive It turns out this is not a bad not a bad idea And so I can say if the value is greater than the largest we've seen so far Largest Equals the value Okay, and when that loop is all done we can print the largest Okay, and so this is just a max loop and we're we're using this value. That's the number the values the second thing Oops I can't type python Oh, it's a typo Yeah, I'm not using value. I'm using v So largest equals v. Let's try it again Okay, so we're all done with seven. So these were the things that we were looking for And it was looking for the maximum and it just dutifully found seven was the largest But we also want to know what the word is And so what we can say here is we can say the word Is none Meaning it's it's it's just like we don't know what the word is And then whenever we catch this new largest number We say the word equals w. So we're so I like to think of this as capture Remember The word That was largest All right, that's what i'm doing our remem em Remember Our m m b r. There we go Um, so we're gonna this this trick here is Not only knowing what the largest number was but the word that was associated with the largest number So now I can print out at the end the word And the largest and that's the count Okay, and so we now we know that Oops, did we make a mistake here? Okay, that does not look good Because it says car And seven v is greater than the largest Oh It's not w I used a really bad variable. See that's the whole value there. There we go. It's k, which is the key key I was gonna say that was quite the bug See what happened there. I had this as w and it just happened to be it was the last word on the file car the last word in the file Because I used the wrong variable No little mistakes little mistakes The and seven Okay, so let's let's get rid of this print statement because we kind of know what's going on here And uh a way we go and this should now work if we run it And we can even get rid of the word done here There we go. The seven now the cool thing about this is this code runs just as easily with one line of code or the intro of the book intro dot txt and Not surprisingly that's still the most common word in the introduction dot txt. I seem to like that word and it's 226 times Okay, and so that is the basic pattern of Reading some this is just a word loop now. Sometimes there would be some You know checking to see if the line is the one you're interested in maybe tearing apart the line but it's at the end of the day this idiom of Starting a dictionary now. It's a common problem to to know where to start the dictionary You want to accumulate the numbers for the whole file. So you don't want to put it in here between line six and line seven Okay, so I hope that particular thing helps a little bit Helps you understand dictionaries Hello and welcome to chapter 10 now We're going to talk about our third kind of collection called tuples, but tuples are really a lot like this There's not too much to them They're really kind of reductionist version of lists there. So they they they function very much like lists and that You know they're they have things and the difference is is there are no square braces There is a parentheses round brace or whatever And they have positions zero one and two just like a list and you can look things up x sub two So x sub two is the x of the third element here and so that prints out joseph You can assign, you know make a tuple here This is the constant syntax for a tuple and print that out and the print statement shows you that This is a tuple not a list by showing your round parentheses And a whole bunch of functions that work with lists work the same way with tuples You can put a tuple at the end of an end statement into four as you might expect And then it iterates through the tuples tuples maintain order. So it prints out one nine and two so Literally this bit of code here could be identical whether it was a list or a tuple It really would do the exact same thing The difference between between tuples are that they are immutable once you create the tuple You can only sort of an assign a tuple, but you can't modify you can modify a list So if we take a look at a list here, we make a list that's nine eight seven And we say x sub two equals six. Well, that just means this seven becomes a six and that's just natural Meaning we can reassign slots. We can delete things. We can insert things. We can Mutate them. We can change them. So they're changeable Right they're changeable, but If we try to do that same thing with a string So we say y equals abc and we know that this is positions zero one and two But if we try to say let's change the c to a d by saying y sub two equals d That is not allowed and it says it doesn't support item assignment and this little Racket, you know x sub two is what they call item assignment inside of python And so if we do the same thing then with a three element Tuple put that in z and we try to change this slot to be a zero It's going to blow up because it's the exact same thing and that has to do with the fact that Once this assignment is made, this is not modifiable Now it turns out that the reason it's not modifiable is for efficiency Um, they take up less storage They are quicker to access and they're really designed internally behind the scenes in ways we don't really need to understand Um, they're just more efficient than lists to if all you want to do is store a list and look at it and then throw it away You probably should use a tuple instead So there's a lot of things that you can do with lists that you also can't do with tuples But they're really just a corollary of this notion of non mutability And so like you can sort a list, but you can't sort tuples You can add a five to the end of three two one can't do that in a tuple But you can in a list and flip the order dot dot dot dot dot dot dot So anything that you can do to a list that, uh, modifies the list not allowed for tuples And so you can take a look at the kinds of things that are inside the methods that are part of Each list a pen count extend index insert pop All of some of these many of these are modifying and then count and index are the only ones that work for Uh for tuples and so tuples are uh limited lists Now at some point there's going to be a but here to say why do we like them? And um the reason that we like them is that they're just more efficient They don't have to build in it python in its own internal organization of these objects It it knows that don't know we never be modified because when you make a tuple you as the programmer saying I'm never going to modify this and python won't let you do it So it's higher performance better memory use and you know to a beginning programmer that doesn't really matter But that's the reason and so we tend to use tuples when in Situations where we're going to make a temporary variable and then temporarily Use it just a little bit and then throw it away without really messing with it And we tend to use lists to build things up etc etc etc So the other thing that's interesting about tuples and we've actually sort of seen this is that you can put a tuple that includes variables on the left side of the assignment and This takes a little getting used to but it's really cool and no other language that I know of does this So if we say x comma y that's a two tuple both have two variables You can't put constants on this side, you know, it's like saying x equals four y equals fred Right, so what happens is is you can put a tuple on the far side of an assignment statement And the four goes to x and the fred goes to y And you say what's in y well y is indeed fred and so this is like two assignment statements Now the way I've got this syntax I would probably do you know two separate statements just not to show off that I know how to do tuples You know and so you can here's another one and they just move correspondingly if you don't have two here And you do have two here Well, if you have three here or two here and three here and you don't match the number there you get in some trouble Now if you just say x equals tuple then that is the tuple in the list But this is just a simple straight 99 value going into a so you can put tuples as the left hand side And you can even do things like return a tuple from functions. That's a real nice python feature that I like a lot Tuples are also related to dictionaries as we've seen in the previous chapter So here we make a little dictionary and we make an empty dictionary by constructing an empty dictionary stick it in d So d is sort of like this place that can hold key value pairs and we put csev And there's a two in there and chen one and there's a four in there So we have this you know associative mapping between c7 two and chen one and four All stuff we know and now we say hey We're going to loop through the key value pairs here and we've seen this syntax before k comma v So this is a tuple So you can think of this as each one of these things is going to get assigned into this tuple Which means the key ends up in and the first one's the key and the second one's value I use the vert variable keep kv all the time in code that I write just for my own sanity So kv are going to iterate successively through the successive keys and values In that so this is going to run twice and k is going to be csev two and chen one four the order Just happened to stay the same And so if you say What is in one of these things you can actually take d items the items method within that dictionary and say hey Give me back. Give me that to me back and then print tops and this is it's a special kind of a class But really ultimately it is a list of tuples You know this is two This is the zero and this is the two the one the first and the second and then within each thing you get you have a two tuple And so in a sense this k and v are iterating Through those things when we're putting d items here and d items there One nice thing about tuples is that they're comparable They're comparable in the same way that strings are comparable Meaning that they're compared from left to right with the leftmost or zero tuple being the most significant And it doesn't compare any further than it has to if they if the if it's asking less than So if it's looking at say this first tuple it starts at the left and says okay It asked the question tell me true or false Is zero less than five the answer is true And so the answer to this overall expression is true and it doesn't even compare those two numbers those second And third number they don't compare them If on the other hand we're asking is this less than that It only looks at the first one and asks if can answer the question the answer is well, they're both zero And so I can't answer the question So I have to go to the second one second pair and one is less than three And so that means this is true and it does not check this even though 20 million is bigger than four it doesn't matter because These are the numbers that cause the the true to happen And the same is true if If you do this with strings Again, we start the first one. So Jones sally. Well, that's the same. So we don't know the answer yet. And so sally sam. Well, okay s s, so they're the same a a they're the same. Oh l and m L is less than m So so the actual letter that makes the difference here is the l and the m and leads to us being true And so this it goes left to right, but then even when it's doing strings, it's going left to right. That's just how string comparison works and If we say say is jones Jones sally greater than adam sam. Well, we check the first one and we check the j and the a well J is greater than a and so we don't have to look at anything else We don't have to look at these any more of these characters We don't have to look at the second thing in the tuple We have to look at that is enough to be true. So it only scans until it has a definitive answer. It doesn't scan any further So now what we're going to do is use this comparable capability To sort these lists of tuples and then bring this all back and connect it more to dictionaries So now we can take advantage of the notion of comparing tuples and use sorting And so what we're going to produce is a list of tuples And then we're going to sort them right And so we can get a list of tuples from a dictionary and then we can sort that list of tuples And then we can add a sorting dictionary items by taking this two step process convert dictionary to a list Sort the list and then and then we can have assorted dictionary values Okay, and so we'll do this a couple of different times. So if we take a look at this code right here We have our happy little dictionary a b c map a maps to 10 b maps to 1 c maps to 20 Like what are we going to get here? Well, it comes out the mapping is the right way But the order is whatever and now we say this function called sorted which takes Inside a sequence and then returns us a sorted Version of that a list that's sorted and so it says sort d items So it's basically going to take this list And compare the a's and the c's and the b's and because it's a dictionary and all the keys are unique There's never going to be equality So it really is going to just sort this by keys and never get to looking at the values You can't you could construct a list that had duplicate You could make a list of tuples that had duplicates in the first like we did before But in given that this coming from a dictionary the first thing is going to always be unique and distinct And so if we say sorted d of items that we're passing This stuff into sorted sort is going to go around move stuff around And then give us back A sorted version it sorted in a sending order based on key without looking at the value And so the so that's a way to see Dictionaries sorted by key is just say sorted of d sub items and sorted is a function And so it just picks stuff and so this is the kind of loop that you're going to write to do that You know we did this before we took sorted and we got these sorted by keys And so you can just make this nice and simple for key value By the way, you can eliminate the parentheses here And I think it's prettier if you eliminate the parentheses, but you could put parentheses This is still a tuple without the parentheses For key and value in sorted so that says go through d items But before I go through them, please sort them So that means k is going to go through a b and c deterministically every single time It's going to go and of course value is going to go through the corresponding values So now we can print this out nicely Sorted by key and that's a real nice succinct little Way to say that I mean that again These are one of the kind of things that people really like about python is that you can do pretty powerful things with Easy to understand. I mean, you know, you might have seen this for the first time But ultimately you look at that eventually you're like, oh, yeah, that's I see exactly what that's doing easy not not hard at all So but let's say we're looking for the most common word, which we have been for weeks and weeks and weeks now And so we want to sort by values not key So this is an example of where we're going to construct a data structure We're going to imagine a data structure And then we're going to write code to construct the data structure and then that's going to make our problem easy So this is an example of using cleverly constructed data structures To do this and the data structure that we're going to create Is a list of tuples where the value is first and the key is second So you can just with items get key value I want value key. So Let's take a look at this code take your time and get it right. So k v goes in c items Well, that is unsorted and going to have go through whatever a b and c and whatever order And we're going to make a new list. So this is a data structure that we're creating temporarily And what we're going to do is this is a list And we are going to append to that list A tuple. So this is going to be a list of tuples Except we're not going to append them In key value order, we're going to flip them and append the first part of the tuple is going to be the value And the second part is going to be the key So we end up with this This is sort of our temporary data structure that we have constructed to make our job really easy So this ends up being 10 a 22 c 1 b now We just kind of flipped and we took this order and then we flipped them around And so now we have this nice little list sitting in memory in a variable and that's really simple We can say oh look we can use sorted and we can sort by now the values because they're the first thing The sorted doesn't know that they're how we produce this list. It just looks at that and says, oh, that's a list of tuples I'm going to always sort by looking at the first first item in any tuple And I'm going to add reverse equals true. So I get a descending sort. So I see that the The value that is highest ends up being first And so that changes this and I'm just sort it and then reassign it back into temp And I'll print this out. And so now you see it's sorted in descending order of key So it's value key value key value key, but it's sorted in descending order Okay, and so that's an example sort of up just like, you know, if I just made a data structure And I flipped those things around I could use sorted to sort these things There's many other ways you could do it, but there's sort of like the more elegant way of doing it And the the clever bit here is like make a new list and make it be a little bit different. Okay So here we're going to print out the top 10 most common words in a file And most of this code is review. So if we take a look at it We're going to open a file We're going to start a dictionary for our counting We're going to, you know, there's going to be words and lines Right and so we're going to have a for loop this for loop is going to go through each line And then of course we're going to split them which is busting them into pieces And then we have a for loop within that and this for loop is going to go through each word And so that means that be by nesting these loops We're going through each line and then within the line we're going through word Then we go to the next line and go through the words And eventually this line of code counts sub word equals counts dot get word zero plus one are idiom for making a histogram Right this line right here is an idiom if you don't know already what that is go back to the previous dictionary lecture And understand it understand it because you're just going to use it over and over again So now at this point and I always like drawing horizontal lines in code when we write it at this point Coming through at this point counts is right counts is the histogram It's not sorted so now we want to sort it so we're going to make a new list We're going to loop through key value and then we're going to make a tuple I'm I'm making this be two lines to make literally is your value key. So I'm flipping it right so Flipping the order of these things that's making a tuple and then I'm appending that tuple to the list Okay, so at the end of this We have a list of tuples in Value key order vk vk right So at this point coming through here, I've got in my lst variable I've got this really useful bit of code that are useful bit of data that I produced and then I'm like, oh Now it's ready to be sorted poof sort So take list sort it back and sort it into sending order and then stick that back in list Now we want to print it out, but we don't want to print it out We don't so we got a nice sorted list coming down here We don't want to print it out uh in value key because that's what it is It's in parentheses v comma k order, but it's sorted and we know that the most the highest value is here On down and so we're going to say, you know, we're going to run through and now we're going to go through this new list Only the first 10 Start at the beginning up to but not including number 10, which is the first 10 For value key in and so value is good So this is the iteration variable that's going to go through each of these things On and down and then we're just going to print it out flipping it So we re flip it flip flip we print it out key value and it's going to work Okay, so That is one way of doing this and this slide right here you absolutely do not need to figure out but some of you We'll look at this slide and you're like, why didn't you show us that in the beginning and others of you will be like No, no, no, no, no keep telling me this stuff here so I don't know exactly the term for this but this is A very procedural. This is a classic algorithms and data structures approach to solving this problem This next thing uses what are called lambdas and they kind of create what's called what I call a closed form Where you kind of do it in all one statement and there's all this implicit stuff going on So if you don't get this right away, don't worry too much about that but Roughly This single line does everything that bottom half of that program does. I mean if you go back If we go back to here, it's pretty much this line does everything Does that in one line? Okay It doesn't create the counts and it doesn't print out the top 10 But it does everything in that middle bit. So let's take a look at this So we all are going to collapse this down. So we have a print that parentheses the end of the print And then we have sorted and remember that sorted takes as input a list And so that's not too bad and returns us a list and so we'll print the return from sorted And then this is the funny part the fun part funny part. This is called list comprehension And we have square brackets and we say to python this is a list But instead of a listing of things or having a constant of one comma two comma three or a pen to pen to pen We're going to create an expression that will act as a generator for all the elements And so this basically says this is a list of two tuples v and k and then this is sort of implied For all k v in c dot items. And so this is like a for loop That is sort of driving this think of this is like stamp stamp stamp stamp stamp However many times it has to make a stamp. And so that's producing a list Right, it just manufactures this list and then that list is sort of manufactured in the moment There's no stock nuts not put in a variable it they it python makes that list according to the stamping pattern That you've told it to stamp out this list and then it passes that stamped out list without even storing in a variable In disorted sorted moves the list around because it is just a list of tuples And then gives us back the sorted list and so I didn't put reverse equals true on here But you see that this is sorted in ascending order now by key and I did that all in one little statement So so look at this there's a this is also one of the beautiful things about python that you can build these things And you can build more complex versions of this and and there's a lot of real elegant things that you can do in python They're really succinct you should be careful because in the beginning I think this is easier to understand even though after a while you're like wait a sec And why am I putting all these extra lines in because this is not so hard to understand But at some point, you know, you will want to master This more powerful and more succinct version of python that that expresses it in terms of the data You want to see rather than the steps you want to take So this sort of finishes up tuples.  I mean really They're simple and elegant tuples lists and dictionaries are all related They're really three different kind of three foundational data structures three foundational collections of python and we combine those in a lot of different ways And now in this little bit of lesson, we're going to talk about some tuples And we're going to create a list of the most common words And find out how to sort a dictionary by the values Instead of by the key We're going to use the clown dot txt file and the intro dot txt file And i'm going to start with the code from exercise nine that I just did From chapter nine It's not exactly one of the exercises, but it's very similar to them And i'm going to make a copy and i'm going to keep it in the same folder I'm going to keep it in the exo nine folder and just call it ex 10 because this Code is going to Do much of the same stuff and it's going to read these same files And so i've got myself exercise 10 exercise nine is still here exercise 10 is now what i'm editing exercise 10 But i'm in exercise nine folder so In exercise nine we look for the the most common word, but we want to find the five most common words Which is going to require us to sort so i'm going to get rid of that code Because it's not really how we're going to do it there We manually loop through it and found the maximum and so i'm going to just run this cd desktop python for everybody exo nine And if I do an ls you see that i've got exo nine dot py intro dot txt So i'll run python three ex 10 dot py And run the clown data and we see that we see the dictionary and it's properly Making it in this code right here that doesn't change it reads the file Reads all the lines goes through and splits it into words and then goes through the words and does the The idiom of using dictionary get to maintain the counters and we print it out at the very end So the new code we're going to write is down here Okay So let's first of you do do a few things um If I can say x is equal to The dictionary dot items And this gives us basically a list print x this gives us a list of the key value pairs this prints out the dictionary But if we do it this way and use items it gives us the key value pairs Okay, and so that's what we got Key value pairs now We can sort this Based on the value because tuples can be compared This can be compared with this And because d is lower than r then this one is lower this whole this ran tuple comes after the down tuple So we can sort this whole thing and i'll do this by just Putting the word sorted here And say give me a sorted version of that now it's going to do it based on the order of the tuples This is going to be more higher precedents than this So if I if I print it this way run it again You'll see that it's sorted and now is after and car it's an alphabetical order by key And so we could actually print the first five Up to but not including five by adding a list on the slice of this as list slice here And so that will show you Only the first five right Except that that's not what we're trying to do we really want to sort by this Okay So We have this mechanism that can take a list and sort it based on the tuple values If we could create a list where it was one comma after instead of after comma one And make it the exact same thing then we could actually then sort it and it would be fine Okay So let me show you a couple of ways at least one way to do that Okay Get rid of this We're going to hand construct a list And i'll just call it temp equals give me a new list temp equals new list and then for k comma v in The dictionary dot items And i'll just start by printing k comma v so we see And this is where it's really nice to do these with the clowned code first and then only do your test on the bigger file later And so it's pretty much the same thing. We are going through in key value Order, which is dictionary order, which is not sorted at all Okay Now instead of printing this out. We are going to Let me let me do this in a couple of steps make a new tuple And I'll just call it newt equals parentheses v comma k Okay, so this is i'm saying make a new tuple This is like a new tuple with two items in it and i'm going to make the value for and the key the the key Okay, so Then i'm going to say temp dot append newt new tuple So i'm going to i'm going to end up with a list of tuples Let me comment this one out and i'm going to then when i'm done here i'm going to print temp So if i run clown.txt You see what happens in temp. It's the it's still well, let's print temp twice I mean it's not sorted. It's flipped Let's print it That's okay Well, just that's the flip one Okay, so it's flipped and all we did is we made it instead of car comma three It's three comma car, but now we have a list Okay So now it's flipped and now we can sort that We can say temp equals sorted temp So it says takes temp and sort it and give it back to me and now i'm going to say print sorted comma temp Okay, so here's the first print When we flipped it we've got two tent But it's not sorted at all But after we sorted it it's sorted by tuple and the lowest is one after so so you'll notice that One is the same as one so it checked the second item in the tuple so down comes before after fell becomes after down Intro on alphabetic order, but now we get the twos so that all the all the ones Sort there and then the twos come Here but then within the twos it's sort in alphabetical order because like a String if it if the first character matches then it looks to the second character and then we see Oh, here we go the threes and then the one we actually wanted the highest one is the seven And so one of the things we can do is we can say You'll notice that we want the highest one not the lowest one So we can just tell this With this parameter reverse equals true And we just say hey sorted do this backwards do it from highest to lowest rather than lowest to highest And now our sorted one says seven the Etc. Okay, and so we want the top first five We can say Up to but not including five So this is now the top five So the sorted one is that's the top five If there it's a tie we're going to go and reverse alphabetic order, but let's not worry about that too much for now So it it makes a flipped list then it sorts the flipped list Now if I just wanted to print it out nicer I could loop through this new list. I could say four V comma k remember this is a flipped list So the sensible thing is what's coming up. I mean coming out of this list Each tuple is value comma key In temp and I'm only going to go up to far up through but not including five. So the first five And so I'm pulling them back out as value key because that's what they are They're value key see value key value key Value key so v is going to go through these and k is going to go through these and then I'm just going to print k comma v So this is kind of my flipping backwards because I I want to see them this way And that's the most common one car three And so it's just going through this up through the fifth one and then printing them out Okay, so let me comment this out Let me comment that out Let me just delete this So we have a dictionary. Let me comment out the dictionary We have a dictionary. We make a list and we make these reversed Tuples where we have the value first in the key second. We're setting it up So the sort's going to work and then once it's sorted we have to flip them back So we we flip them for sorting from key value to keep value key for sorting We do the sort then we flip them back with key value and print them out And it works fine. So let's try our big file intro dot txt And there you go Those are the five most common words in intro dot txt So you might ask yourself. Why did we use tuples? We probably we could have really used lists for this But tuples are more efficient than lists and you notice that we weren't going to modify We did modify the temp list. It's a list of tuples, but the tuples within the list We're not we weren't going to modify and so we tend not to Make lists if we can get away with using tuples And so that's why we made this was what this flipped tuple thing. Okay So I uh, I hope that was useful to you. I hope to see you on the net Oh and welcome to chapter 11 regular expressions The fun thing about this chapter is unlike all rest of the chapters you sort of had to Really understand every single thing in chapters one through 11 built on one another one through 10 built on one another But you can really get along without using chapter 11. It's not a really required topic But it's a fun topic and an interesting topic. So uh, so you can relax a little bit and Realize that you may or may not like regular expressions.  It's a little character-based programming language where you can You know do smart searching basically start searching and as you'll see in a bit will smart smart Smart extraction and it's uh, it's really almost programmable wildcard expressions There's no looping but there is looping and there's all this implicit thing You say look for patterns that look like this and then you give back things that match those patterns You know, we we do searching for everything And we're looking through large blocks of text Say go find me everything that has the word python in it or something like that So that's just such a common thing to do and regular expressions are a very structured way to go about searching for information They're very powerful, but they're also very cryptic and you may not like them But they're a lot of fun actually once you understand them and learning how to program them takes a while Writing good regular expression programs requires some try it play with it. Check it try it check it try it check it But once you get them they're they're really quite cool. It's a very old programming language You know, it comes almost from the 1960s the concept of it's the theory of computing Where they were trying to come up with theory of languages and regular expressions was one form of languages that computers could understand and so it has some fun old words and um One of the advantages of knowing regular expressions is that you're kind of a cool person You can take a quick look at this xkcd that sort of captures the The devil may care awesome power that regular expressions do and uh, and while we're at it You know and while we're talking about awesome. I do want to take this moment and show you my awesome tattoos and so You may not know this but I got a couple tattoos here. Here's the first tattoo This is where I went to got my phd and this is my university michigan faculty member position I got phd in engineering and I teach in a school of information in library science And then I have this other tattoo and this tattoo is what I call the ring of compliance I work on learning management systems and educational technology and standards And there's this standard called learning tools and our operability which if you're using this course And doing the auto grader it uses learning tools and our operability to integrate into whatever learning management system you happen to be using And one of those learning management systems is the open source learning management system that I helped write called sakai And these are the rest of the major vendors and the idea of that tattoo was that I would put the tattoo of every vendor that would comply with learning tools interoperability And so you'll notice corsair I help corsair put learning tools interoperability in and so the auto graders integrate into corsair Blackboard or canvas or sakai or moodle or often those are other things. So it's just like a cool techno thing Just like regular expressions So I've got a url here for regular expression quick guide. You might want to print this out so that you can Look at it even while you're watching this lecture Because it's a little programming language except that it's character based not line based and not keyword based It has certain active characters that That the character means something versus the character represents the character itself And so the regular expressions is not part of the base python, but it's distributed with python So you have to put an import re to at the top to say that's really saying pull in the regular expression library And there is a couple of functions inside that re.search Which is kind of like a really smart version of the find method inside of strings and re.find all which is kind of like Like taking and stamping your way through a loop through a string and finding all of the things that match a particular A particular pattern and then extracting those and we'll talk about both of these in this lecture So here's a really simple piece of code Where i'm just going to sort of show you sort of before and after So here's a thing where we're looking for lines that begin with from colon And so we open a file we loop through the whole file We strip off the lines a text and then we say if line dot find from is greater than equal to zero Then we print it it gives you negative one if it's not found and so Reads all the lines and once in a while it'll print it out reads all the lines one to all print it out So that's kind of like a needle in the haystack Do use regular expressions to do that we have to import the regular expression library These lines are the same we're going to loop through we're going to strip And how we now we're going to say if re.search the way to say this is within the library regular expressions go find the search function And search for the string from in the string line Okay, so this is the line to search whereas here it was more object running where we say line dot find Here we say re.search and we pass in line as parameter These two things are equivalent Which means most of the time it's going to run and once in a while hit a line and it'll print that out And then it'll finish the whole thing. So that is Taking and doing what we would do with the find operation with regular expressions now Searching with regular expressions has these special characters. And so here we have the same Basic code except now we're saying if line starts with from So we're not using find anymore And um that way we're only going to get that thing in the first position not like blah blah blah from Colon we don't want that to match we only want to match here at the beginning of the line And so that's we use line starts with so it's going to do the same thing and find lines that have the prefix and print those out and then be done Now in regular expression search we don't in a sense change the method We we have a certain number of things we can do with strings based on what they've built in But in regular expression, we actually can turn this first parameter into code And so what's happening here is the carrot if you go back to a little cheat sheet carrot means This is the beginning of line. It's a virtual character that matches the beginning line It's like from that starts at the beginning. So from At the beginning does match and from in the middle does not match by putting that little carrot there Same thing line is what we're searching and then from is what we cap carrot from Lining from at the beginning is what we're looking for and so again It does the exact same thing only prints lines that have from colon is the first character in the line So the difference is as we look for a method and the other one is we program the regular expression So we're going to run out of methods in the string Class long before we run out of things that we can do with regular expressions And so a couple other special characters that carrot matches the beginning of the line So carrot matches the beginning of the line. This capital X matches itself Dot is a wild card that matches any character and then some of the characters in regular expressions modify the immediately proceeding character and so that says Look for a line that starts with x and then has many characters. That's these two things Zero or more characters followed by a colon And so you can see that it's sort of it's this sort of like expanding stamp It's like oh, there's next to the beginning of the line that line It looks good. I got some characters here and then I got a colon. That's good So this is an x some characters and a colon check x some characters and a colon check x and these things You know a way we go and so you can that's what's going to match And so you can see how some of these characters are special and again go back to your cheat sheet Some of them are special and some of them are actual characters and this colon and x are just they're they're not special They're just the characters okay now Sometimes you want to be a little more clear on your match So let's take a look at these lines that that match that particular thing that we just did So we have these two x dash of colon x dash d stamp dash result Like these are from mail messages and then one of the mail messages has a line that says x dash plane is behind schedule And this matches Is that what you really wanted and so what we can basically say is because this is an x This is some number of characters and that's a colon It matches it has to match that that's this rule applied to this line results in a yes It does and so how can you be a little more clear as to what you want to match and what you don't want to match so we can write code so now what we're going to say is We want to match the beginning of the line and we want to care capital x and we want to dash So now we're going to match those first two characters x dash at the beginning of the line Carrot x dash says first two characters of the line must be x dash Now we have another special character again refer to your cheat sheet back slash capital s means a Non-white space character right Any character other than white space and then plus means one or more times one or more Non-white space characters. That's what this whole thing says one or more non-white space characters and followed by a colon Which is just a character. So now we have x dash followed by one or more non-white space characters followed by a colon x dash followed by one or more non-white space characters followed by a colon Here we have x dash followed by one or more. Whoops. There's a space there And so this doesn't match even though there's a colon there It means that between the dash and the colon You can only have some number of non-white space characters. So this is a No, it does not match. And so you just can if that if you didn't want to match this you then sort of create a more precise You know, we could even have a thing that said I want x dash with an uppercase character uppercase letter If you wanted to and so there's all kind of fine tuning if you sort of learn the structure that you've got to do And so that's kind of the matching where you're taking a whole line and taking this template and deciding if the template anywhere in that line matches And now what we're going to do is use this to actually pull data out of strings Using the regular expression library So now we're going to move from merely matching to matching and extracting So we're going to say hey, I would like to not only have you take this template this little pattern the string pattern regular expression pattern Run it across the line I want you to give me all the ones that match and I want a list of those and that's what we're going to use the find all So search gives a true false find all gives a list of all the strings that match So if there's four of them you'll get four things in the list If there's nothing that matches you'll get an empty list so Let's take a look at what we got going here. So instead of calling Search we call find all we still pass in the string that we're looking through and then we have our little template pattern And this is a new bit of regular expression any little bracket operation square bracket Is one character. That's just a character, but then they're in between here is a set of Of allowed characters So zero dash nine means eight single digit zero one two three four five six seven eight or nine But that's really one character and then we have so that's one character And then when the plus applies to that which means if we look at this whole thing This whole thing says one or more digits. That's the code We write in a regular expression that says one or more digits And we're just going to use that in our regular expression by itself So we're going to look for any string That's one or more digits and pull it out and give it back to me. So we look it's going to look So that's my little template stamp stamp stamp stamp. Oh got it stamp stamp stamp stamp stamp stamp stamp stamp stamp stamp stamp Oh got it stamp stamp stamp stamp got it. So what we get back after we ask find all to find all of the one or more digit strings is Two nine and forty two so it actually parsed it it split it it found all these things and said I found them all for you And here they are two nineteen and forty two so it's a list Of three strings because that's how many you found now might have found none And we've got an empty list at that point But it found some Okay, so just as an example, you know, we did this thing we get two nineteen and forty two But if I said this that basically is A uppercase vowel a e i o or u so that's one letter and that's one or more So it's saying, you know something like a a would match E i would match O o would match But if you look now it's saying, okay, that's i'm looking for one or more minimum one or more uppercase A e i o u is a set of characters one or more uppercase letters And so it's like look do you find oh there's an uppercase, but it's an m no no no no uppercase no uppercase no uppercase no uppercase found nothing Did not find anything And so it gives us back an empty list and so it's like find all the things that match this and the answer is none matched Here's your list of nothing Okay, and so that's and so you have to check that's how you have to check even if you got something because It's not going to return you false. It returns you a list with no items in it now The way it works like I said, it sort of is taking this template and stamping it across the line Scamping across the characters now there might there's a behavior that might not be intuitive you Intuitive you at the very beginning But the notion of what we call greedy matching and that is when it can match more than one possible string Overlapping string it chooses the largest overlap of the overlapping strings And so the easiest way to show this with an example And we're saying I want something that starts with an f with one or more characters and ends with a colon So that that's there. That's my little stamp. That's my template. So starts with an f good. That's good One or more characters. Da da da da have a colon. That's so that could be from colon That would match But look I've got another colon here and this is just continuing on with one or more characters And this so the question is do we get this or do we get this part? Right and the answer is with greedy matching is we get the larger of the two Okay, and so what you get back is somewhat counterintuitive You get the whole thing is the match from colon using the we could have got from colon But the reason it picks this is this one's longer. So anytime it has a choice It picks the longer one and that's what greedy is meaning you probably better described as larger or Tending toward the longest string Or something like that So you can of course suppress this behavior like everything in programming regular expressions use simply add another character and so now It's going to say I would like to start with letter f Any character one or more times and then this question mark. This is still one, you know one little thing um non greedy okay And so that's just says do the do it not greedy Which just means that it prefers the shorter of the strings and so now it could still match this string or this string But because it's been told to not be greedy It chooses this string instead and that's the string that we get and so that's the not greedy And you just had the question mark after the asterisk. So it's usually An asterisk question mark or a plus question mark. Though that's a two thing That's zero more characters non greedy and that's one or more characters non greedy Actually, most of the time the the it seems to me that the Non greedy would be the more reasonable default, but that's not how it is a greedy is the default and non greedy is optional now We can play some more with this stuff. Okay, and so Let's take a look at this little example where we have a non blank characters back slash capital s One or more of those non blank characters Followed by an at sign and then again one or more non blank characters So this is looking for strings that have an at sign with non blank characters on both sides This is an example of where it sort of comes to this at and it goes this way and it does it in a greedy manner If you told it to to not be greedy it would give you this These three characters, but we're telling it to go greedy so it goes all the way to here and stops at this blank And then stops at this blank and so that's a nice little thing find the at signs Go to the the first blank blank and pull that stuff out And so that with one little match you pulled this thing out now, of course, we've done that before other with other techniques So that's just another way to pull stuff out now if we We get this whole thing, but what if that's not exactly what we wanted we can tell We can We can give it a matching string that's different than the extracting string by adding parentheses And so here's another example where we basically say This is our string. We want to match from at the beginning followed by a space Followed by ignore the parenthesis for the minute One or more non blank characters followed by an at sign followed by one or more non blank characters So this is also going to if there's no from it's not going to be looking for that, right? So it's it demands the from is here So it matches that and the space is demanded as well And then it says oh non blank characters great. I got an at sign great non blank characters. Oops stop there And so This is what's going to match now the key is that we don't actually want that back in our extraction What we really want back in our extraction is this part right here So what we do is we put parentheses in parentheses don't our code They're they're code in the regular expression world parentheses say start your extraction and end your extraction And so when you do this with a parentheses when you when you do it, you know without a parentheses You get you get the whole from right without a parentheses Oh wait, no, okay, that that doesn't have the from in it so um But if you do that with the parentheses The you match the from but you only get the This bit to come out as well so you can add this to make the matching part more precise But without changing what you get returned and you specify what you want to get returned with the parentheses So next I want to show you just a couple of different ways to use these newfound skills So now what we want to do is use some of these newfound skills in some more practical applications of regular expressions So let's go back to the way we first tore apart strings and And look at the situation where we if you recall we just wanted the host name, right? This is an email address and we're interested in the host name So we have this string And we go find the at right the find looks up and tells us the at is it position 21 And then what we do is we say, okay, let's look beyond there to the space And that tells us the space is in position 31 And then we're saying we can extract starting at beyond the at sign up to but not including the space by saying at pose plus one colon space position And when we get that now we have to have a thing that decides to only look at this on from lines But then it can print out the host that is extracting of this information So that was one way that we did that right one way. We did it The next way we did this was the double split pattern Right, so we said, okay, let's take this line. Let's break it into words Based on spaces. That's what words is so that's 0 1 2 3 4 5 6 And then we know that the email address on lines that start with from space Is the second one so we pull out email address which pulls this bit out Into email and then we're going to split that again Based on the at sign So we're going to split this part again based on the at sign So it splits right there and then this becomes the zero and one in pieces And then pieces sub one is that host and if we print that out We get the host So that's the double split pattern. Nice thing about that is you don't have to keep track the little plus ones kind of annoying to use the space position The previous one that's just hard to remember. It's just I've written this code way too many times in my career and I've made mistakes And I have to debug it every single time and I print all these numbers out. I'm like, did I get it right? Oh, I did it in python. I did it in java. I did it in c. Wait a second. I did it differently. And so it's So this is a lot cleaner I mean I can write this every time and I know it's going to work every time I barely even need to test this code because it's so obvious So double split is another way of extracting stuff But if we look at this thing with the regular expression we can say, oh, okay Let's um, let's use a regular expression to do this So we'll start looking through the string. We'll start by saying, hey, let's look until we find an at sign Then let's start extracting with the parentheses And then once we have found the at sign Let's look for for for non-blank characters. This is a set of characters. This carrot as the first one means not A blank So that's another way to do non-blank not a set of characters which are everything but blank That's what this little bit is saying star means Zero more times which means it's going to run run run run until it finds a blank which is going to stop it The greediness is what keeps pushing it, right? It's this is a greedy match that asterisk is greedy because there's no question mark after it And so that does go and starts at the at sign With the parentheses Goes to the space and that's the end parentheses and that's what prints out now Y is going to be a list that's a one item list that has the string in it that we're looking for but You can just go sub zero to get that guy right out of there Okay, so that's sort of the regular expression version of it, but we can Make this a more fine-tuned thing So we can say look we don't we also want to pick the line And we want to know if there are if we don't get that line We want to skip it if we do get the line we want to extract the data and we can do this all in a single regular expression So again, we say start from the beginning of the line And if it's got to be a from followed by a space and then followed by any number of characters Dot star followed by an at sign. So it so this has to match We see a space then we're going to have any number of characters And then we're going to see an at sign and then we're going to start extracting And then we're going to go non-blank non-blank non-blank non-blank non-blank up blank and extracting And out that comes and this has the advantage of the previous one and that makes it much more precise There if we look at the previous one while it works on good lines It might actually trigger on lines that we actually don't want to see So this allows us to refine it. So it only actually does this to lines that we care about So it's sort of a both an if statement And and a splitting extracting going on all at the same time by having a bigger String that we're matching than we're extracting It's a way to kind of Clean up your data. So here is a simple program that we're going to just put all this together and actually accomplish something And so we're going to we're going to read through and look for lines in a file that have this form And we're going to extract this number and then we are going to Compute the the maximum of this. Okay, so we're going to extract this number and then convert it to float and compute the maximum So, you know, we're going to open a file We're going to write a for loop We're going to strip so we're going to do this for every line of the file But the first thing we want to do is not get line we want to discard all the lines Except ones that have this so our our regular expression is look for lines that start with x dash d spam dash confidence colon So that's a pretty strong match if that's not there We're not going to get anything and then there's a space. There's a space and then start extracting And then go as long one or more digits and dots That's a single character and that's one or more and then stop extracting So that says start extracting da da da da greedy greedy greedy greedy stop extracting And so that's what we're going to get now If the line doesn't have this it means missing in some other some way Whether it's this prefix or this number if the number is missing it's going to fail too We're going to get back a list An empty list So the first thing you have to do is check to see if you actually got a match So you say if the number of items in the list len of stuff is not equal to one Continue and so this is the this is the skip all the lines that don't match Skip skip skip skip skip skip skip so there could be thousands of lines that don't match But then when this match hits it's going to come down and fall through Right so so that Most of the lines will skip up But then when we actually get one and we know Instantly that we've got one and stuff sub zero because that's what we extracted Is this number and we can take the floating point of it. We impended to our list We made a list to store them that runs The list grows And then we just say what was the largest one and so you can run this And see that We have an escape character and the whole idea is is sometimes all these little special characters that make a lot of sense to us We actually want to search for it. So what if we want to search for a dollar sign? Well, we just prefix it with the backslash and that just means this is a real dollar sign So backslash dollar is a real dollar sign. So this says I would like A dollar sign followed by one or more digits or dots And so that's going to match a dollar sign followed by one or more digits dots are okay This is a set remember zero dash nine or dot That's a set of the list of legit characters. This is a range of characters That's a shortcut to how to make the set you could make it be Zero one two three five seven eight nine dot or zero dash nine and it assumes that and that's one or more So then stops because this is a space It's greedy matching Then it pulls this out So that's kind of why greedy has to be the default because because otherwise if it wasn't doing greedy matching Oops come back come back If it wasn't doing greedy matching It would If it wasn't doing greedy matching it would stop here because it would find a dollar sign non greedy would find a dollar sign And one character and then it would give us dollar one rather than dollar ten so in summary regular expressions are a cryptic but powerful language and They're they're an acquired taste. I think that I bet eventually you'll find them fun Even though on your first impression you might not think that they're so fun Welcome to network programs. This is chapter 12 now We're going to learn a little bit about how we talk to resources on the network using python Now this is a really quick introduction to how the network really works. I have a whole book that I wrote Um, it's also translated into Spanish On how the network works starting at the very lowest layer packets and everything Right on up and it's actually really easy to read. I wrote it for a high school audience It's a short book and pretty easy to read So if you read that book you will understand that there is this layered architecture The tcp architecture that sort of runs our network at the lowest layer that on one side here This is your computer and this is a server computer And if you sort of want a web page goes across the network does this like 15 or 20 times Then it goes up into the server reads the data and then the data comes back 15 20 hops for the packets and then it's shown to you as what you see And so that's how it works and there's all these layers that we're not going to talk about in this section But I talk about in that book The layers of the link layer which talk about how to get over one hop the internet layer which talks about how to construct Say 15 or so hops to get packets back and forth That's the the sort of lower level bits We're going to start at what we call the transport layer and that's the layer where Your computer sort of assumes that it can make a phone call To another computer another process running on a program on this computer talks to a program on this computer And then it kind of comes back Okay, and so we're going to we're going to leave this alone. We're going to ignore it We're going to assume that there's this nice reliable pipe that's going from point a to point b And what are we going to do with the pipe? But if you're interested take a look at the book So we're going to start with a pipe of some kind of a connection. We have two processes Process process and we have some kind of a connection between them And it is a connection that we can both use to talk And to listen In nerd terms we call these things sockets and that is one process running on one computer another process running on one computer Another second computer connected through the internet somehow and One computer speaks into that socket and it comes out and the other computer returns something and it comes And so this is a bi-directional protocol of data, which is a series of an effect Data phone calls between applications. So the application might be on your side. This might be your browser Chrome firefox internet explorer on the other side. This is a web server Might be internet iis internet something something from microsoft or apache or Java tomcat There's another program and you are making phone calls between these programs now in general These servers here stay up all the time and you sort of just can make a request when you feel like it on your In your program, but that's what we're going to do and this is what we call a socket So that little connection that phone call that data phone call is what we call a socket Now you have to decide which of the systems you're going to talk to and then which of the services on those systems or which process And so we have this concept called port numbers and they're best thought of like extensions on phones So one organization has one phone number and it says please enter the extension of the party. You'd like to talk to Well, that's kind of what ports are there like here is i'm a server and i'm connected to the internet Please enter the extension of the process that you would like to talk to And so for example, there might be processes running on various computers And so the email is known to hang out on port 25 or extension 25 log in insecure login lives on port 23 Insecure web lives on 80 and secure web lives on 443 and there's a couple of different protocols Say if you have your mail stored on gmail and you have a local Mail client say like thunderbird or apple mail that talks a protocol to pull that mail across and those live on various ports So these ports are those extensions and by convention We have standards that tell us what to roughly expect at those ports So when you're talking to port 80, you expect to talk to a web server or an http Server if you're talking on port 23 You expect to talk to a telnet server and on and on and on and on and on and so these are the extensions the typical commonly used default extensions for various network application Processes that are serving us data Now sometimes you'll go to a url and you'll see in that url There's a colon and a number that means it's a web server That's running on a port other than the official 80 or 443 port Now in python we can talk to these sockets right we can just talk to them and it's really easy Surprisingly easy We have to import socket because that's a library. It's Comes with python, but until you can use it. You can't use it in your program until you say it And then you basically in the socket library call socket function. That's what that syntax is saying um You're making a socket now the key to a socket. It's it's sort of like a An unopened file handle. It's half of a file handle It's an it's an outward looking thing that's not yet connected these parameters You're just going to type them in this says we're going to make a socket that goes across the internet And it's a stream socket Which means that it's a series of characters that come one after another rather than a series of blocks of text There's another kind that's harder to deal with but we're going to do this So this don't worry about this line. Just know that this creates a socket but not does not associate it The very next line we get back a socket a socket object in this variable that i'm storing in the variable mysoc And then when you want to make a connection across the internet to the far end You say oh, hey dear socket extend yourself across the internet Make the phone call to this host data dot pr for e.org and on that port 80 So that's making the phone call. This is like the phone number and this is like the phone extension So that's we haven't sent any data yet. We have simply Rung the phone of a process.  Maybe you don't have a network connection Or maybe that service is down on that server or the whole server is down but So I just it's kind of amazing that we're sitting here in python and in three lines we have Uh, probably a half a million engineers have built this thing called the internet all these protocols and all this software And we just made use of it in three lines of python in a case This is one of the reasons that people absolutely love python. Absolutely love python So now that we have a socket we have to ask ourselves What kind of data are we going to send and then what kind of data are we going to expect to receive across that socket? So now we have a socket. We are going to talk about what we're going to do with it, right? So the socket basically functions at this level your application is saying make me a socket Which is sort of this endpoint and then the connect actually connects to an application on the far side And there's a port involved. So that might be port 80 and this this is the far host and that could be Py4e.org or data.py for or for e.org Okay, and so the socket is solving this and and the question then is What are we going to send and what are we going to expect to get back and that's what we call the application protocol So we know that these two have made a phone call There's no different than making the phone call and saying, you know, hello, right? And uh, everyone knows that when you the phone rings and you pick it up, you're supposed to say hello And that's part of our protocol. So who talks first, right? So the dominant protocol that we use on this in this section is the http protocol It's the key is hypertext transfer transfer protocol. It's dominant. It's really easy to use That's why I use it as an example But realize that there are many others like mail and file transfer and remote login and all kinds of other protocols Each is a different application protocol. They all use sort of sockets at their lower level But then on top of that they layer the rules of the road for retrieving Hypertext web pages and we have used these for all kinds of other things So the protocol like I said is like who answers the phone first. What do they say? What happens if the person doesn't answer right? Can you hear me now? Those kinds of things and it's a real simple thing and it and all you really need to do Is so that both sides can agree You have to write a thing that's like the rules in the middle and say okay everybody as long as we all do this We'll be fine It's as simple as picking on which side of the road the cars can drive on it works fine no matter which side But if each car randomly picked it would be really kind of a mess So if you look at the typical url and this is one of the things that the web innovators in 1980 Really invented that was wonderful and and it seems second nature today But in 1990 it was rather revolutionary and that these uniform resource locators Encrypted included in themselves a protocol the host to connect to in the document to retrieve So this is one of the Clever ideas that the web came up with because we used to have to pick a program like fdp or telnet or whatever SMTP then we had to go to the right host and then we had to talk to that host a certain way So in HTTP it's a really simple protocol invented in 1989 and 1990 by Tim Berners-Lee and Robert Caillou at the World at the at CERN And they created a protocol that we have grown to know and love and use for way more than retrieving documents As we'll see in the upcoming chapters So we're going to talk a little bit about what happens when you click on a page that has a link now There's all kind of fancy stuff that can go on but this is the basics And so let's just imagine for the moment you start sitting looking at a web page Dr. Chuck comm slash page one and inside that there is a hyperlink. It is a Indication that says when you click on this page go to a different page and in that you see The name of the page that you're supposed to go to So we click on this link And that is a browser. This is an application. This is a process Or an app That's running on your computer. This is the browser Okay, and when the browser sees the click inside your computer Then the browser makes a connection to port 80 on the web server dr. Chuck comm And sends the request this request that it sends Is precisely specified by a standard which we will see in a second Then the web server does some magic work Oops, let's go back Then the web server does some magic work in here read some files run some code does whatever Constructs an answer to our phone call and sends it back And it sends in this case back a web page in the format of HTML the hypertext markup link which is different than HTTP Which is the protocol that we're exchanging HTML is the format of the document we're getting back and in this has an anchor tag href and the end of anchor tag and some highlighted text and now your your browser gets this back And then renders it according to the rules of html and css and JavaScript etc parses it and then makes a pretty web page And this web page happens to have a link back to the first page And if you click there it will do this over and over and over again And that is the request response cycle and that's governed by a series of internet standards These are standards that were built in the from the 60s 70s 80s and 90s and continue to this day Brought by a group called the internet engineering task force or ietf This documents they produce are called rfcs which stands for request for comments The rfc the word rfc is kind of like a sort of joke as it were it's a it's a They're they're trying to be kind of funny in that Funny is not the right word. It's it's ironic in that they're trying to say even so in the protocols of the internet that we've used for Several decades They're always interested in improvements and that's what the rfc stands for and they're all named rfc dash whatever And if we were going to cruise around we could find some various rfcs and this is rfc 2616 There it might have been revised since then but this is like a document And this is what they look like hypertext transfer protocol Version one and so you're reading this document You're going to write a browser and you want to talk to the application protocol that is http This is one of many documents that helps to find what http is So if you look down and look down said, oh, here's what a request looks like This is how i'm going to get a get a document from the server and you keep reading and you keep reading and it says Um, you're supposed to have the request method with a space with the request You are the crest method with a space with the uri with the space the http version and the carriage return the line fee That's what it's saying And so it looks kind of like this right we say get The document followed by a space they've got to be one space you do two spaces And it's going to be quite frustrating. Okay, and so this is an example that you can run on a number of Uh On on linux operating systems and when uh macintosh operating systems with no changes if you install telnet on your windows box You should be able to run something like this as well so telnet is a Program that we used in the old days It used to be how we logged into servers, but because it doesn't encrypt your data back and forth We don't use it anymore, but it basically is a program that can open a socket to a host on a port And i'm saying telnet to this host on port 80 And at this point I am connected and whatever I type on my keyboard is going to be sent to that server Now if you're doing this you probably want to cut and paste this really fast Because if you take too long most web servers will be like you're a human. I don't want to talk to humans I want to talk to programs. So remember to type this fast enough and then you have to hit enter Twice so you have to have a blank line here Just type this exactly as it's shown And then you will get back to a server if you do it right the server and the server is properly configured The server will give you back some headers And this is metadata about the document you're going to get for example It's saying it's got text slash html Which means that the remaining stuff is going to be an html hypertext market language It has a blank line and then the actual document and then the connection is closed And so if you do this you can set this up in a way that you can run this on your own computer and in effect hack Through the back door a web server now you can't hack the secure web servers and Male servers used to be easy to hack but they're harder to hack now because they challenge you for information but far the reason i'm so obsessed with The command line is this is how real hackers work and they know how to talk some of these protocols more directly And so we think of this beautiful sophisticated application talking to some other thing And it's all pretty and we got wonderful clicky buttons and nice usability But the reality is like in the matrix reloaded here The kinds of things that really talented hackers are doing Use command lines and um, and they really know what's going on and that's how they do it They understand what's going on better than the developers of the computers that are trying to be resistant to the hacking So i come from a long line of using the command line and that's why i encourage you to use the command line in this course So the next thing we're going to do is we're going to go up into the application layer and instead of typing those commands by hand We're going to actually send them from python and write a very simple python web browser In this section, we're going to write a web browser using python So we've already got a socket. We know how to write a socket in the previous section We played with the protocol and used telnet to do it by hand and now we're going to do it in python And what you're going to find is it's not that hard So here we go So the first three lines of this program import socket make the socket remember the socket Isn't really got the connection. So when you make the socket Again, we're going to make a stream based socket and when and it's suitable for going across the internet The connection that it's like ring phone call Connect to data.pr4e.org and port 80 and so that basically says extend the socket across And connect to a web server and so there's got to be a piece of software running And this will blow up if the software is not running Okay So then Now we've got a phone we've made a phone call Now whether or not the remote side says hello or not Is up to the application protocol and in this case the web servers say nothing and they wait for you to talk first So we're the web browser in this case and so we're going to talk first And we know what because we read the documentation. We know that we're going to send get blah, blah, blah, blah, blah, blah Space blah, blah, blah, blah each t1 and then two new lines return return. Remember you had to have a blank line We'll talk a little bit about this end code. It's preparing the data to go across the internet And then we say send it and so this basically takes that little string and sends it across the network And then this piece of software is waiting for it And then the software goes and reads a file or does some other stuff and then it starts sending us data back Which we can then choose to receive. So now we write a real simple loop We're going to receive the first we're going to receive these things 512 characters at a time So we're going to loop through receiving 512 each time And if we get zero characters, that means it's end of the stream. The stream is closed And if you look at the little example from the previous one, you saw connection closed When the connection is closed, we get an indication that it is because we ask for some data and we get zero data Otherwise, if there might be more data, if this will wait if the network is slow You'll see if you do a print statement in here You will see that this will pause from time to time on a really slow network If your network is fast, it'll just go blank and it'd be so fast. It won't matter But this is how we go. So this is basically until the entire Socket until the socket is closed. We're going to read this data And because this data is coming from the outside world, we have to decode it before we print it And then when we're all done we break out of here and we close the socket So literally that Is an entire web browser written in 10 lines of python And again, this is why everybody loves python So this is what this program will show if you run Um, the get is sent. It looks exactly like doing it by hand. Um, you get some headers Again, this is metadata that tells you something about the file. In this case, one of the important things is what kind of thing is coming next There's always a blank line between as a break between the headers and the actual data the metadata and the data And then here is the actual text of that romeo.txt file And then it's going to run this when a print data dot decode all this is coming from the print statement If you're going to parse this you have to know that you're going to read the headers up to a little blank line The blank line is your indication as a software developer that the headers have stopped and the actual text begins And you know the syntax. This actually could be a jpeg Or png or some kind of image, right?  And so now you have built a very simple web browser Next I want to talk a little bit about the what happens when, uh, characters transition Out from outside your computer. I mean from inside the computer in strings out across these sockets to servers and then back Hello, everybody and welcome to some worked sample code If you are interested in the source code you go to materials And download this sample code dot zip. I have this Downloaded it'll be in a folder called code three on my computer. This is where I'm at. I'm code three folder And this has a ton of bits of code here So if I do an ls you'll see I got all these files here and so, um We'll just leave those there And so this is the one I want to work through right now is this socket one dot py code and Basically what we're doing here is we're simulating. Uh, what is going to happen in a web browser And the cool thing about the html the htdp protocol is that we can we can do this by hand And i'm actually going to hack this htdp protocol This is going to go to data dot pr for e.org and retrieve a document um And so i'm going to do telnet To now you can do this on a mac and linux and if you put telnet on a windows box You can do it here data dot pr for e.org And I want to talk to port 80 and the port 80 is a different port It's a non-standard port, but what we're doing here is talking to the http port And so i'm going to be able to hand send commands To the web server and retrieve a document So i'm gonna cut i've already copied this string this get httpromeo dot txt i'm copying that into my buffer because if I wait too long This won't work. So here I go and now i'm going to type that and I have to hit enter twice And that literally was the http protocol What I typed there was the htdp protocol and the web server responds with some metadata about the document How many how much data there is the kind of data is there? um A blank line separates the header information From the the body of the document if I was to go to this in a browser right there you would see And if I turned on developer console And I went to the network. Let's make this a little bit bigger You would see that It retrieves this file romeo dot txt And it gets back that it tells us it shows us the headers and it shows us the response So this is all the same Way of doing the same thing and that is how to do the http protocol Okay, but now we're going to do this in python. And so here's the code. We're going to write So we're going to import the socket library and we're going to make a socket Now this doesn't actually make a connection think of a socket as a file handle That doesn't have any data associated with it yet And then what we're going to do is we're going to reach out and connect That socket to a Destination across the internet with the domain name of data dot pr4e dot org And the second parameter in this tuple this This is a function call with a single tuple as a parameter And so tuple sub zero is data dot pr4e dot org and tuple sub one is the 80 which says I want to talk to port 80 That could fail it will make the connection and And if the port 80 is there a way it goes and then we're going to actually send the http command So get this is the http rules followed by an end of line Followed by a blank line. So you saw me do this there This was what I typed here and I had to type a blank line Now if you want to go read the rfcs for how to do this you can figure this out So the only other thing that's kind of weird here is We have to add this dot in code Um, and that's because there are strings inside of python that are in unicode And we have to send them out as what's called utf 8 and in code converts from unicode internally to utf 8 So this command is a set of utf 8 bytes that we're then going to send it still has that same set of characters in it Um, and now we're going to send it and that's After after we've made the connection We're going to send these two things and then we're going to wait And my my sock is like a file handle at that point because it's been opened and we've sent data The htp protocol told us what this we had to send and the fact that we did have to send it So now I have just a simple while loop And I'm going to ask up to 512 characters And you know receive up to 512 characters and get that back If I will know that this is the end of file if I got no data back So if the length of the data the abider ray that I got back is less than one then I'm going to quit Otherwise I'm going to print the data and I'm going to use this decode which is kind of the opposite of this end code What I'm getting is utf 8 encoded data Most likely and decode basically converts it to the internal format called unicode that runs inside And so this is going to run a bunch of times pulling in the blocks basically 512 up to 512 characters at a time Printing it out. And then when it's all said and done, we will close that connection. And so it's not too exciting python 3 socket 1.py And you'll see that it's just going to python is now going to do what I did by hand Now, of course, the interesting thing is these are all in strings, right? And so You know this way we could write code that does stuff with this But all we're really trying to do in this particular situation is show how you open a socket Send a command and then retrieve the data Okay, so now it's time to teach you a bit of complexity about text processing up till now We've kind of been ignoring the complexity of text processing Everything that I have been doing most of what I've been doing is in ascii Uh the latin character set the character set that you know United States Europe lots of western civilizations use this character set and if you go back to the 1950s and 1960s They we were happy to have one computer and we didn't care what the character set was as long as what you typed on the Keyboard came out on the printer the internal representation uh didn't matter and As the 70s and 80s came along certainly 70s we needed some interoperability And so they standardized that character set But they standardized that character set Certainly in the west that did not represent anything and so if you look at this At this sheet basically what it's telling you is for the various characters There's some non printing characters white space Non printing characters and then here's some printing characters like the and key the zero And then the uppercase characters and then the lowercase characters and there's 128 of these possible values And there are nothing even for spanish or french in here And it's also why by the way uppercase lower letters in latin sort lower than lowercase letters And we saw that in some of the string stuff And what these do is it maps and says okay um And a lowercase a maps to the number integer number 97 which in base 16 is uh 61 and in octal it's 141 but in binary this is that it's eight bit numbers and so these are eight bits Otherwise known as a byte And they're very efficient like you know when you buy a buy a disk drive It's megabytes or gigabytes or whatever that's how many of these kind of characters that can store but unfortunately This doesn't work for more complex characters. You can figure out these numbers inside of python by By using the ord function And so you say what is the ordinal or the numeric representation of the uppercase h or case e And new line is a character as well And so like 10 is the ordinal position of new line And this actually has to do with sorting so that e lowercase e is higher than uppercase h And that's just because in the simplest of sorts. We just sort them numerically So new line if you go back to the previous little sheet new line is this 10 right here It's that 10 which is a line feed and that's a 10 and that's why when we print new line out we get a 10 And so again we in the early days when sim strings were simple. We just represented them as one byte per character, but the problem is is that You know as we have gotten more complex and in today's modern world It's simply unacceptable to say that the only thing computers can understand is ascii And so this leads to a very very From the simplest of character sets to a super complex character set called unicode which basically is billions of characters potential billions of characters For every language and every character set and because there's so much space in unicode It's easy to take very small variations of characters and give them a space. It's so large that you can have You you can have you can have pretty much any character that you want. So that's unicode The problem is is that if we sent unicode across the network It would be way too large. It'd be this utf 32 which instead of being Eight bytes per character would be four bytes per character And so it would take all of the data that we build and make it four times larger And and it'd be very difficult. And so what they've come up with is ways to compress this And utf 16 is this weird thing utf 32 is really sort of the full unicode pretty much utf 16 is a subset of unicode. It's it's it's used in some countries But the best practice for moving data across the internet or In a file that you're going to move between computers is what's called utf 8 And so what happens is is that utf 32 is fixed length ascii Ascii is one one byte utf 16 is two bytes utf 32 is four bytes and utf 8 is dynamically has dynamic length Meaning that it is one to four bytes And if it's only one byte long, it's perfectly compatible with ascii meaning that an ascii file is also utf 8 And so here's this little sheet and it's not critical that you understand this graph too much But basically as time passed 2000 internet's coming coming coming coming out 2014 Pretty much overwhelmingly the documents on the internet that you might retrieve our utf 8 Now so utf 8 is the recommended practice and it's sort of a compression utf 8 can represent All the things utf 32 can represent. It's just a compression of it so that with an overlap of ascii, which is Awesome. It's what you want I don't even talk anymore. So in python. We have always had sort of Two ways of representing strings in python 2 Uh, the normal string was a byte string was an ascii string was a latin string And if you wanted to represent unicode, there was a separate kind of object that we had and um, and so you would you would do that and um In python 3.0 Or later One of its main features of python 3 was to make unicode and string the same so that that means inside of python When you have a string variable It's a unicode Whereas inside of python 2. It was a byte variable and so Now we have this notion separately in python 2 and on python 3 Where we have byte variables and so byte variables are in effect an array of bytes So if there's through a b c that means it's three bytes. It's three bytes long Whereas a string might be three if three character string might be anywhere from three to 12 bytes long So python 2 had bytes and strings that were the same Bites and strings are the same and unicode is weird and in python 3 Strings and unicode are the same and bytes are weird Okay, and so that's that's what we've got to deal with and there'll be times when we get bytes From apis when we call things we have to then figure out what kind of thing those bytes contain because the bytes might contain Ascii they might contain utf 8. They might contain various things And so internally all the strings in python 3 or unicode Most of the time if you're inside the program or reading and writing files We just work and that's why we haven't mentioned it But now that we're talking over sockets and we're talking to the sort of random world out there We have to be a little more aware of the data we're dealing with now The good news is 98 of the time or 95 of the time It's utf 8 Which might also include ascii and so it's quite nice, but we have to we have to be aware of this And so if we are going to take data that comes off of the network in the bytes Then we have to make sure that we interpret it or decode it And in the right way so that internally the strings which are unicode are properly represented And so that's why when we read data in from a network connection like a socket We have to say hey decode it now. There's a couple things going on at that moment of decode And so this is where we're doing it We're see this we have to manage this in this code where we before we send this stuff We're going to encode it which takes a unicode string and turns it into utf 8 bytes There's actually parameter here that you could do a different than utf 8 But no one ever does you might have to for certain situations But so that says that we're going to encode this into utf 8 Before we send it and then when we get something back before we print it We're going to decode it and that's how this ends up working out And if you look at the documentation You will see that sometimes it says it's a string Or it's bytes and so So the you take a byte array and you decode it to get a string and you take a string And encode it to get a byte array and so that's what we're doing So you can think of the process as this way and that is The network has these utf 8 Mostly utf 8 resources not ascii If it's ascii it's okay So you read with the receive so this receive here pulls data Well, let's we have a unicode string. We're going to let's start with the send. So up here We have a unicode string. That's a unicode string even though there's no special characters in it No asian characters or french characters. That's a that's a that's a unicode string And before we can send it we have to send it in utf 8 if that was if that had asian characters It'd be okay because and that would be set up just right so that the utf 8 would be right So we encode it first and that's the cmd. This is now bytes Okay, cmd is bytes and then we actually send the bytes and that goes across the network We get back our thing and we receive and we receive into data. Well data is bytes Not string. It's bytes. We can say how big it is Functions kind of like a string and it has len except that it is one byte per character Which m