eans some of it might be utf 8 and then all we have to do is say decode again You could if you were dealing with a situation where you weren't expecting it to typically be utf 8 or ascii You could tell it utf 16 or something and it's more complex But the simple thing is do you just say i'm going to clean up my data on the way in I'm going to clean it up by running it through decode and i'm going to encode stuff on the way out And so sockets are the place where this comes into play And so you'll see we'll always do this encode and decode every time we're sending data kind of outside of python and inside of python So now that we've talked a little bit about character sets. We're going to make this even easier So you don't have to use sockets. We're gonna url lib is a bit of python Code in the library that does all this hockey stuff for you Okay, so now we're going to write a web browser again in python But it's going to even be shorter than what we did before we did it in 10 lines using sockets Now we're going to do it in four lines with url lib so url lib really Is just because the idea of opening a connection sending a get request sending the new line Retrieving the stuff breaking the headers out doing all this stuff That's so common. Why not put it in a library to save ourselves some effort So here's how we do it. We're going to read it in All right, we're going to import this library So it's not part we had to import sockets before but we're going to import url lib now And so this is really quite simple.  That's a library. That's a part of a live module within the library and this is a function So let's call url open and then give it the url Now that's a string which it's going to encode automatically for us So it's taking care of all kind of pretty things for us. It does the get it does the encode Look back at that previous code. That's kind of what url lib is doing for us. Okay Now what url lib also does is it makes the connection Encodes the get request and then it actually retrieves At this moment it retrieves all the headers and keeps them for you for later You can get the headers But we're not going to see the headers and it returns to you an object that looks pretty much like a file handle Because you can put this In the for clause after the end now it's going to read Run that loop one time for every line of this file And so the lines we're going to get back are bytes and so we have to say decode It doesn't do that for us automatically We are going to have to decode them and that's because we might need to decode them with a particular character set here And then we're going to do our strip and we're going to just print this out So that's just that's like open a file read through it and print it This is open a url read through and print it and that's as simple as it is And so that's what happens. This is romeo.txt and it prints out now the thing to notice is that There are no headers here. The headers have been sort of consumed in the url open Again, there is a way to say hey, give me my headers But for now this just going to eat the headers and keep them And then you get to read all the data and the loop runs and this loop runs four times and I'll come the four lines You can go ahead and run this one. It's super easy. I mean literally super easy And if you you can do anything you want, I mean treat it like a file You just have to remember to do the decode bit When you treat it like a file and so we that code import it. We're going to open it. We're going to make a dictionary We're going to loop through We're going to split it. We have to add the decode just to make sure because that line is bytes not string And then we're going to go, you know our words We're going to go through the line and then each line we're going to bounce through the words The inner for loop is bouncing through the words and then we're going to go to the next line And then we make ourselves and this a dictionary and we print that dictionary out now this is This in effect other than you know importing this opening it differently and doing the decode This is exactly how we would process a file and so by using url lib You really sort of reduce the complexity of retrieving and reading network resources To the same complexity of reading and dealing with a file locally on your hard drive, which is Kind of pretty so One of the things then we can do is read web pages. That was a text file, but you can get html And so here's how you read a web page and it's the same kind of code. We open a We open a url This one happens to have html on it and we read through it and out comes the html. Remember that the headers are there But they've been eaten by url open for us And now we could write a browser that would parse these less thans and greater thans and Make links etc etc etc So if you can come up with ways to find these links You could actually write A bit of code that would then have a loop that would go up and open a new one Pull out the links open a new one pull out the links open a new one And so you could you could make a thing that would retrieve a great program that would retrieve a page Find the links in the page and then retrieve those links and we'll actually do that before the end of the class And so python is a very popular language at google and I wonder if they're I'm gonna I think it's a pretty safe bet that the first crawler that they wrote just crawl the web to build the index was python because literally that's all it takes to read web pages and Pull those web pages into your web crawler database So I don't know are those the first four lines ever written to google? Who knows? So the next thing that we'll talk about is how you handle that html HTML is kind of yucky and nasty and so it's not as simple as regular expressions regular expressions might help Strength parsing and slit might help, but it's just too crazy So we'll talk a little bit about how to use a library to make html parsing a lot easier We are going to be talking about Some code if you want to download all the code. It's right here. It's all single big zip file and All this sample code the one i'm going to talk about is urlib 1.py it is Not very exciting. It's short That's what's kind of nice about python code And it's really if we go and take a look at the code we played with just previously which is socket The idea here is urlib is something that python has produced for us to make socket communications and http communications a lot better So socket what's this is making socket calls underneath it, but there's a library that makes this quite simple And so we have to do some imports so instead of importing socket We'll import these or we are going to create a handle you are able to request url open and just pass in a string So we're not encoding this we're not sending get command all the stuff we did in the previous Sockets example is gone and then we can just put this as a for loop And so we're not using this lower level read and write code We're just using a for loop and so that literally is going to read the text line by line And the line does come back as an array of bytes So we have to do a decode but then we got a string and then we can do a strip on it So this is like a super simple Uh super simple So there we go now the interesting thing is is you also don't see the headers We just read the contents now it turns out in url lib And we'll see this in later more complex application You can get the headers if you want you can get various other things So that's url lib a simple url lib tool Now we can also use this in url words to to to show you something quite interesting and that is If you look at this from right here other than the decode This is exactly the code we wrote to compute the words Right, so line other than this line dot decode. This is just a open something up in this case We're going to open a url. We're going to create a dictionary We're going to loop through each of the lines in that thing We're going to decode them and then split them. So once you do line dot decode This is now a legitimate internal python string. We split it We run through the words and run the counts and so this is exactly like code that we did before to run counts and so python 3 url words And so that gives us a dictionary which is the word frequency and we could do all kinds of crazy stuff in here You know with sorting and all the kinds of things the important thing is once you've done this in this The code other than the need to decode these lines when you first get them It really works just like makes a url. Lib Makes urls function inside python very much Like files. So these are start short and to the point and very simple and I hope that they were useful to you So now we're going to talk about what you would do with the web page once you've retrieved it in a python program Call this web scraping And so web scraping or web spidering is the act of retrieving a web page Extracting the links from those web page making a queue of un retrieved links and then moving on And eventually the idea is if you had enough time energy bandwidth and storage You could find your way to most of the web pages on the internet that are pointing that point Point to or are pointed to by other web pages And so you might have all kinds of reasons to scrape data. You might have a blog that you posted you might have um, who knows maybe you put some data in a system, maybe Maybe the system is being shut down because it's being Retired you can do all kinds of things. You could write a little thing Just talking to somebody who wrote a thing to retrieve something and check and then send a text when something changed All kinds of stuff or you might make yourself a search engine But be careful Not all websites are happy about you Using a robot to retrieve their content Some of the websites as we'll see you demand that you're logged in and they track what you do And if they think you're doing something bad, they will shut your account off Other websites will track what you're doing without you logging in but then shut your address off And uh, and so you have to be careful. You should read up. You should figure out what sites Allow you to scrape them now. I have some sites that I've set up that you can play with to make it so that it's a um legit so Parsing html is difficult you some of the simple examples You know, you could probably write a regular expression or You know, certainly some splitting and some whatever and what you would find is you would write that code And you're to retrieve your first five web pages and it would seem to work and then it would encounter some really weird But legitimate html Or maybe even sort of slightly broken html So the web is full of broken html and your browsers just look at it go like oh, wow more broken html But they don't put up error messages and so people just leave broken pages up But your python program is going to see those broken pages So what you would do is you'd be like, oh, here's a new weird way to do an anchor tag I'll change my code Oh and then run for another hundred pages and like, oh, no, here's a new weird way to do an anchor tag And the problem is is that you're going to find a lot of different ways to mess up an anchor tag And someone's already done that. There's a software called beautiful soup Um, and we have installation instructions on how to use it and really what it is is it's somebody just spent Months figuring out all the nasty things that could happen and compensated for it and gave you a nice Wrapped interface that just says look you give me the html and I'll give you back the tags, okay? And so it's called beautiful soup And so you have to install this there's a couple of ways that you can install this If you're good at extending your python you can just you know extend and install beautiful soup for all python programs If you can't change your your computer's configuration because you're on a school computer Or you're using a usb stick or something Then there's a way to download this file that i've created called bs4.zip And so what you do is you end up with your file called, you know url links Dot py and then a little folder called bs4 Which is a folder that has a bunch of files in it from the zip file And then you can run it and so it'll pull it in And you'll import from bs4 beautiful soup and that is either going to pull it in from the folder you do Or if you have installed it using the python installer It will also just you don't have to put this file in so it's up to you. You can either do it one or two ways So this is a little bit of code now beautiful soup is a complex Library and so just because this looks easy you the doing things in beautiful soup you might have to actually you know Read a bit more to figure it out, but we're going to just read this we're going to Import beautiful soup we're going to ask for url right here. We are going to take that url We're going to open it url open they give the url and read the whole thing That means we're not writing a loop. We've read the whole thing. That's okay as long as you know that the file's not so large And then we're going to pass the data we got back and this is going to be bytes But beautiful soup knows all about bytes and all about utf 8 and it figures that out and you just say hey Take that stuff. I just got and tear it apart using html and give me back An object a soup object now the soup object is something that you can run queries against so it parses it It deals with all the imperfections and inconsistencies in this this html byte array Um And it fixes that and gives that back and so there's various things you can do and you got to go look at the beautiful soup documentation It could be a whole class on beautiful soup. So here's the thing you can do is this object You can sort of call it like a function And say hey give me back the anchor tags and anchor tags, of course are the tags that say href equals Blah blah blah slash a So all of this is an anchor tag And then we're going to loop through the tags because there could be more than one of those anchor tags in the file And then we're going to pull out that href and that's what this does We're going to loop through all the tags and print out the href So if you tell it to go to dr. Chuck com it will show you the one external link in dr. Chuck com And so i've got an assignment that sort of goes into that in some more detail But this chapter has been A whole bunch of interesting stuff We started with a tcp ip model and talked about sockets that are phone calls between computers And then how applications Protocols are developed to say what we say on those phone calls and we've explored then the http protocol Which is probably the most likely thing you're going to see and then we played with all this in python And saw that a python is really good at this You can write extremely simple and small programs to do some extremely complex and powerful things And again, that's why people like python is because it makes the complex simple We're going to do a little bit of sample code if you're interested in getting the sample code You can download this zip here at python for everybody dot com materials dot php And you will download and you will get all the files Um and all the files that i'm looking at here And so the one i'm going to play with today is the file called url links dot py So the first thing you got to do before url links dot py works Is you have got to install beautiful soup and i've got some simple instructions at the beginning of the file And so one way to do it is install it using Python install process to install this beautiful soup for all python applications And if you are the owner of your computer and you're going to use beautiful soup a lot It's a fine idea to do that But i want to show you a simpler way that if you don't own your own computer and you just want to make it so that beautiful soup works Um, you can download this file Um This file right here Beautiful soup four dot zip unzip it and put it in the same folder As here and so if you look in this folder I have a subfolder called bs4 And that's the unzipped version of this and it has these things. I didn't write this code So i'm sorry if the name is bad, but the this is the code to bs4 And this is what's in bs4 dot zip and it's in the same folder as url links dot py And so what happens is when you do this from bs4 important beautiful soup that either can go to sort of this Global magic place that python install stuff and pulls in the beautiful soup object Or it can go to the folder bs4 and pull it in Okay, and so that's how that works So you have to do one of these two things I prefer to keep it simple download and unzip this file and put it in the same folder as um as this code and away you go So from the previous example, we're going to use url lib, of course And then we're going to pull in the beautiful soup from the beautiful soup for our library We're going to get the beautiful soup object now If you do this with ssl if these websites we're going to play with have ssl You pretty much have to do this little pack And these three lines don't worry too much about it the whole idea You can do google on stack overflow and figure this out But this is the way that you ignore errors when you have ssl certificate errors Um And so we have to add this parameter context equal ctx, which is this variable that we create So this part and this part sort of just do them if you don't you can take them out actually Um, but otherwise you won't be able to do htps site. So let's take a look at what we're doing Other than height uh dealing with the htps problem um Going to ask the user for url. We are going to Um Retrieve all the html. We're going to do a url open just like we did before now This would return us something we could loop through line by line with a for loop But instead we're going to say hey read the whole thing and that basically returns us the entire document at that web page um In a single big string with new lines at the end of each line And this is not an unicode, but it's probably utf8 string But it turns out beauty beautiful soup knows how to deal with utf8 and it also knows how to deal with unicode strings So what we're saying is beautiful soup read through and deal with all the nasty bits, right? So html is like very very flexible. So drchuck.com page one htm And so if we take a look at the source of this view page source Make this bigger You know, you might be able to do regular expressions, but do it does things like break stuff across lines There could be a line break here. There could be all kinds of things, right? And so writing, you know regular expressions or splits or whatever is really hard for html And so what we do is we someone has written this it's called beautiful soup and it's basically This is the code and it's it's based on a joke from a children's story. Um It it basically someone has just went through and figured all the bad things that could possibly happen when you're reading and parsing Uh html. So either you use it or you will slowly but surely derive all the things that it doesn't work and so When we look at this line right here This line at a high level is saying We're giving you ugly nasty html that could make no sense whatsoever Please read it and have all the brains that you have And all the weird stuff figure that out for us and give us back an object I happen to call it soup. You don't have to call it soup An object and that is a proxy for that html But this soup object is clean And so what we can do is we can sort of retrieve all the anchor tags So we can talk to this object and say ask it give me the anchor tags. What's an anchor tag? Well, if we take a look at this source the anchor tag is the a through the slash a that is the tag It is the tag it is attributes that are on the tag It is the text within the tag and everything and so that's what we're going to get now I call it tags plural not because plural matters at all But because we're going to get a list of tags because even though this web page has lots and lots of tags If we look at say dr. Chuck calm And view source. Well, that's kind of small View page source right And we go look for a Anchor tags We got 45 of them and they all kind of have weird stuff in them, right? So this line will give us back a list of tags It will give us all the tags in this document So it goes the tag goes from there to there And then we're going to do is going to write a loop to loop through all the tags So that's basically hopping like it's hopping through the document sort of like this. That's what it's doing hop hop hop hop hop Hop And it's pulling out the text of the hreth attribute. So it's going to talk pull out this bit right here Whoops. Oh darn. That was so cool because that's a flaw. Look at that. This is my own page There is no closing quote here But it's going to work because html soup is like oh, I know what to do about that I can deal with that So let's check to see if that one works because that's like a mistake But that's one of the things we like about beautiful soup So we're going to read through and then we're going to pull out all the hreth. So This is probably thousands of lines of code that you really don't want to run. So python3 url links dot py And so let's start with a simple one http colon slash slash www.doctorchuck.com And it reads it. Oh, that's the no, that's That's actually the card one because we got a whole bunch. Let's see if sugi see the sugi one worked It found that one It's right after Sakai project.org. Where is that? Is there another sugi? Oh, no, I didn't find that one. That's kind of funky. Look it found it wrong But that's okay. So you see it found all these and did a lot of nice stuff for us if we do it python3 url links Dot py and do the easy one htp colon slash slash dub dub dub doctor dash Chuck calm page one dot htm We will only see one And there we go now The the ssl is if you are looking at a page that has a ssl python url links too.  And so you'll see If it wasn't for that So all kinds of stuff coming back And if it wasn't for this bit right here and this bit right here This https wouldn't have worked and it's not that that website had a bad url It has um, it has a certificate that's not in python's official list Um, and so the the url is okay So that gives you a quick summary of Using the beautiful soup library in python along with the url lib Hello, and welcome to chapter 13 web services. So what we've been doing so far is we've been using the request response cycle We've learned about sockets. We've learned about url lib And we've actually learned how to pull html and even flat text off the internet But what we're going to talk about now is using that same request response cycle to to retrieve information that is specifically designed For programmatic consumption so that you know, we had to have this beautiful soup which sort of did a hack job or Hard solve the hard problem of parsing html But why not produce data in a format that makes good sense to a program because programs want to talk to each other If you recall the whole idea of a socket is to have one application process sending data to another application process And so if we if we think about this for a moment And we realize that we have all these programs They could be written in different programming languages and they're all connected And so they might want to send data back and forth or through the network php programs javascript programs java programs And so we have to decide on a protocol that is independent of any programming language And then we call that the wire protocol because if you were to sort of take some connection and watch The exact characters that go back and forth. That's what you would see if you are monitoring the wire So that's what we call that the wire protocol And so the idea is is that we have to agree on a format that is going to represent the data And we can't make it a python specific format or a java format And when we take the data from the internal representation, maybe a python dictionary to send it to the wire We call that act serialization and that is going from sort of the internal representation To the serial representation or the wire representation And then here is an example of a person with a name and phone number With using less thans and greater thans This is an xml example and then in a far end in a different programming language It receives this and then deserializes it and then turns it into some useful structure Inside that programming language and so this is an example of a wire protocol that's using xml And that's one of the formats we're going to talk about Another format that we're going to talk about is a format called json javascript object notation And it is simpler and easier, but it's not as Precise and descriptive as xml is and so while you'll find that most of the things you run into Especially if you're talking to apis of one form or another You'll find that json is very common xml still holds sway in places like documents So if you look at doc x at the end of a microsoft word document Doc x means that it's an xml version of the representation of a word processing document So the first thing we'll talk about is xml So one of the two ways that we mark up data is xml the other of json first we'll talk about xml We'll talk about xml more for a longer time when we talk about json xml stands for extensible markup language There was a number of markup languages in the 90s that were out there ways to send data between computers And none of them was like amazingly better than the other but in the late 1999 early 1990s as html came out The idea that we could use less thans and greater thans, you know or angle brackets some people call them Um Once html made angle brackets popular as representation format It was pretty natural that we would find a data representation format that would take a similar approach and So inside xml we're going to talk about tags we're going to talk about attributes We're going to talk about data and we've already talked about talked about serialization and deserialization Serialization is the act of taking data inside of a computer in one programming language setting it up for transport transporting it across and then Taking it back apart and turning it back into the data in whatever internal data it needs to be In the destination system. So here's some basic xml So we can take a look at the various things that make up the xml So it's very much like html and that we have tags less than greater than the differences We get to name the tags anything we want rather than the a tag or the p tag or the h1 tag And there is a beginning tag and the ending tag and they're bracketed together and there's syntax errors in xml Syntax errors in xml are more severe than syntax errors in html It's supposed to be right and if you send bad xml, it's likely that the far end will not understand it So we have a beginning tag an ending tag and so like name and slash name or a beginning and ending pair Then there is the actual textual content and that is the material between it And then here's a phone and slash phone and we have this thing called the attribute key equals value The key doesn't have double quotes. The value always has double quotes And this is this is like href equals on On an anchor tag and sometimes you have what's called a self closing tag where you don't actually have a closing tag You have all the data that you need In the attributes and so you don't even bother putting an empty text area in in a closing tag So that is a start tag an end tag Attribute and then a self closing tag. Those are some basics of xml In general xml doesn't care too much about white space It does in the text areas So in here it matters and here it matters But things like we can indent this a little bit differently And we tend to indent it in a way to make it look reasonable Although once you have programs sending it back and forth they tend to send it More compacted just for efficiency purposes So one of the concepts is that there is a hierarchical structure within an xml document And there are parent nodes and child nodes And you can think of these as simple nodes that is a tag in some data Or a complex element that has a tag that includes other tags some child tags And there's a couple of different ways we can take a look at this The the simple and more natural way to think about this is a tree with parent child relationships So here we have this a tag on the outside and that's the top level one You can only have one outer tag and you can only you can't have you know another tag down here So you have to have one tag that's sort of the root tag for everything in this xml document And it has two children so that the c tag and the b tag are two children So the b tag is a child of a and then c has a d and an e tag that are children there And then the the textual data we model as a as a child Of each of those tags and so that and you'll see in a bit why it's best to do that So that is the way to think about this as a tree to represent that xml As a tree if we add attributes to it and this is where you kind of see why it's nice to take The text area and make that be a child of the node an attribute is a different So the text is a special kind of child and you can literally have more than one attributes You could have x equals Two you know zap equals whatever and these could have a couple of different attributes The w attribute is a value of five and that's the five down there And so you could have multiple ones. You can only have one text node Now in the case of a you have a whole bunch of text nodes But these are because there are child nodes Within one simple node. You can only have one text element You can also think of xml as paths and the easiest way is to sort of look down this tree version And look at from the path from the parent So you go to a then the parent of the child b and then x so at position ab you find x So ab is the path up to the root So acd that's this one Is the path to y And a c e is the path to z and so you can think of these as paths Part of what we're doing is we're coming up with ways to Walk through and parse trees of xml data So the next thing we'll talk about is how we determine if a particular xml document is legal or Meets the contracts the two applications have set up We're going to do a little bit of code if you want to get your hands on the code go to the materials Website materials dot php Actually materials dot php and download the sample code The code that we're going to work on today is the xml code and we need to be able to talk xml To work with web services. And so here's one of the examples from the book. It's xml one dot py And so later we'll be pulling xml and json from the web But for now, we're just going to put it in a triple coded string So data And we're going to use a built-in xml parser in python called element tree And when we say import xml e tree element tree as et this as et gives us basically a A shortcut handle for it And so the idea this is a string it has less sends and greater thans It looks like structured information and it is but really at this point It's only a string now. We have to call this et from string to read this and give us back a tree object And what it does is this might blow up this code might blow up right here if there was a mistake in it Okay, and um matter of fact, I can probably put a mistake in let's see if I can delete this And save it and run this code and we'll see that it will blow up Right and so it blew up here in line eight element tree blew up It are I mean it blew up in line 12 of the code which is right here This failed because the line eight of the xml string was wrong. So let's put this slide that back in So now it's properly formed xml. So this tree we get back I name it tree just because I always name it tree, but you could name it x So the key is is tree dot fine goes and looks for a tag named find and it tree is no longer got less Then's and greater than's in it. It is went and and turn these into objects within objects within objects So tree fine name says I would like to find the tag Name and that's what this bit is right here And then dot tx dot text is going within that and grabbing that text, okay And if we say tree find dot email, then that's going to give us this And then that's that object and then dot get Asks for the contents of the hide attribute, which is the string. Yes Okay, and so if we run this Now that it's fixed python 3 xml one dot py It will pull in and get the at the name And the attribute so it's it pulled the chuck out and so you get this object and then you kind of dive into that object and so that's xml 1 dot py it if you've got a tag You can either get the text out of the tag or you can get an attribute out of the tag So now let's take a look at xml 2 dot py again. We import element tree And we have a tag and there's xml's always got to have a single outer tag But this time we're going to have an effect a list Um, now let's let's line this up a little better There we go. That looks a little prettier And so users not the fact that it's users doesn't mean anything but we often Come up with semantically meaningful names for these things Users is going to have as its children a list of user tags Okay, so the children under user users user under user and then this has each of these as a tag So we want to parse this and this is a common thing we want to do Um You know and so again the first thing we do is we read the string to just take this It's a triple coded string going from here to here And then we're going to instead of doing fine, which gives us one tag We're going to do find all the users tag the user tag that is a child of users And we get back a python list of the tags not of the text, but of the tag So there's a one tag and there is another tag and so we can do len of that So we can see that we got two and then we can write a for for loop And the this item is going to iterate through the tags that are the user tags that are children of users So the first time item is going to be this tag a tag remember And then the second time is going to be this tag And so we can do things like find and and get just like we did with the in xml one So running this is not too exciting python 3 xml 2.py You see that there are two users that comes from This print right here. There are two users in there and the first one if we go into name And we go find the text within the name tag Within user Then we get chuck and then we get the id which is 001 So we find the id within that item and then we get the text and then we look and we grab the x attribute Off of that and so we see chuck Chuck 001 and 2 and then in the next tag we the for loop continues and we print that out Okay, and so that's just a basic run through of the the xml from The the chapter in the python book. Okay. Thanks So now we're going to talk a little bit about xml schema xml schema is a language That allows you to decide on whether or not a particular xml document meets a contract in arrangement So you have two pieces of software exchanging data using xml And what if one of them if they're all working nobody really worries too much about it But if all of a sudden one breaks you change one side another one breaks whose fault was it, right? Was it the side that got changed or the other side? And so you could argue So what you like to do is before you set up these arrangements between these applications set up a contract In a way, they're kind of like the rfcs are Except that their scope is between pairs of applications and so that is it itself is xml and It it basically what we do is we we take an xml document and an xml schema contract And then we either say that's good or that that is bad and that's called validation A piece of software that validates xml when given a schema is called a validator. And so an xml document Here we have our little xml document. We're passing it to the validator and then we have a schema contract which is a Itself xml. It's kind of a particular kind of xml that xs colon complex type That's just a tag colon is a legitimate character for the name of a tag Name equals person. That's just an attribute. And so xml schema is a particular format of xml that a lot that That renders an opinion about what xml is supposed to look like So there's a number of different xml schema languages the one we're going to look at As one that's kind of came a little bit later. That's very common Called xsd, which is the worldwide web consortium's schema specification Often you'll find files that have suffixes of dot xsd that actually contain The xml just like we're going to show you So if you recall, there are simple elements which have text children And then there are complex elements where other other nodes are children of other nodes And so we can say this and so here we have a little bit of xml and the xml schema that makes sense with that So um, what we're saying is the outer tag of this legitimate xml is supposed to be a complex tag With a name of person. And so there we go. That looks good. Good good Then there is a sequence And then there is a simple element The name of last name looks good and it's a string that looks good Uh, uh another tag that's of in it of type of a named age. That's of type integer. That's good and then a Thing that's called date born and then there it looks like a date So we check all these things and we can basically say yep You know that is a good xml document according to this schema and You don't have to write this generally, but there is software that reads these two things and comes back with a true Or a false and not even have some detail as to what went wrong With this particular schema Here's some more that you can do with a schema We can do things like, you know, have a complex type. We have a sequence here We have a string full name and a string child name But we have this min occurs and max occurs. So min occurs is the minimum number of times it can occur And maximum is the maximum. So min occurs equals one max occurs equals one means it's required And so this is required and we don't have two of them two of them would be an error One of them is fine. So that's good here. The child name is Min occurs zero max occurs 10. So we have four here And so that's good too. And so that is another kind of xml schema constraint that you can have Here's a few other data types that we can do We've done the string we've done the date the date looks like this dates are four-digit year two-digit month two-digit Day with dashes now, there's lots of different ways to represent dates But the nice thing about this and you don't you have to put the zeros in so zero nine for september It means that these are sortable as strings so that if you do all your dates this way They're sortable as strings. So you could argue what is prettier, but for computers We don't worry about that. We're arguing about what's the most functional And then the date time Is that same date format with zeros followed by the letter t letter t and then followed by Hours minute seconds zero filled right so nine o'clock is Zero nine and then the time zone which we'll talk about a second in the next slide You can have decimal numbers and you can have integer numbers as well And so we are able to sort of render an opinion as to what is good and what is bad in the resulting xml So dates are kind of interesting. There's again, we have lots of different formats of dates, you know Nine ten, you know nine slash ten slash two thousand and two right? You know, that's that's a format of a date, but that's that's one. There's another another format of the date, which is, you know 12 december Two zero whatever and so this is how people show dates Computers don't want to have all those different dates and don't want to figure those out They have libraries that produce dates and make them look pretty and for particular locales, but Computers really want dates that work best for them. So we just say, okay, we're going to have this year month day Time and then zero fill hours minutes seconds h m s and then time zone Now computers even prefer a time zone Uh, if you I don't know if you've used something like your google calendar And you take a flight or take a train trip and you want to put a different time zone everything switches And that's because google calendar is not really storing the time zone That you're it's not storing the dates Related in your current time zone. It's storing them in what we call universal time or greenwich mean time Zulu time is another word for that and z means this time that is the time in You know, london england greenwich mean time um And so the thing is is that that means if this data moves between time zones or crosses international dateline Or standard data like savings time or anything like that None of that changes and so so we have this internal Date and time that's very common in situations where computers are exchanging data That then gets shown with a time zone converted to the time zone or the local format That's the right way to do that and there's a standard for how dates and times are supposed to look So here's another little example of uh, some some stuff. Let's see what we got now If you see this little question mark xml That's not a problem. That just is a way of sort of putting a header on the whole document that says it's an xml document Telling it that's a utf 8 document Um, and that that's not really a tag that's sort of like a marker on the file So that you can put that there, but it doesn't harm the xml The outer tag is this tag right here excess debt polling schema and then, um What else we got? We got an address. We've got a string string string string string. We've seen all those Here we have country and we're going to have a restriction that basically says this is a simple string But we're going to make it so that you have to list One of these four as the country code and so here we are down here and that's uk and that's uk and so that is A valid xml Another couple of examples here Uh, let's see a string string string string string Uh max curves unbounded. That means infinite number. There's no limit on the number. You can do that Um in occurs of zero excess positive integer. We've we've seen integer, but you can also say it's got to be a positive integer A decimal we've seen that and then u sequels required is just another statement that you could make I'm not trying to get you Uh to the point where you can do xml schema just get you a sense of the kinds of statements that we can uh speak about when we're talking about what is and is not legitimate xml So let's talk a little bit about how we might talk xml inside python And so like most things that are in this extended part of python We have to import something and so this is the name of a library xml e tree element tree And then as et this ends up being a shortcut So we don't have to type these long things and so et is the same as typing that it's almost like a macro Now normally this xml is going to come somewhere from the network, but i'm just going to put this in a string I'm using a triple quoted string And so that means that this triple quoted string starts here and ends here And all these new lines that are here are actually part of the string So this is kind of like I opened a file and read the whole thing in but just to keep this Totally self-contained. I'm putting it in a string. So the xml would come from Some server on the other side of the network. We would get this xml. So that's that's how it would normally work. Okay, so This is the xml right there and We parse a string of data And we call et from string So we're passing in the less thans the new lines the greater thans all of this stuff We're passing in and this could have syntax errors in it. So this might blow up If this had a syntax error like we forgot the little slash or something there was a syntax error But this doesn't have a syntax error. So then what we do is we get back an object I just happen to call it tree because it kind of is like that tree version of the xml That is an object that we can then query to pull data out of it. So we say tree dot find And look for a tag name name So that finds the tag name name is this it's it's everything It's the tag and the text if we want the text we add dot text and then that dot text That dot text that actually Refines it to only the word chuck And similarly if we do tree dot find email That tree dot find email That finds the email tag, which is this tag it has a child attribute and you can get and any of the attributes you say dot get There's only one text child, but there are many Attribute children and so you have to tell it which one you want and so this said this this here This bit right here all of that will reserve will resolve down to that string. Yes That's what you're going to get there. Yes and so you You kind of build up these little finds and call methods This is not a clearly a full introduction to element tree But you get the idea that you sort of dive down in With these methods the call methods the call methods to get little pieces out and parse all of that Here is a different example In this one again, we're using triple quoted string We always have a single tag on the outside And then I have a complex type of users and in it there are two user objects. So this is kind of like a list Right. So this is more than one of these things. So this user can occur more than one time And again We take this we pass that into from string and get back an object that represents The name stuff is not necessarily have to represent be the same as this outer tag Just a variable this could just just as be easily as x if I wanted So now what I'm going to say is hey, hey stuff I want to find the tag the path user slash user I want to find all tags that match user slash user. So that's going to give me a list of two tags one tag Two tags in a list Tag Tag Oops So two tags Now I can print out how many I get that'll be two in this case because I got two tags and then I can actually iterate through the list All right, so I can I can iterate through the list So this item is going to iterate first to this tag And that tag now it's like it's like in the previous example We can look for the name tag within there and pull the text out So we pull that text out find the name tag find the name tag and then within that find the text and we can find the id tag and pull the text of that out so that pulls out this zero zero one and I've scribbled too much And then we can item which is this is item is that whole tag dot get x so that gets the attribute that gets the two that two comes down here okay, and then Item goes to the next one because item is looping through so item iterates down to that one and pulls out The name dot text the id dot text and the attribute dot x and pulls all those pieces out So this is the basic pattern you saw one where you just you're tearing into a single Um a single thing and here you're tearing into something that is expected to occur More than one time and so that's a quick summary of how you talk to xml in Python up next we're going to talk about the other serialization format JavaScript object notation So now we're going to talk about the other serialization format javascript object notation Chances are good as you go out there You will very likely encounter more json than you will xml not that xml is bad xml is better for rich and hierarchical documents whereas json is best for Just pulling data out of a system and moving it between two systems with the minimum of fuss This is douglas crockford. I have a great interview from him. He's a funny guy very very smart He claims he didn't invent json. He discovered it because it really is based on The literal notation for javascript and it actually looks a lot like the python literal notation for objects and for lists Now douglas crockford is a quite a sense of humor He wrote this book called javascript the good parts That's the little one right there and then javascript the comprehensive guide in the the sense of humor is all the stuff That's in javascript. That's not too useful and while this is sort of a tongue in cheek It also is trying to say that javascript What crockford is really saying here is javascript is a great language As long as you avoid the tricky bits and sort of keep it very very simple And javascript is indeed a great language, but but json comes from javascript You can read about json at json.org. Uh json is not a international standard. It's not like an rfc It really is douglas crockford decided to register json.org and typed in some pages and people started reading it And people started using it and partly that was because it was truly derived from the javascript javascript literal syntax So we're all ready to code. Here is some python that's going to process some json Keep it straight python process json. So again, I'm using the triple quoted string here Now you'll notice the syntax that we are using is not angle brackets, but instead curly braces And so the curly brace and then within the curly brace you have key value pairs name colon chuck and the Key colon value and both sides have quotes You can also have objects within objects curly brace key value pairs key value key value looks a lot like python And then you can do this and so this is a structure that has one key value pair That's a string another key value pair that's an object another key value pair That's an object and then these are key values within those contained objects. So this is a string That again probably was retrieved across the network from some other place And we're going to pass that string into You know the json library load s load s stands for load from string So it reads this parses it looks at all the white space white space again Doesn't matter too much here unless it's in between double quotes the white space doesn't matter And so it parses it and then returns us a Dictionary so the thing that's different about json is that its Structure and representation are simpler than xml So in in python everything either comes back as a dictionary or a list or a dictionary within a dictionary or a list within a dictionary But it's all dictionaries. It's not a separate structure that you have to do gets and finds and find alls and look ups So it's right there. So when we get this back because This is a curly brace info is a dictionary And so we can just use the standard syntax of python info sub name. Well, that will bring Let's clear this So info sub name will will go find chuck So if you compare that with the xml, that's just a lot easier Now when we have info sub email, that's this thing. So info sub email is that thing and then sub hide Is this so that's what comes out here Okay, so it's really nested dictionaries and lists We haven't seen a list yet, but this is a set of nested dictionaries that it parses And it's equally simple in other programming languages This is a little more complex version where the outer element is A square bracket, which means it's going to be a list And so we have a list of one comma two things. So this is a list Of two dictionaries So there's two dictionaries inside that list So again, we take this string and we load it into You use the json parser to read the string and give us back In this case info is a list. It's got two items If we print out info, it'll give us two and we're going to iterate through And so if we're going to iterate through Item is going to is going to first be this And then it's going to iterate to this And it's going to print out item sub name, which is going to print out chuck Item sub id, which is going to print out uh, 001 Now you'll notice that there is no attributes and that's because json is simpler But we can have the x and it just is another item. So we say item sub x and that's going to print the two out And then it'll iterate to the next one and it'll print out the same thing for those guys and so Json is simpler because it is you can't represent as complex a data structure Or you have to compromise and map it into a simpler data structure But then it is lists and dictionaries and so once you've got it parsed It is easier to understand and to make use of So that was quick. So that's part of why everyone likes json better Is once you have come up with a format that you're going to send it back and forth It's easy to make it and it's easy to read it Now what we're going to talk about is sort of moving up a level if you've got all these data formats and URLs that you can hit to pull those data formats down What Approach do you do as you start to construct applications that increasingly go from a single application to a networked application? we're playing with the web services chapter right now and If you want to get the materials for this course you can Go here and download the sample zip sample code zip. I've got this all sitting already on my computer I also have the whole thing in github if you want to get it out of github So the thing we're talking about now is we're talking about the json 1.py example from the book And so json is kind of like xml except a lot simpler And that's why a lot of people like it. It's not that json is always better But json is is better in a lot of situations that don't require the complexity of xml So we always we start to import json json is built into python But we have to ask to import it again. We're using a triple quoted string to put the json in there And json looks a lot like python dictionaries key value pairs key value pairs in this case This is a key and the value itself is another dictionary or in json terms an object But again key value pairs within key value pairs within key value pairs And all these little cursor guys have to all these little curly brace guys have to line up properly And so uh like all the time this is Uh a string which we normally would read and decode from the internet But for now we're just going to have it in there load json.loads says go into the json library pull out load string and parse this which turns this set of curly braces spaces commas and perhaps syntax errors into a structured object And if we had made a syntax error in here then this would blow up But if this doesn't make a syntax error if this doesn't blow up then we have a structured representation Now the difference between xml and python json is that this turns into a python dictionary with key value pairs Okay, and so once we have this this is a dictionary and we can say Info sub name and that's the exact syntax that we would use to get the dictionary and that's going to extract this value out of there And if we want to go in deeper We can say info sub email and that's what info sub email is right there and then sub hide So that's like that's a dictionary within a dictionary. So if we run this python three json One dot py It digs in really fast. And so this is why people tend to like json is because You read the json, which is actually a syntax derived from javascript But it looks just like the syntax for a python So it's moving and that's moving an object a json object that turns in directly into a python dictionary with nested dictionaries So now we're going to look at json 2 And so json 2 we're going to see a list Are an array in json terms, but it turns into a list in python terms. So this is a list Of dictionaries in javascript. That would be an array of objects, but in python. It's a list of dictionaries So we'll just pretend that it's a list of dictionaries again We load the string parsing looking for syntax errors. So let's just make a syntax error here and run python json 2.py and you'll see where it blows up It blows up at line 15 Which is right here. It's like this low desk blows up now You could put a tricep around it to save it, but we're not going to do that And it even complains it says look we're expecting something here in line 11 and that's line 11 of the json which starts at line 4 And so I'll put my little square brace back in so it's not syntactically broken So let's run it again and make sure that she runs and yes, she does so this parses it And converts from the json syntax into a python in this case list Because it's got square braces instead of curly braces the previous example had square braces And we can then take a len of it. It's a and it's an array. It's a list And we see that there are two things in there and then we're going to iterate through and this item is going to iterate through These dictionaries that dictionary followed by that dictionary. So the first time it's item sub Name, which is this value right here and then item sub id, which is this value So you can dig right into this, but you're using you're not using get and you're not using the weird extra find or find all or anything You just are going at these structures directly and so you can quickly extract this stuff out and we Read through id use Name is chuck Name is chuck The no, there are no attributes by the way x is two and so we had to make x so if you look at the xml We had this concept of attributes on the outer tag there. These things are also not named We just have to know what we're looking for. So json represents simple structures, but it's all it's much simpler to use so I hope this has been useful to you And talked to you in a bit about some more json So the service-oriented approach is a way we Approach solving a complex application problem where all the data really isn't present in one computer system It's somehow spread out Over the internet connected by the internet or internal network And so the the idea is is that Some applications just can't contain everything the the perfect example is a travel website That can book you a flight book you a car Buy tickets Book you a hotel and do all these things Well that travel website is neither a hotel nor a rental car company nor an airline But what it really does is it talks to all these services somewhere else on the web on your behalf And it makes reservations for you. And so you have this convenient user interface that says, oh, here's your whole vacation I'm going to figure all this stuff out Now you say go and it goes book book book and books on all these other other systems Now it requires a lot of infrastructure a lot of coordination and a lot of Effort to make sure that your application can talk and these other services that are out there in the internet Have good contracts and you know exactly how to send data to them and get data back from them And so initially when you're building a service-oriented architecture often you have one application and it's all internal Often it's all one language and then maybe you'll say, oh, wait a sec We want to take part of what we do and put it in a second system And then sort of come up with a set of rules between the systems and then more and more and more So now that we're solving our problem using a series of cooperating applications Communicating across the network. We're going to talk a little bit more detail about the notion of what we call web services And in this we're going to take a different perspective instead of building our application and breaking it into pieces We're going to have an application that's going to really consume an api from somebody else So there is some other provider of this api. That's not us And so that if you're going to talk to somebody's data like google or amazon Or twitter They're going to say you have to use our api. So what's that? So an api is a contract that says look if you do this and this and this and this We're going to give you data this way and they set the rules They tell you what the URLs are they'll tell you if it's xml or json And this is called the application program interface and it's something you read and you understand And so you go look at the documentation. This is the documentation for the google maps api So it turns out that google knows a lot about maps It knows a lot of data it knows how to search maps And it actually provides some of those features to you that your application can take advantage of I took advantage of this at one point by Asking all the students in one section of one of my online courses where they were from And I just let them type in where it was and then I said well I don't know how to code any of that So I use this api doing what's called geo coding to look all those places up and get precise Latitudes and longitudes for the ones google could figure out and that saved me a lot of work now These are expensive resources, but I could be patient and make use of these resources which As long as you use them, not too much. They can be free We'll talk a little bit more about rate limiting and what's free and what's not in a bit But you start by reading documentation and it says do this hit this url hit that url So if you read that documentation You will find that There is a url that you can hit and they tell you where to go And then you go to this url You had a question mark and then you say address equals and then an hour plus and there's all these rules These are called url encoding rules when you have key values on urls The plus means space and percent to c means comma. So these are called url encoded But don't worry too much about that because we're going to have a magic library like we always do in python That takes care of this and so if you were to hit this url You type it in the exact right way in your browser. You will get back adjacent document It's an object that has key value pairs The first value is this status then it has these results and it's a list and you dive down and eventually you can kind of find the latitude Longitude of the thing that you are looking for and so the idea is can we write a program that can read this And so here's our little program that reads this and a lot of this is sort of Comfortable you've already seen some of this You import url lib We have to parse them json. We grab the url And then we're going to write a little while loop that's going to ask for a location and we can type that location in And we've got to concatenate With this url the location equals and there is a bit of code a library that called parse url and code That takes the key and the value So the address equals and then whatever this text is that we read in from the user that goes in here And it does that url encoding with the pluses and the percent 2c And all that stuff is taken care of and that is our url that we're going to pass To url open so we print out that we're going to retrieve it Prince this out and if you look at this it's too long. It has all that fancy stuff on it And then we read it I mean we open it with url open And then we read it and decode it. So these two things hit this url Decode it and then we retrieved 16 69 characters because it's just a In this case because we've decoded it data is a string now that's read as bytes and data is a string So we read uh that many characters 16 69 characters And then we're going to take this data and we're going to parse it with json And we might get bad data here. It might blow up, but it might work. And so in this case it works Um, we have an error that basically says if we got a bad thing we're going to blow up But in this case it doesn't blow up and so now we're going to sort of dig through and If you go back, let's let me just go back So the results sub zero geometry. Let's show you how that works So results is the first key So this is a dictionary with a key of results, but then it has a list and the zero item This list starts here and goes there and there's i'm only going to show part of it, but there's many things here So the zero item is this this is the sub zero And then geometry within that sub zero item. So if we look at that it is The outer outer dictionary the first item in the list sub geometry. So that grabs one part That grabs This part right here And then we're going to go into location and lat and those are just keys within keys a dictionary within a dictionary And so you see it says Sublocation sub lat and so that is literally going to pull out of that complex structure That will pull the latitude out and then in the next line pull the longitude out So we can pull the latitude and longitude out and then we print it out We can go into results of zero formatted address and that goes into results zero Formatted address and that pulls this little bit out Now it takes a little while write this stuff and you have to put a lot of debug And you don't necessarily figure out this complex bit here at the end But you know you print it you don't get what you want you say oh wait a sec That was an array so I got to add a little sub zero there to get the first one out of the array But eventually you figure it out and it's not all that difficult It's the first time first few times you do it and like what am I doing? But after a while you realize oh, I'm just sort of tearing this apart and digging deeper and deeper into This data structure, which I just retrieved over the internet from google and I learned something good from that So up next we're going to talk about how sometimes these apis protect themselves with keys or signatures And why that happens and how to solve those problems We are doing some code samples here if you want to follow along you can download the sample code all is the in a big zip file I've got it. We are going to be working with the google maps api Uh in the old days this maps api was free and did 2,500 requests per per day But now they've made it so that parts of it are Behind api keys and you start having using o-auth and stuff But not they haven't put it all behind this one address service that we've been using that continues to work And the basically idea of an api is you go read the documentation you find A url and this is going to google servers and you pass in The address and and we have to pass in the address using what's called url encoding. So spaces are pluses That's a comma and then that's a space and so we have to pass this in a certain way But if we do it right We hit this we're going to get ourselves some json back and that's really cool And so deep inside here we get the real address, you know a good address we get a geometry Um You know we have the location we got the latitude and longitude and we can extract stuff out of here And so we're talking and this one here is still rate limited to 2,500 But it's one of the few parts of the google maps api That is not hidden behind an api key in a later chapter We'll show you how to actually talk with the api key in the geo data code The geo load uh shows you how to use an api key if you uh if you want to jump ahead and take a look at that But for now, we're just going to take a look at geo json, which is going to retrieve one page and tear it apart. So let's take a look So we're going to grab the url web stuff and import json So now we're going to use json, but we're going to actually pull the data out of The out of the internet and so I just take that service url for google maps api I found that somewhere in the documentation And then I'm going to have a loop that's going to run forever. I'm going to add for the at the location And then if I hit enter, that's what this is saying get out of the loop And then what I'm going to do is I'm going to concatenate the The service url, which is this and this url Parse url encode gives a dictionary of address equals and this this bit right here Um gives me the string that leads to Putting this address equals, but encoding these spaces the right way. So if you type a space That bit of code turns it into the plus. So that's important And I've got the question mark sitting here at the end of that Then what we're going to do is we're just going to do a url open to get a handle We're going to read the whole document and because it's utf 8 coming from the outside world And we want it turned into unicode inside our application. We say dot decode We can ask how many characters we got And we put our json load s now up till now We've been just doing load s's from internal strings, but this is now a string that came from the outside world and We'll put a try accept in and we'll set js to be none and that'll be our little trigger Now we can look for They give us if we take a look at the output They give us this okay and that status can be a problem and it can complain about things So we have to check to see if we got a good status. So At this point if you look at the outer bit of this the outer bit that we get is A curly brace. So it's an dictionary. Then there is within that dictionary a key results, which is a list But then the second thing in the outer dictionary is status and so we can ask if the the word the the um If we got a False if we got nothing that we'll quit if uh, we don't have a status key in that job in that J the object or that dictionary or it's not equal to okay Any number of those things if this or this Or this are all either of those are true We're going to quit failure to retrieve and print the data out And when you're starting to read stuff all the net you often have to put debugging in here like this to like Oh, something quit. I got to figure out some and so debugging it Next thing we're going to do is called json dump s, which is the opposite of load s which takes this array that include Dictionary that includes arrays and we're going to pretty print it with an indent of four And then we're going to print that out And so if if you look at my code, we'll see that the first thing we do once we parsed it Is we print it back out so we can see it And then we're going to dig into it. So let's go ahead and run this code python geojson dot pi R One of these days I will always type python 3 And arbor comma michigan Okay, so it ran and so you see that it retrieved this url this url was constructed and retrieved 1736 characters And it's json pretty printed with an indent of four And this is that this is that json dump s all the way down to here So that's just json dump s and then it starts extracting So it's going to pull things out now when I when you write this code, it's really easy to look at this and say, oh great It's easy I tend to have to print this stuff out over and over and over and as I kind of construct this expression But if we look at it The outer dictionary the outer dictionary sub results Leads to this array And if you go look at this array carefully you find there is only one thing in it And so that the results is an array sub zero Gets us This this dictionary I keep on to say object because that's what it's called And that goes all the way down to here So that's what we get there and then within that we now have an object and we look for Geometry within that object where is geometry right there geometry Geometry goes from There to there. There's geometry in there. You got to get used to it. That's why it's nice to have this stuff indented geometry sub low oops come back come back And then we go to location within that so location within geometry and then within location we have lat and long And so this is pulling out this 42 and 83 And then so we print that out Take a look and that prints that out pulls that right out of the json These are tricky to write but after a while you win and you get it right and it's just fine okay And so we do the same thing Results of zero formatted address gets us this and so that's how we print the location out And so that's a real quick look at how we would do that With the json talking to the Google maps api Okay, hope this helps Now we're going to talk about api rate limiting and security But the key thing is is that the google api and the google data is super valuable And you could build a website that did nothing But sort of like asked the person for something and then showed them that place and make it be a map searcher And you added so little value and google did all the hard work And so they protect these someone sometimes they'll say you can only do 50 of these a day or 500 a day or whatever That's called rate limiting and sometimes they say you've got to log in You've got to create an account and get a key with us and then present your key So that means that your account only gets so many and they keep track of who's using their service and how much they're using it Google gives you even sort of a dashboard that tells you some of this stuff. It's kind of nice and so and The other thing is is sometimes an api is free and then it becomes popular and they decide they're going to put a key on it Or a rate limit on it So you got to kind of play this game with them and the rules kind of change As things progress so that geocoding api That we're talking about has has at one point in time 2,500 requests a day You can get more requests if you get a key Now another api that we can talk about is the twitter api now a twitter api started out as a free public api But then twitter realized that people were making more money off of twitter's data than twitter was making off of twitter's data And so twitter makes it so that you have to have an account You can only you can only request data from their api is if you use your account key To sign that and so there's a whole series of getting and issuing keys And then using those keys And i'll just give you a short summary of the kind of code that it takes to build those key build those requests up that have to be signed So you'll look through the the twitter documentation and it'll say oh this url to get the tweets et cetera et cetera And it says do a get request to this url and that url and maybe substitute a little bit of things here For the screen name you're looking for or how many tweets you want And they tell you how to carefully construct these URLs And so here's an example bit of code that talks to the twitter Now for for now i'll ignore the security bit That's all hidden in this tw url So it looks a lot like the last one we're going to use json and url lib And we have found that this is the api name Blah blah blah blah list dot json getting a friend list for a particular person And so that is the base url that we're going to do And we're going to ask a person for a twitter account if we hit enter we're going to break out And tw url augment we're going to say give me the first five friends of this particular screen name The one we just read in from input And this tw url you'll see in a second It adds a bunch of stuff to prove that you are who you are It's signing that url So you're sending a signed url, which is nothing more than a whole bunch of crazy characters We'll see that in a second we retrieve it and this is pretty straightforward We can just you know open the url Read it and decode it decode solves the utf-8 thing Makes it all so that data is a real string and it's in the unicode internally Now we can actually get the headers remember i told you earlier that url open Bypasses the headers, but it's stored them for later and we can say hey give me back those headers And it gives us back a dictionary of headers and the headers if you go all the way back are a bunch of key value pairs Key colon value in the headers and in twitter if you read the documentation There's this x-rate limit remaining that tells you each time it returns to the api Response to the api call that you made it says look you've got 12 left you've got 11 left you've got 10 So you can print that out. So this prints out how many you've got left Then we parse the json data We're going to print it so we can debug it This dump s dump dump to string and then print it indent equals four. This is called pretty printing And it's indenting things really nicely So that you can make more sense of it whereas when these things are talking when programs are talking to each other They don't really make the output look particularly printed pretty And then if you We're going to go through we have the outer thing of users and we're going to print out this screen name And go grab the for each user and users. We're going to print their screen name We're going to grab their status text and print that out. And so this is what that data looks like Kind of chopped a bit.  And so those are the bits that we're extracting from that if you look We're going to grab the screen name. We're going to grab the status text and away you go. So You can start with this But you realize that once you're looking at this and you're printing this out with pretty printing You can sort of work your way in knowing that it's an either a dictionary or a list If it's a dictionary, you look up the key if it's a list You say which position it is and then you get more dictionaries within dictionaries within dictionaries and away you go And so this code Actually, you know when it runs it prints out the screen name and then that status and the next person So it's my first five in that case my first five friends And their most recent status the first five people now Let's talk a little bit about how this security works And so you have to go to the the website you have to have a twitter account You can't talk to twitter api without a twitter account And then you go to this website And then you set up a key you say i'm going to have build an application That is going to consume the twitter api And then you go in you have to work through there's documentation on how all this stuff works You set up an api key. You set the application. So I made a key called python on my laptop And it gives us some values It gives us a consumer key a consumer secret a token key and a token secret And you get to regenerate these and there's this file called hidden dot py And you edit them and copy and paste all the stuff from those pages those four values into these strings Now if you download my code, I don't have my keys in there. I got some placeholders for this stuff So you got to get to this web page that's on twitter copy these things in And then the twrl code will start to work It uses a technology called oauth which is a way to Sign a url in a way that proves that you have the key in the secret And the tokens and the tokens But uh, and it can't be modified in the middle. So once you send this url They can check the key in the secret to make sure that you truly signed it without actually sending the key in the secret That's actually kind of cool and fascinating, but we won't go into it in great detail here And so if you look at the code In tw url dot py This is the code that does it. It actually pulls in an oauth library that hidden dot py That is that code that you've got Um, and it's got the consumer key the consumer secret secrets, you know, this is pulling that from hidden dot py Uh, this is a lot of stuff that's using this oauth library. Don't worry too much about that Eventually it produces a url that looks like this and the way what happens is this was the base url You were told to use Then you have count equals two and screen name equals dr. Chuck those parts are your parameters to that web service call And then all this oauth stuff Is produced by this oauth code and the consumer seeing key in the secret What happens is the key gets sent the key gets sent the And the uh secret does not get sent but they send this signature Which is based on the secret and then what it does is it re checks the the signature on the far end The signature is a long string by regenerating the signature because this the secret's on available to both you To generate the signature and to them to check the signature. So it's kind of like a hash Etc etc You don't have to worry about all this these urls get really long And your values that you need are in the name of the urls in and you call this routine That's called augment that takes a url and then parameters and then augments it by adding all this oauth stuff And so that's why it's called Augment to augment the url and once you got this set up and hidden working Then you sort of just augment the url and then hit it Now, you know, if you don't have the right keys or secrets or you don't have an account on twitter Then it's going to blow up, but if you get it set up you will be able to talk to the twitter api with this So this whole web services section. We've done quite a bit of stuff, right? We've looked at how instead of reading html or flat text We are creating structured data according to contracts whether it be xml or json We can retrieve and parse that information in a deterministic way We talked about schemas that define the contract so that you know if the data you're getting is wrong you can know who to blame because the schema gets violated and We've played with apis where you're talking to someone else Who's defining what the rules are and how to read their documentation And even if they have an api key or need to sign urls Showed a little bit about how to do that We're doing some code sample code playing through with some sample code samples And you can get this by downloading it. I've got this whole thing downloaded and um, I've got all the files here And these are the files we're going to play with today Today what we're going to do is talk to the about the twitter api and then and The one thing we got to learn about the twitter api is we have to authorize ourselves and so we have to You know make sure that We have a twitter account and then we get some keys And so for in this particular application if you want to duplicate what i'm doing You have to go to apps dot twitter dot com click this create new application button and then get some codes Okay, and the codes show up as soon as you hit this button and then one more button, which i'm not going to do on screen um And so what happens is there are four codes that you've got to put in this file hidden dot py The consumer key the consumer secret the token key and token secret These are just messed up. So i'll show you how this works and blows up if First and then i'll i'll put my keys in here without showing you Yeah, but basically this is a little file you got to edit or these twitter ones stop don't work You'll see what happens So the first one i'm going to do is is do the simplest one of all And that is i call call this thing twitter test and it just is going to go Ask for the user timeline and we can take a look at this and we're going to Take the url And we're going to augment the url. This is the base we found this looking at the twitter api documentation We're going to pass a parameter of screen name dr. Chuck and a count of two So this is just a python dictionary and augment comes from this little bit of called code called tw url and This uses a bit of code called ooth, which is Built into a python as well Right. Yeah, that's built into python as well and it augments the url And it takes the the key the secret the token key and does a thing and signs it and then makes this big long ugly url what you will soon see and Does this it's a signature of the url. So we we passed this data back and forth to twitter With a signature and then they recheck the signature and it's a digital signature that knows that This url came from a program that knows the key secret and token and token secret And so this augment basically is something that i wrote tw url Augment is something i wrote to make it easier to add all these ooth parameters and you feed This code by putting your data into hidden dot py lots of people get this to work So don't worry. It's kind of cool when you finally get it to work So let's take a look at what it does just to just know that this makes an awesome url That does all the security and we'll see one of those urls um So ignore the certificate errors this has to do with the fact that uh ht We're using https and python doesn't have enough certificates put into it by default For a lot of reasons, but our quick and dirty ways to turn them off Thank you python for reducing security by teaching us so that this is the best way to do it That's a crumpy moment from on my part. So what we're going to do is we're going to do a url open This bit here is to shut off the security checking for the ssl certificate And then we're going to read all the data and then we're going to just want to print it out And we're also going to um ask The connection this url remember i told you a long time ago that url Lib eats the headers, but you can get them back and now we're going to ask to get a dictionary of the headers back And so we'll print those out Okay, so this is really kind of just Testing the the body and the headers and printing them out sort of in as raw a way we can do so Let's go run this now. This is going to fail the first time we do it because we haven't put the Hidden variables in there. So if I say python 3 tw test dot p dot p y It's going to run and blow up and it's going to give you this 401 authorization required That's a good sign because that means that you haven't yet updated your values in hidden p y and so The this this is the url. This is that augmented url And you can see the consumer key and the consumer secret and the oa token and whatever Okay, so these tokens are like wrong. These are the these aren't oops control c They aren't real and then and but you'll notice it doesn't the key in the secret in the token key The token secret in the secret um, and that's all actually encoded in this signature. It turns out that You you need to have the key and the token. I mean the secret in the token secret to generate the the uh signature And um, where is the signature? Oh, there's the signature, right? There's the signature and so This signature combined with the knots that you can only do the signature as a time and includes all kinds of things So even if you type this in well, you'll see these go by And it's not really breaking my security too much when you see these afterwards So don't get all excited when you say oh, are you revealed your token in your your key? Well, I can reveal my token in key, but I can't I'm not going to reveal the secret okay, so This adds all this oa stuff off nonce all a time stamp And these time stamps and nonces are made it so that you can't replay my url Even if you see the exact url Once I hit it then you can't hit it again and so that's what the nonce does So i'm going to close hidden dot py here And i'm going to update hidden dot py in another window Okay, so so I just in another window. I updated hidden dot py i'm not going to show you that But now i'm going to run python tw test dot py so tw rl It's going to read hidden and now these keys and secrets are my real ones that I haven't shown you So this should work fingers crossed Yay, it worked. Okay, so It worked So i'm calling twitter. Here's the url Now don't worry the token and the consumer key are not enough to break into my account And neither is the signature because you can't replay this in about five Minutes you can't replay this anymore. Okay, so You can't generate the signature. I've done one this the signature The signature includes the time and date so you can't Trust me go read up on owak. Don't worry. I haven't really revealed anything But so the first thing we see is this so we see and we should put like the line of dashes here This is the json. It ain't very pretty. It's not very pretty. Okay, and so that's the json from there to there It's just what most apis give us back. It's really dense json, right and so this is a byte array remember How you have to do a dot decode I didn't do a dot decode here And so this is telling a python is telling us this is a byte array, which it's a raw set of bytes That came from the internet which probably are utf 8 and if I put a decode here Then it would decode if I say dot data dot decode there Then it would be fine, but we don't care. This was just a dump. Do we get anything? and so then Okay, let's do this print I'll just make this code different Put some equal signs here a lot of equal signs so we can show them easily see the where the Where the thing starts and stops so we'll run that again if you look at those URLs So that was all of that stuff and then This is the headers and so the headers again are not pretty You get the headers. It's a dictionary. You got cash control. No cash comma. This this is the string Key value. You got to find your commas key value, but the one that's really interesting here is Uh, which one is it? x rate limit remaining right there x rate limit remaining So that means that for this particular api And this header tells me that I've got 898 calls left and this is when I will get more calls and Uh, yeah, so so let's see. Yeah. So so watch I want to do this again And you will see that I can only do this 897 more times now Run it I can only do this 897 So I am being tracked at this point. I am being tracked by twitter You know twitter knows that it's dr. Chuck that's doing this and dr. Chuck has done 900 he's done 899 897 and if I keep running this eventually twitter will tell me You got to wait for a while and that's because twitter doesn't want me under my dr. Chuck account pulling out like lots and lots of stuff out of twitter and making my own website I do actually have my own twitter website Using some cool software Dr. Chuck com slash twitter And this I have to run and it rate limits and causes all kinds of you know, whatever so Okay, so great limit So I'll save that So that's tweet. This is just to test it. Okay, because we're doing I want to do something interesting So we're not parsing the json that comes back. We're not doing anything tricky with this And away we go so Let's take a look at some more code. I think I don't need this anymore so now I am going to Parse this so most of this looks the same. I've got that same user timeline json I'm going to ignore the ssl certificates. I'm going to write a loop. So I'm going to ask the twitter.  It really is decoding the data and only printing the first 250 characters. So let's run that dr. Chuck Boom and it's got 896. So that's just a little simpler version that with a little less brutal debugging Okay, so now let's do something even more fun. Let's go to twitter 2.py and tear it apart and so So again, we're going to look at my friends list Or someone else any of his buddy's friends list. We're going to ask for the friends And ask for the screen name Ask for the first five friends and then look at their statuses Open it decode it get the headers print the rate limit remaining all this stuff is the same as in twitter 1 And but now we're going to parse the JavaScript.  I'm going to guess that twitter is going to give me the right stuff You'll probably want to put a try and accept here that I'm going to do a debug print. I'm going to do a json Uh pretty print. Let's make that b2 so it looks a little better Um And then well, I'm going to run it and then you're going to see how we have to parse this and we're going to see that It's a list Um, so we're done with that and now we're running twitter 2.py So I'm going to go to dr. Chuck And this is going to ask the question who dr. Chuck's friends are Okay, let's go to the top so it hit this api and it has the Our screen name dr. Chuck county equals five and all this oauth stuff again This is not a security breach by showing you all of this Because the signature the secrets aren't there Okay, so if we look at it it's an outer It's an it's an outer object Or dictionary and then the outer has a users which is a list And then each user has some stuff in it. So this one's Stephanie teesley. It's got her screen name It's got some descriptions Keep on going. It's got her status her latest status For my friend her status Her source where she's at I don't know man. She's got a lot of stuff here. Okay, there we go. That was the first one Okay, and then the next one that I'm following is live edu Etc. And so you'll see that this is an array. So that outer thing is an array of users now j s here is A dictionary So I can say for you in j s sub users. Well j s sub users is a list So the first u is going to be this Stephanie teesley u and the second u is going to be live edu So that's all it took to get through all that stuff and figure that out And then I'm going to say Get me the screen name of my person.  There's sub status sub status is all this stuff More more more more more more right there.  So it's going to extract this bit right here Okay, and so You use status text and I print out the first 50 characters of the screen name status And I do that for the first five because I told I told it I only wanted five And then of course I get to see the right limit. So let's go down to the bottom So all of this is the debug print of the json. I got back Here is the program starting to print here is the screen name of my first friend And here's the first 50 characters of her most recent status Here is the screen screen name of my and these are in a reverse order who I've been Following so I've been playing with this live coding stuff. So I'm following them What Key error status that didn't work Oh, that's because live coding tv somehow doesn't have a status So most of these work. So now you'll get to see me fix something and when you download it, it'll be fixed And so it says key error status. So that means that I've got to do a thing that says If status not in You Print no status found Continue. Yeah, since sometimes there's no statuses. Who would have thought I did not know that? Yeah, so you Okay, so let's run this again On it even did I get to see my remaining? Oh, actually, let me change the order of this Let me Put this down here That'll be wrong from the slides, but it'll be prettier now Let's put the headers after the dump of the data Okay, so let's run it again Did I save it? Yeah Dr. Chuck Blah whole bunch of stuff. So I got 13 remaining calls on this one. So it's not the same as the other one I'm you know, I don't get to call this too any more time. So hopefully I'll get the debugging to work sort of I got a bad space here. I'm gonna know not status found no status found and I'm putting I need to put three spaces there No status found I'm making asterisk. So let's run it again And see I got 13 remaining. So it's important you write code that's aware of your remaining That's why I made so so obvious about that go retrieve all that I got 12 remaining But my code starts to look oh dang it. I now another space here. Hang on gotta fix that I need yet another space hopefully I can make this as pretty as I want it to work always I didn't even do dr. Chuck. I did that wrong Type my name wrong. Okay. So now it works Oh, well So so now I have my first most five recent friends are this deft easily live edu official life coding tv nancy gilby Um, and greggy cruder. And so there are their statuses and I tore all this jason apart using, uh twitter 2.py of course after fixing Hidden dot py which I'm not going to show you because it actually contains My real consumer key and consumer secret You're seeing the consumer key and the token key go by on each of these URLs But what you're not seeing is these two things which are the thing I'm protecting so that it's not a problem Okay, so I will send that up But uh, there you go. Welcome Uh, thank I hope this I hope you found this useful. Uh, the code will be fixed when you take a look at it Uh, and download it here from samplecode.zip Hello, and welcome to python objects. I'm charles severance and uh, we're well on our way to uh To get through all this material in the python. So this lecture is in a weird place I even debated where to put it in the book um I don't really want to teach you how to write a lot of object-oriented programming but we're going to start using objects and I want to be able to use the terminology and so as much as anything this lecture is about terminology and understanding The words things like methods and method signatures and variables and inheritance And so think of this as a terminology lecture rather than a learn how to program or learn how to Use this it's not something you're going to figure out right away And there'll come a time when you as a programmer really want to start using object-oriented programming It's really a powerful and wonderful uh technique But uh, I think it's uh, too early as a beginning programmer to really say oh, let's write a bunch of objects so just relax and enjoy and learn this material and think of it as sort of a A theoretical thing rather than you know a how to program thing and so Part of this is we're going to start reading data structures and I mean data On how to use all these libraries et cetera and we're going to see the word objects, right? And then we're going to start hearing them and I want you to be able to read the python documentation so that you understand what's going on and So, you know the word objects should make sense to you even though you're not going to write a lot of object-oriented programming And so page upon page upon page database stuff, which we're going to talk about soon is Uses objects all over the place and the beautiful soup talks about uses objects We've kind of been using them and I've been waving my hands and I use the word method without defining it But now it's really time to Define it and go go to it. So um I want to review From the very beginning what we think of as a program So the classic program my favorite little minimum program is our little elevator floor converter with Which converts from european elevator floors to united states elevator floors And the key to this is that It's input processing and output and this is a good way to model any program um And in that process we've got variables and we've got Logic we've got algorithms. We've got loops that we write. We've got all kinds of things and we construct a series of steps to achieve some goal In object-oriented and and frankly you've been using object-oriented all along The program has lots of objects And we're sort of putting stuff into these objects taking stuff out of one object and putting it into another object And you you've actually been doing this all along as soon as you're looking at dictionaries and lists You're doing objects and so it's An object is is quite a little thing. It's sort of its own little space inside of a program that contains code and data And so We're working together all these objects are now working together It's a bit of self-contained code and data And it is one way to take a very complex problem And make it easier by breaking it into separate things that can be engineered and and develop separately So you've been using string objects or maybe you'd used beautiful soup or something These are powerful capabilities and if you had to look at all of them It's just hey, here's a thing use this object It'll do these things for you and there's lots of details inside of it Just don't look at it. Don't worry about it And so there's boundaries that things that you can use things that you can look at and things that really you don't bother looking at You go read the documentation and use it and A way it goes but then someone had to write that and so they built an object So we're going to look a little bit under the covers of what it takes to build some of these objects And so if we think of this program that originally just sort of did processing We can think of it as having some kind of an input right Coming into our program and we have a string object a dictionary object Maybe eventually some objects like a database object or an object that we eventually define And you can think of us. We're receiving data It comes in an object which is a string object or start putting the strings in dictionaries and do whatever We pull out a list of them and and so you can think of data is moving between these objects And like I say even strings in the first week First lecture the first week first everything we We were using objects and we've been using them all along And so you can think of every string and every dictionary as a little Program all by itself that has a bit of code and a bit of data Um, and so a string has the data which includes all the characters that make up the string But then there is a method called a Upper that does uppercase or our strip that strips off the right a white space from the right And so it's it's like they're almost little programs that have inputs and outputs themselves And we can make lots of them and there's lots of cooperating objects that make up an application um And one of the nice things about the object oriented pattern is that they Form boundaries and within the boundary if you're inside the object you can say look I'm going to build you a string object or a database object or a beautiful soup object And I'm going to build this capability and I'm going to give it to you in the form of an interface And I'm not really going to care how you use it And so we have this sort of visibility wall where I'm going to make an object and I'm going to let you use it And the maker of the object doesn't necessarily have to know every single thing About the use of that object But so just like Inside the object they don't have to worry about what you're doing with the object outside of it When you're outside the object you don't have to worry about what's going on inside of it We as the user of the object we talk to its interface and we get things from it and give things to it and Use functionality within that object, but we don't have to look inside of this We can just say oh, it's a nice little magical thing We read the documentation We read a web page and it told us to do this this and this And away you go and so it is a it is sort of this isolation boundary that works both for the Programmer who's writing the object and the programmer who's using the object and so it's uh, it's a very nice pattern Um, and so you'll see how we're going to build code and we're going to group it together And then we're going to be using it sort of is a big blob of stuff So some definitions in this space words that I want you to understand um When we're going to create one of these Things one of these objects instances That has some data in it and some code in it We have to be able to define the shape of this object What code will each object have in it and what data will each object have In it and that's called a class The key to a class and this little picture that I've got up here in all these slides Is a key the class is a template. It's not the thing itself. So it's a cookie cutter It knows a lot about how cookies are made and if you have cookie dough and you hit the thing Then you make as many cookies as you want and so this nice little cookie picture is a great You know Mental model of how it works the class The class oops The class is the template and then the object Are all of the cookies that are made from that template But the template defines the shape and the nature of the class. So the code that we write is Is going of each of the objects the code we write Is the class code and then later we say oh, let's take that template and make ourselves an object or an instance Now as we're defining a class We have two basic things that we put in the class and there's a couple of different terminologies for this One is method which is code. It's like a function that lives inside of a class not a function that was Inside your program, but one that lives inside of a class. And so this is a scoping thing a method is really just a function But it's lived it lives inside the class and then fields or attributes are data items that are in the class And so there are variables that are defined in the class You can define variables outside the class that you use in your program You've been doing that all along But if you're saying i'm going to build this capability and it's going to have data inside of it and code inside of it The code is the method or message and field or attribute and they're just There are just two um Two different sets of terminology Method is what i'll probably use if you look in some object or any patterns like small talk or Apple they often don't call these messages So you can either like access a method inside of a class or an object or you can send a message to the object The same is true for field and attribute. There's just a chunk of data that's in the object that may you may or may not have the right to access So like I said a class is a template it defines the characteristics of the objects that we're going to use to make it It is the cookie cutter So dog is sort of the exemplar Lassie is a particular dog and so dog has fur and dog barks and dogs do all these things And so we know something about dogs But it doesn't mean we have a dog, right? We and the and the the class is a more abstract concept that that when it's time to get a dog We know certain things about dogs instances or objects are Once we say oh time to make a cookie from the template time to get a dog. We know something about dogs That's the creation of an object and that we call them instances instance of a class so the class is a Doesn't exist, but we say make me a new Object using this class as its template. Oh, and I'll make me another one And so we can have many many objects from one class. So just like many cookies from one cookie cutter Method is a bit of code that lives inside of an object It's like a function, but it's scoped to within the object or within the class Okay, so that kind of gets us started on some of the terminology and we'll come back and we'll take a look at how we write code And that's object-oriented Okay, so now that we've gotten through the definitions. Let's work into some sample code But hey look at this. We've got ourselves a cookie cutter and some cookies So remember that a class is a template. It's not the actual thing an object Is an instance of a class So you have to take the class and do something to make the object And actually you can see here some other classes There's clearly a sort of a snowflake class and a gingerbread man class That's an object object object somewhere out here.  So Class is the template Object is the instance. So here's a bit of python code. So let's take a look at what we got here Class is a new reserved word kind of like def We have the name the class that is a name that we choose We're gonna that's the name by which we'll refer to this class for the rest of this program And it has a colon at the end of it and which means it starts an indented block which ends when we de-indent Inside the class there are generally two things There is some data and this just looks like an assignment statement in the class x equals zero And then there is a def def look this looks just like a function and then it starts with a def Has a colon indents so that function finishes right there The difference is is this is a method because it lives inside of a class And so there is no function called party. There's a function called party within party animal class And we'll talk in a second about this self thing It is the way that inside this code refer we refer back to that variable So this is not actually executing any code. It's sort of remembering the template defining the class party animal This is what we call constructing. We're constructing a using the party animal template or class We are making a party animal and then once we make that we stick it in the variable Uh an and then we're going to call this party animal This party method three times one two three Now this self thing and we'll take a look at the self the self ends up being An alias of an and so you can look at this syntax. It's just kind of an equivalent of this syntax It's calling the party method within the party animal class And passing the instance in as the first parameter And so self ends up being an alias of an each time these are called Now if we make a different variable and a second object, which we will eventually You will see that that that works a little bit differently. And so this syntax is a short version of that syntax So if we watch how this executes It's oops It starts up here It just defines it and then we construct it And that's what basically Constructing it we know how to construct it because we look at the class and we make a variable x We make some code party and then we construct that that's what the party animal does and then we assign that into an And so an is now pointing at that Then when we call the party method That basically takes this an and passes it in passes it in as the first parameter Which is used as self and so self dot x which is what we're doing in this line right here Self dot x is a variable x starts out as zero X starts out as zero because when it was constructed it was set to zero So we're in here an is an alias of self and now it looks up self dot x Which is zero adds one to it and so this becomes one And then we print so far so far one and then the code returns and it goes down and does it again And x becomes two prints out so far two comes back down And does the last time calls it again self dot x is two add one to it and stick it back in So this becomes three and we print out three and then the program finishes And so you can think of this as constructing the object and then associating it with this an variable Now that we've created this object we can play around with things We've played around before with dur and type we use dur and type to kind of inspect variables and types And objects so we've been using objects all along We this code here says hey make me an empty list Well, it turns out that what we're saying is there is already a list class inside of python And we're constructing an empty list and when we get back this empty list We're assigning that into x So x in a sense contains or points to an empty list So then we say hey, what is in x? What kind of thing is x? Well, it's a list This is a thing. It's a list type. It lists have list of things in them And you know use a pen and all the things we've been doing before They're just objects and then the durr if you remember the durr the durr is the capabilities And there's all these internal capabilities that do things like implement the bracket operator, etc Those double underscore ones we can ignore them although you can even look them up and figure out what they mean If you feel like it but the methods that we tend to call are in this class and so things like x.sort I've always told you that is the sort method within the x thing and the dot operator Is the operator that we use to look something up within an object And so you've been using the syntax All along x.sort dictionary.items all of those are methods within the corresponding class If we take a look at this line of code that we've been doing for a very long time Which says oh stick hello there into y it's if I reword that as more o o or object oriented What this single quote does says make me a string object And put some text in it and then when that is done being constructed stick that into y right and so y now Points to a string object that's been pre initialized to the string hello there Now that's a long way of saying hello there ends up in y But in o o terms we can talk about that if we do a durr of that we see a whole bunch of internal Methods which have double underscores and then we see all kinds of methods that we've been using we've been using methods like upper we've been using methods like find we've been using methods like r-strip right we've been using these methods so we're going to like y dot r-strip parentheses again that's a method that's an object Not a class it's an object and that is the object lookup operator Now if we do the same thing to code that we've built or a class that we've built so now we have A party animal class remember this up to here is just definition Now we construct it and we store it in an So an is a variable that contains an object of type party animal We ask it what type it is and it prints out here it says this is a class And it's main underscore party animal and this whole thing here is the underscore main It's scoped underscore me but you can see that you have made a new type You built a type by using this class keyword and then we use the durr remember durr looks for capabilities And again you will see You'll see a whole bunch of underscore things they have meaning you can look them up But eventually you'll see the two things that you've put in it One is the method party and the other is the attribute or field x and again These are the things that you can say an dot x Or an dot party because this dot Is the object operator the object lookup operator that says look up in the object And the thing x or look up in the object an the thing party Okay So up next we'll talk a little bit about how objects are created and destroyed We also call that object life cycle Now i'm going to talk a little bit about object life cycle and what we mean by object life cycle is the Act of creating and destroying these objects and i've been using this term constructor already and so When we declare a variable, whether it's a string or a dictionary or a party animal where there we create them And then they're discarded and there's all this dynamic memory that comes and goes And we as the writers of objects have the ability to insert ourselves at the moment of object creation And at the moment of object destruction and we make Special functions that we call the constructor the object constructor or the class constructor and the destructor And we don't actually explicitly call them. They're called automatically by the by python on our behalf And so the constructor is the most more commonly used it's used to set up any initial values of variables if necessary Etc etc destructors are we'll cover them, but they're they're used very rarely So here's a bit of code that we've got It's our party animal and a lot of it is the same as what we've been doing so far Um So we have this variable x and the constructor has a special name underscore underscore anit underscore Again, we pass in the instance of the object self and in this one all we're going to do is print out that you're constructed And here's this code that we've had before and now we have underscore underscore del And then we pass in self and we'll just print out that we're being destructed and what the current value of x is for that particular instance So let's go ahead and run this Um And so again, this doesn't really do any code up to here that just defines party animal But this is the constructing of it and basically that says oh And it really kind of creates these variables and then it also runs the constructor And so in this case this line right here is causing the iam constructed message to come out Then we do and party an party and that says, you know one and two And here's an interesting thing.  It's out here It's vaporizing it and throwing it away And so before this line completes it actually calls our destructor on our behalf And so that message comes out So we are allowed as the builder of these objects to add these little chunks of code that says I want to be involved at the moment this object is created And I want to be involved at the moment that this object is destroyed Now in this last line an is no longer a party animal an is now an integer It's got a 42 in it. It's gone. It's been created. It was used and then it was destroyed Okay, so you got to be careful if you overwrite something you can sort of throw the object away So the constructor is a special block of code that's called when the object is created to set the object up So we can create lots of instances everything we've done so far is we make a class and then we create one instance one object And each of these objects end up being stored in its own variable. We have a variable an and we've been using it But the more interesting thing begins to happen when we have multiple instances of the same class Sitting in different variables and it has its own copy of the instance variables. So let's take a look at this So This code here. I've taken out the I've taken out the destructor And it shows a little bit more information. So now we're going to put two variables in here We're going to have a current score or whatever and a name and we're going to start it out as blank And this time we're going to add a parameter onto the constructor And so the self comes in sort of automatically As the object is being constructed But if we put a parameter on the constructor call, which is this party animal call Then this comes in As the z variable and so self is the object itself And z this first parameter is whatever parameter we put here everything we've done so far has no parameter here But now we have a parameter here and then that means that when we call this constructor This line of code comes and then name is no longer blank name is going to be sally in this particular thing And then I'll say oh self dot name, which will be sally was been constructed And so then then we have this and that Object is now constructed and then we put it in the variable s And then we call the party method on that and we construct a different one And so this time it calls and z is jim and we basically have a Oops Another copy of this and so this is how it's going to look right as as it runs down here As it runs down here when this is called it makes one instance And stores that in the variable s and there's a variable x in there. There's a name in there There's an init method in party and that's all in here right all that stuff is in here And now we say let's make and that's going to have a sally in there All right sally in there And then we're going to do another constructor and so it's going to make a whole new thing And it's going to store that in j and this one's going to have jim in it On s party then this turns into a one and then we're going to call j party That turns that into a one and then s party will cause this to be a two Okay, and so what happens is is we have now two objects One in the variable s and one in the variable j and they have separate copies of their instance variables These are the instance variables or the object fields or whatever, but they're the variables But the key is is that every time we do a new construction It duplicates this and there's another copy of it. So there's an x within s So s dot x is this variable and j dot x Is that variable? Okay So the next thing we'll talk about is inheritance and that's the idea of taking One class and extending it to make something new So the last topic we'll talk about here in object orientation is the notion of inheritance And this is a form of code reuse and it's one of the more advanced aspects of object-oriented programming So just kind of understand what it is at a high level And then you know where to come back to when you need to learn a bit more about inheritance So the idea is instead of making a new class from scratch We actually make a new class by starting with an existing class We are extending it or another word for this is subclassing and it's sort of a A situation where you're like i'm gonna i've got this code and i've got this data And i just need to add a few things to it and then i'll have a whole new thing Um and as you design objects and what we call object hierarchies You often do this and it's a form of sort of real clever code reuse But again Don't necessarily think that you're supposed to know when to use this or why to use this is right now It's just terminology. Okay. Just terminology We have what call these as parent child relationships The original class is called a parent and the new class is called the child class So subclasses are another word for this you have a class and then you subclass it I think extending and inheriting and parent child are probably better ways of expressing it than subclassing So here's a bit of code. Let's take a look at this Um, this is this code's unchanged. It's the party animal code that we've been saying all along It's the one that you we construct and put a name in And now what we're going to do is extend it and so you'll notice that this code down here is the part That's doing the extending so we're making a new class football fan And by putting in parentheses before the colon party animal that says Football fan inherits everything that is party animal meaning the x the name the init the party All those methods and data are sitting there And now we're going to add a new variable So football fan has in addition to all those other variables has points and it has a touchdown method And you know point self points is added, you know to we add seven of the points and then we call the party And when that does that so this is calling this method because football fan includes x name and party and init and everything and all this this constructor So so this football fan is really an amalgamation of all these things together party animal Is just this stuff, right? But and so we still have two classes We don't just have one we didn't erase the party animal class And so we take a look at the code that we can run here We can say oh, okay. Let's make a party animal sally And so that constructs a an object like this and then stores that in s And um with an x starting out at zero and and then we call s party Oops better change that color Starts out at zero and then we call the party method and that changes it to one Okay, and so this is This bit of code it's as if this part doesn't matter at all because it is a party animal It's not a football fan, but now if we take a look at This code down here Take this code down here. We're going to construct a football fan And pass in gym But football fan has no underscore underscore init So that actually uses the underscore init from party animal because We extended party animal to make football fans So we inherited all of the good that was in there So there it's going to make a name a variable x which is going to start at zero A variable name that's going to have gym in it and a variable points It's going to have a zero in it. So this j variable has More things in it than the s variable has and so we can call the j party And if we call j party that goes here and adds one to x All right, so that adds one to x and then we call j touchdown Well that comes down in here And adds seven to the points Right and then calls party within us and so so self dot party is the current object i.e Self and j are the same thing right self dot party and then it goes up here And passes self in and it adds one to the x in this case of this j variable So this becomes two And that's where it prints out it prints out, you know Seven and two and away you go and so It's a way for you to kind of take all this stuff and stuff it Into a class by making a new class and just add the extending bits the bits that are In addition to the other stuff. So like I said inheritance is a powerful and wonderful concept. It's a form of uh excellent form of reuse, but uh basically The whole purpose of this lecture was so that I could in the future just use these words and you would Understand them as compared to I just want to say method and I've been saying method all along in this high time that I defined it So let's just review one last time class is a template It is not actually a thing. It is a shape of a thing And we define it and say when we make one of these things it's going to have these variables in it It's going to have these methods in it Attributes variables within a class method is a function that's inside of a class Object is once we construct a class we get back an object and so object here is The snowman cookies class is this snowman cookie cutter And a constructor is a bit of code that sets up our object our instance When it first is created an inheritance is this ability to Create a new class but take all and import and affect all the capabilities of an existing class so Object oriented is awesome For the rest of this class, we're not going to write any object code We're not going to use class at all But we are going to use objects and literally you've been using objects from the beginning of this course as soon as you said Um print whoops as you as soon as you said, you know x equals high That's an object and as soon as you said x dot upper You were calling a method right you've been calling a method all along when you're doing something like F h equals open This thing you're getting back that's an object and then you do fh dot read or whatever You're calling a method in the dot operator. So you've been using objects all along I now i'm just finally explaining to you when I say call the read method or call the upper method or What's this little dot and why is that there? So again, it's time for us to Understand that but you will it will take you a long time before you encounter a problem That's large enough where as part of your solution You're going to make a new object. But when you do it's really a powerful thing. I mean It's a really bad idea for me as a teacher say oh write a bunch of objects It's like it's it's premature for that. It's later is when Um, you will actually learn how to use objects and you'll be like oh, thank heaven that these objects are here. Okay So That's all for now. Thanks for listening. See you on the net Hello, and welcome to our chapter on databases. We're going to learn a lot in this chapter Learn a whole new programming language sql and learn how to use that So you're going to need a new piece of software to run all of the exercises that i'm going to do Called sqlite browser. We're using a database called sqlite Go ahead and download this you might have to pause and come back if you like go to sqlite browser.org And download it and install it While you're doing that, uh, we'll talk a little bit about the history. So in the old days 1960s 1970s I started doing computing in uh 1975 um We didn't have a lot of storage. I mean this is you know 16 gigabytes right here and You know, we didn't even have megabytes. I mean the computer I had had a few megabytes of stuff. So Well, so we didn't have a lot of disk drives and so permanent storage Was often sequential in these tapes these tape drives that we had Tapes and tape drives were the scalable part of storage because you could just make more tapes and you could rack them up And so that was our way of greatly increasing the storage of the computer The problem they had was is they were sequential you read it advances read it advance read in advance Now interestingly We've been writing programs that do this that everything we've written so far pretty much reads the whole file Reads the whole web page reads this everything we read it We read either a loop or read the whole thing and that's because we have plenty of memory, but We're still reading sequentially and um And so the way you would do this when you didn't have enough spinning Storage or online storage is you'd use offline storage, but the trick would be that you would sort it So let's imagine that you're a bank and you have a bunch of Accounts only a few of which are active on any day And you have a tape that has In account number order from low to high The the prior balance last night's balance of every one of your bank accounts And then you do all the transactions and you record how much money was taken in or out For e