welcome to this UK Data Service webinar on getting data from the Internet. The presenter today will be Peter Syneith... of the Cathy Mawls' Institute for Social Research at the University of Manchester. Thank you, Gail. In this webinar, what we are going to do is look at ways... of getting data from the Internet. We are going to cover a four different aspects in pretty short order... dyna'r cyfwels o'r gweithio. Fe ydych chi'n gwelwch chi'n gweld y cwpion, y cyfwels yma, i'r gweithio'r ffordd, eitha syngel chi o'n gweithio'r ffordd, fe yw'r prôl ynghylch. Mae'r apis yn gwybod, yn ymryd, yn ychydig i'ch gweld o'r apis, ac yn y gwaith gwaith. Ac yna'n gwybod i'ch gweithio'r webscraping geniw. Mae'r cyfwels, ac yn y gweithio'r cyfwels. Mae yw yma. Mae'n ddefnyddio'r modd yn llawer. Doedd hwnna'r llyngr yn ôl. Mae'n debyg ei hunain. Mae'n meddwl a'r meddwl reiftwn yng nghwm yn d espaço y internet. Mae gynnal i wneud i ddweud y gŵr ar gyfer y tu hynny. Mae'n ddweud i ddweud hynny, ac mae'n gyd yna yn ei fawr i'r internet i fod yn ddim ar y ti erbyn hyn. Mae'n dweud i ddweud sy'n ei ddweud i'r teithnikau i'r tywiau gymryd y byddwch gyda'r brwysig, fel ychydig yn gweithio'r brwser. Felly mae'n gweithio'r cyfwyr cynhyrch yn python a yn y tu o gyfwyr, a'r cyfwyr gyda'r cyfwyr. Mae'n grwyddo i'r teimlo. Mae'n grwyddo i'r teimlo'r cyfwyr, ychydig yn gweithio'r cyfwyr. Mae'r teimlo'r cyfwyr o'r teimlo'r cyfwyr. Mae'r ddechrau'r cyfwyr yn gweithio'r cyfwyr. Mae'n gweithio'n gweithio. starting off with copy and paste if you want a small amount of data an article or a table or so you may find it easy to just copy from the browser screen you're seeing and paste it into a word documents or an Excel document or whatever. What you get isn't necessarily what you see on the screen gyda'r cyd-dweud ymgyrch yn dal. Yn gwybod,yn gynedig ar gyfer y brouwyr, ydych yn cael ei fod yn gwybod yn cerddoedd. Felly yn gwybod yn gwybod yn gwybod. Maen nhw'n meddwl, ymgyrch yn cael ei fod yn ddweud ymgyrch. Dwi'n meddwl yn y ddweud ar gynnig. Nid o'n meddwl, maen nhw'n meddwl. Yn y cyfriffeil, mae'n meddwl ymddiadau yma, Dwi'n gweithio y gall wedi gŵr o'r anhygoel. Rhe騵a'r dda haf sydd hynny'n ei gweithio'r ddefnyddio'n dda, oedd sydd eisiau plent yn gennym. So rhaid am syniaddd hynny'n gweithio'r ddefnyddio y Mydd ychydig ac yn gymryd cy Teddy C per oes cymdeithrig. Mynd i fyddi gwybod y peth, mae'r ddweud phefydau merddion o'i bleu hynny i mewn gweld, neu'r dweud o'n eu teimdeithig, Tynyrch gweithio'r llwyddon, ond yn dwylo rydyn ni, felly rwy'n meddiwch y cwm yn ei wneud, mae'n meddygwch y llef, ac mae'n meddygwch ar gyfer. Mae'n meddygwch. Mae'r meddygwch ar gyfer, mae'r meddygwch ar gyfer, efallai eich meddwl yn fwy fydag, ac y dyfodol meddwl yn meddygwch, ac yn ystafell yn meddwl. Felly, fyddech chi fyddai'n meddwl ar gyfer.krwfall o gyd gyda'r ysgrifnadol ar y llwyff er mwyn i fod yn ffarnig amdodd rai oed, a ydych chi'n ffordd o ran y cwmwis ar ben yr oed, o prosth cywlau y ddweud o fy modd i'r cwmwis, o'r ddweud o'r ddweud. Ac mae'r ffrindig o'r ddweud o'r ddweud o'r ddweudio at dyma ei ffordd. Wel, mae yw meddwl, y brwyddaeth yma, ychydig y bwyddiad hynny, ychydig y brwyddiad hynny. Dydw i'r cyfrifau gyda'r H1, rwy'r gwaith, ond am ymddangos, nid i'r wath. Mae'r bwyddiad y gallwn ei wneud o'r bwyddiad hynny, yn cael ei ddechrau'r bwyddiad hynny. Ychydig y brwyddiad hwnnw, mae ein ddaeth o'r dweud, y ffyrdd, y crobe yw'r bwyddiad hynny, ond lot of people like to use and if you just look at the different formats which I come up with though on the edge using I did it with both source pasting and destination pasting so we get source pasting with edge now this seems to have given a different result this time because here I've got all of the the teams in there and notice all of these are filled in as well edge destination looks very similar except that these are no longer links as they were the firefox source yes lost teams lost column headers destination very similar for chrome similar sorts of things none of them are absolutely perfect properly the edge source is about as close as you're going to get but even on the edge source if you look at these column names up here these are the one which seemed to disappear and if I go back and look at the natural page in firefox so here I've got played one drawn lost whereas in my spreadsheet it came up as pwdl etc so again you're not always getting what it is you think you ought to be getting so instead of using the straightforward copy and paste what we can do is assume we want to put this into excel as a table we're going to look for we can go into excel go to data from web in the getting transform data this is available in all of the office 365 versions of excel now I think so providing your excel is reasonably up-to-date you should have this available to you and you just just get a little form asking you to put a url in there and so I'm just going to go back up here copy my url and paste it in say okay and this will go away and get that whole page which actually had more than the table on it but it knows I'm probably looking for a table so when it comes back it will actually present both options and saying well you know what table do you want okay and it's not very good at labelling the tables but you can get preview so if I go to table zero I can see this is the table I want and then all I've got to do is click on load and then it will put all of the data into the table or into the spreadsheet it does actually format it as a nice little table you've still got these extraneous things which you probably didn't want team has moved and this stuff at the end which was a little graphics are still the same but this is generally a lot cleaner table if I delete that oops I'm looking to bother cleaning them but you can see what part is the table and which part isn't the table and that's a far cleaner way of importing tables from the internet use excel directly and import it that way so back to the slides I think copy and paste demo cutting paste doesn't always work use excel because it's genuinely better that is just what I've shown you that's good make progress right downloading files of different types this is interesting well well again you've probably all done it where you there's a link which says download or download here or this is a fine name that you want to download and you can just click on it and directly download it and it goes into my document or something like that there are other tools if you haven't if you're not using the browser there are software tools wget and curl which are unix based options they don't naturally occur in windows although there are ways of getting hold of them if you want to use them and I'll tell you the other software too is actually write it in python code or it could be an r code but the demonstrations I've got today are all going to be in python and the advantage of using the code is that you get better documentation and reproducibility because if you just clicked on a link in a web page and downloaded the file you've got no record of how you got that file and so on if we're into coding we can also automate the download so if we wanted multiple files we could find a way perhaps of downloading them all together now automation isn't always going to be possible for example some file some all locations put u i u id in the path or other randomness what i mean by that is strings of hexadecimal numbers which are part of the file name or the path in the file name which it makes it impossible for you to predict what other files in the series might be another problem might be if it's using javascript functions so these are typically the the buttons that you press to download the file because that goes away and runs some javascript to do the download so you can't we need something where we can actually get hold of the file name that we're actually trying to download and if we've got the file name in the path that we can do a fair job with automating some of this and this this approach you will also work for a lot of api calls as well so demonstration again let me just escape out of that for all of these demonstrations i'm going to be using mainly python in the form of jupiter lab jupiter lab comes in the anaconda release of python it's already built in there it's very useful not only because of the python environment but it's got nice lots of little tools as well to help you do things i'm not going to go into the actually downloading by clicking on the button and just downloading into my documents because we've all probably done that before i'll very briefly show you the command line tools and hopefully deter you from using them unless you happen to be a command line user and you you've used them before and you know what they're going to do for you as i said these the the wget and the curl don't occur naturally in the windows environment but are various ways of getting it and windows 10 there is a linux subsystem which will has them in for our demonstration today i have got this little thing called MOBA exo which is actually useful remote accessing of the systems but as part of it you get this little linux terminal session in here um and here i'll just type in a command and it's going to run the command so what i want to do is just demonstrate wget i'll just limit myself to wget and we're just going to copy that go back to my command and you can see here this is a command i'm going to run wget i tell us what it is i want to oops i've got two commands at once say let's just get rid of that wget what it is i want to want you get this is going to be a web page the minus o means the output and i want to call it pbc get dot html okay and if i just run that by clicking enter hopefully it goes away it gets it 100% it's got it all and it's told me it's saved now i've also got open somewhere here i think is the folder that way it's saved and you can see here bbc wget html quarter past three and if i double click on that it will just load it in the browser but you see it's not a very good representation of the page most of the page is most of the information the page is there what is missing is all of the the the graphics now that doesn't always occur it depends on how the page is being constructed some pages will download graphics automatically some don't i just look at the other one i could do exactly the same thing in curl if i wanted to i won't bother you because it's the same results the other one i want to show you is this one here wget um what this is going to do it's going to go to this um this is actually an api um api beginning there doesn't actually mean it's an api it just happens to be an api in this case and effectively what i'm going to call is i want to get hold of all of the football competitions that this api knows about we're going to be using this this api later on okay but the moment this is going to give me a list of all the competitions i know this gets returned in a format called jason so i've cleverly named the file dot jason just to make it obvious to me and again it goes where it connects it oh waiting for a response hopefully that's going to come back and eventually get me some output uh i'm waiting for it have i got a yes i've got an example that i've run previous on at the same one it's over in it okay that should come back and give me a file but we don't need that at the moment that's really just to demonstrate how these things work um control c to get out of that forget about it and the only thing i just wanted to show you um for those of you who haven't used these tools before if i look at the help file for w get there is an awful lot of things you can do with it an awful lot of parameters so as you can imagine it's very large very flexible very but very complex so my advice would be that if you if you're not naturally a command line user you're it's nice to know about them but you're probably better off avoiding them and instead what we're going to look at is using um similar techniques only using python so in here this is my um python environment this is jupiter lab um one of the reasons um i use jupiter lab particularly in this demonstration is that along with able to being able to run all the python um it has lots of little file viewers which makes it easy for us to look at results so i look at the csv file by clicking there we'll come back to that one and even the jason a jason file it will format jason very nicely or very conveniently for you to read jason um is what apis tend to use and an apis really designed for a computer to talk to another computer or programs to talk to a program so naturally speaking that jason isn't particularly well formatted for you and i to read um but there's lots of tools around which will format jason to make it easier to use so in order to use um our our um python we're going to need to slide recall requests again request is built it's automatically installed when you download the anaconda version otherwise you can import it yourself um and then this will allow us to download and save files of a variety of formats so i'm going to start off i'm going to set up a url variable and in there i'm putting this string now this you can probably tell by the last bit of this dot csv it's going to take us out to um or it's going to connect to a csv file and i'm going to save that with this file name here okay so i'm just going to run that and then that's there's your second variable such as python here this is the actual call using the request library where just all we have to see is get which is the method name and pass it the parameter of the url that we want and then i'm just going to print the status code and then i'm going to save the file open the file close the file write the file so r is is an internal object which will contain everything which comes back from the request um part of that is the status code which you might want to look at um the content is effectively the file content and there's lots of other things which we may or may not want to look at so if i just run that hopefully it comes back and it tells me if anything i printed was a status code of 200 this is what we're looking for 200 means okay everything worked okay um other ones which you're likely to come across are the the 400 range so 404 being the most popular as i'm not found but 403 is forbidden you're not allowed to have the file 500 server errors and so on so if it's 200 that's good and you can carry on and just just the other thing one of the things we can look for is just out of curiosity the url itself our .url is what is the file name that we asked for okay and we can see here um file just refresh that oh come in the wrong folder that's why if i go back into files there's my file which i've just downloaded okay um i'll just quickly do the others so i've got copies of these i just want to demonstrate that this technique works for virtually any type of file you want so this is going to be the jason file it's the same file i i failed to download a minute ago using um wget here i'm just going to run it again again i've got 200 coming back and we can see here competitions jason and again um we've got a nice little way of looking at all of the results all of competitions okay we'll come back to using jason later on look at the apns and finally an html file so here i'm going to bbcbusiness.html and i picked this a couple of days ago i hope that file still exists so that's just setting up the the variables run it come back as 200 um bbcbusiness.html and i'll double click on that again because it's it's um Gip delab it's got a nice little presentation of the the page including the graphics in this case it's not quite this quite the same format as you would see in um if i'd done that natively on on the screen here i think it's actually the um the mobile version because of the little hamburger up here which takes me down to the options but it's a it's a live page and i can move around so i'm able to download files at any time using simple code okay and i think you'll agree this is far looks you know you just give it a URL it's a lot simpler than trying to use wget or curl the next thing we would do is is take that a step further and having got one file we're going to look at oops we're going to look at some multiple files so we've already seen that slide the idea here is we want to find a way of identifying the files just like in bulk if you like so back up to Jupiter and now i want for this demonstration we're going a little bit topical multiple files using covid information so this website here getthedata.com is very useful lots of various types of data but we've obviously updated it very recently to include covid-19 data and if you go to that that link which i probably got up here it's got all manner of useful information about stuff that would rather never existed but never mind it covers all of the UK in different areas some type of you and if you go to the bottom it's got a whole load of files available for you to download okay and if i click on one of those files i'll get the usual thing oh you want to save this file blah blah blah okay this is a normal save type dialogue what is interesting about it is that up here it says cases by blah blah blah exactly the same as i was clicking on there and it says which is from htbahttps www.getthedata.com well that doesn't quite match this whole url up here so where exactly is this data coming from now if i'm going to just use this dialogue box download one i don't really care i'm just as long as it downloads it but it would be nice to know where this file actually comes from if i just close that if i am hover over this file right down in the bottom left it's very small so you probably will be able to read it it does actually give me a little graphic saying what the file name is really called and what you see down there is the file name that you really need to be downloading the full path and file name the trouble is it's not there it's not that plus that it's it's something else and so you've got the options of oh what can i kind of read that write it down copy it somewhere what have you uh rather tricky but there are other ways of doing this something that is almost full screen okay now all modern browsers are very sophisticated bits of software and they will allow include all kinds of tools for the web developer they'll all have this sometimes just called developer tools or web developer whatever and they will all have this thing called inspector and this is pretty scary if you haven't seen um hmlfo or web pages before but within within the set of tools there is an item called inspector and there's this one here which is also an inspector they tend to use a similar symbol to this and it's always on the left hand side and if i click on that whoops if i click on that it so it goes blue then when i move into my web page which is up at the top it gets this little greedy type situation and highlights individual elements so i can actually highlight that one file that i want and if i click on that what happens is down below where it's showing the html code it highlights the line of html which is which puts that bit on the screen if you like puts that into the web page and what i want to do is i'm just going to copy that uh notice if i go to copy here i've got options of inner html and outer html so again going back to what we did on our simple scrape when we weren't getting the same information that we thought we were scraping this goes some way to explain that you've got inner html and outer html and what we're interested in is outer html and i'm just going to put that whoops into it into no patch so we can see it better and what we've got down here this is um an html tag it's called an a tag and a tags enclose links so this is the a tag this is the end of a tag and everything in between is uh related to the a tag so what we've got here now that is actually the the inner html and that is what you see on the screen what we've got here in this parameter called href is actually the file it's going to try and download okay now the fact that it's having an html at the front of there it means it's a download relative to where we're starting from okay so i can use this information along i mean if this was just an html i could use it directly otherwise i could use it in conjunction with that information to work out exactly what that fine name is and once i've got that i can start downloading back to our workbook so you can see here i've just written it up i've done exactly that process and i've constructed this fine name and here i'm just going to do exactly the same thing as i did before um all this code is exactly the same as we've seen and if i run that let me just run requests for survivors of fail if i run that um i didn't print out the url but you can see up here that's a i'm sure you got a new version of the file and it looks exactly the same as the other one did but what we're really wanting to do here is download lots of files together and the way we can do that is by recognising the format of these fine names they've got dates in them that's good dates we understand because we know what the next date is going to be we know what the previous date is going to be so in this next little um section down here this is just a little test section and so what i want to do is try and construct um urls corresponding to lots of dates i lots all of the different file names and at the same time um i want to store these somewhere so i'm also going to create um uh the name of the file i want to save and if i ran all that what i would get is this output here so you can see at the end here everything is the same until i've got 16 17 18 19 and that's just a little bit of looping um python code and similarly for the outputs uh the data the output files i'm going to produce their their 16 17 and 80 so we can use just put it because we know the structure of the fine names we can create a little loop which will iterate through and get all of the files and then we can put it all together in our little loop um 16 32 if you're a python person you you understand that means starting at 16 and ending it at 31 um the stem is the same um oh the little f type of the end dot csv to name all the files and then all the code in this loop is essentially exactly the same as we were running before for a single item but this time it's going to loop around and get all of the files and where we're going to put the files the data to 19 blah blah blah which i've already done and you can see here in this um folder i've collected all of the files from 16 to 31 and they're all individual files um or for each different day for the those two weeks in in in march so the next thing we might want to do having download you know half the battle is being to download multiple files but the real problem is that we're probably going to want to put these all together at some point so again we can use this using excel um and here we're going to go to data and we're going to say get data from file and specifically from a folder just make this a little bit okay i can browse where the folder is uh where did i put it data data covid-19 it's a folder at once i'm just going to select that folder highlight the folder and say okay it's then going to tell me what it's found in that folder a list of all of our files we knew that what we want to do is combine these files so the little box down here combine i don't want to do any of just just combine them and load them into my spreadsheet don't do it eventually um it shows you a little preview of the first file notice the three columns that we had in all of the files all the same obviously condition is that they tend to have to be the same um column names eventually um it creates a square which is doing all the hard work which you don't have to any more and then they put all of the data from all of the files into your spreadsheet as a table and you can see what it's also done for us which is very useful it's actually put the file names in here so they um 16th if i go far enough down i get to the 17th and so on so all of our data is in there and then you can start um do whatever it is you might want to do oops insert pivot table um we will have um the names in the columns we want the source name which is actually the file name but in our case it really represents the time the date rather and then total cases down there and this will for each of those dates it will give us all of the the values across from all of the files and if i just restrict this to a couple um one said you i did oops you can pick any ones you want so i thought kinks upon Thames and kinks upon Haw quite diverse locations to let that insert line chart and there you can see a little graph comparing the two together okay so we started off with setting out to download multiple files we've stuck them all together in excel and then we can use the normal charting type thing to draw nice little graphs and you see so so each point in these along here all come from different files that we've downloaded okay moving on because time is pressing uh we're now going to use an api um apis always read api documentation it's very important you know how it works because it might have restrictions on it um like the need for keys and how many calls you can make in a period of time the one we're going to use um you do need a key but it's free um and are some call restrictions back to our demos football api football what we're going to do here is um we're going to start off with a very similar approach of first of all i'm going to show you um what jason looks like i'm just going to import my um libraries at play and we're going to use the the requests module again just as we're doing before and you this time it's a little bit more complex in that in addition to having the url which is all we needed before because the this api requires you to have an api key we need a way of specifying that and to do that we use this headers parameter and in there we say this is our authentication token and that is the value of my token you can all get your own tokens all free there's nothing hard about that and we can also it also includes things called parameters and what we're going to say here is match day equals one and season equals 2080 now i just want to um because i insisted that you all we all do it um let's just go back to um this is a website for the football data org get your free api key api documentation and if you go to the quick start it will show you how to what can be got from it how to use it and it's got examples and things that you might want to do so this one up here the v2 competitions is effectively you don't need a key for this one you can just download it and that's effectively the call i made when i was downloading files before that gives you a list of all of the competitions and you need to know that because the competitions are worked on numbers and if you want the premier league you've got to know what number it is because that's why they give that free the other thing i just want you to point out in this is um if you go to pricing hello there is there is a free option oops at the bottom of it it tells you this is the free option 10 calls per minute okay you have to bear that in mind otherwise it will stop you and other apis will have similar kind of restrictions potentially certainly twitter so back to our code then the lab is this is our simple code if i am just show you jason looks like i know we sort of had it before at the same time i'm going to save the file because you want to limit your time usually using the internet will have you so downloading the file and saving it is usually a good idea so i'm going to print it and i'm going to save it at the same time and if i run that oops you can see what i'm seeing here is nicely formatted jason this indent4 makes it nicely formatted and you can sort of read that and see where things are and see all the information that it's bringing back for you just going to clear that um if you oh the other oops if you want to know what jason really looks like it's that and that's totally useless anyway you can't possibly use that in any practical way which is why all you usually end up with tools which will um which will um format it nicely as i so the next thing what is is use the same approach as we were doing before to get multiple downloads because um you'll notice that in up here i've said match day was worn season i'm going to keep the same but if i go from match day worn to match day 38 i'll get all of the matches from all of the games um or all of the match days okay so here i've got md 1 1 to 39 so again that's 1 to 38 i'm going to look around and each time i'm going to get the data okay now this is i'm looking to run this because as we've just noticed now if i try to do this 38 times and i'm only allowed to do it 10 times a minute it's going to fall over i'm going to get stuck so what we're seeing here is just the results of the prints where i ran it previously and again you can see exactly the same way match day was worn right up to match day was 38 so i've correctly formulated the urls i need and then we just need to run it um now we need to pass jason to extract the bits of information you've got so if you remember up here i saved this file one of the files games.jason and i've got that right so if i look at games.jason this was just the first one i did so it's only one match day match day one but you can see here in here i've got bits of information like the match day and i've got details of the matches um regular scenes i can get the score from that and i can find out who the home team was and who the away team was okay so all of information i need from the the match is available to us including the referees if i wanted the referees yeah so what bits are we going to extract well i guess what we're going to try and to do is create a football uh a league table so what we're interested in knowing is the score but let's start with the match so um what we're going to do is the the response we came back we're going to put into a variable and we're going to use that variable to extract from it the matches and from the matches which we know there are 10 of them we want the first one so that's zero again uh because this is python and then what we're going to do is progressively i'm not going to go through all of these but we're going to drill down and for each of these um we can drill down and extract more and more precise information so that's the the full-time home team score and the waiting course that's the score and down here i can get the halftime scores and i can get who the who the the teams were just as i was um highlighting in in here okay but this is using program coded python to do it and then what we want to do is just save them into a csv file and this is the code that's going to do that uh this is just pure python it's not nothing to do with the api we've already done all the api work that's going to do all that and then i'm going to use this piece of code towards the end right at the end here to actually generate a football table okay and fortunately i've already got these here so if we look at the result csv you can see here these are the items which i extracted to match there the home team away to home goals where it goes halftime scores well okay and what from that the last bit of the code in there actually uh oops the last bit of code generate a table so again that is just simple python stuff so from that data which are downloaded across 38 different api calls i can recreate the 2018-19 premiere league table okay if you're interested in football you may find that fascinating if not well it applies generally to lots of other things as well so finally um apis we covered real web scraping this is where it gets a little bit trickier um we need suitable tools uh in python we've got this package called beautiful soup again in stores part of anaconda you need to know a little bit how hml works but not a great deal the most important thing is the one i've bit i've already shown you when we're looking at the inspector you need to be able to match up on the screen with the underlying html okay and the tool for that is the inspector as i've just shown you what we're going to do we're going to use the um we're going to try and create get some information on tesco stores okay now tesco like lots of other stores they have the store locator if you go to the store locator um you can put in a code like n 13 19 and it comes up with a little map to pick one of these via the store details that will take me to the actual web page now each of the stores has a web page and they all look very much like this little picture image map with the the pin in it and various information about the store itself and what i want you to note is up at the top here in the url for this it's got this number in here oops six seven seven eight okay and again we're going to use that in the same way to find multiple stores so for example i've got some example calls here where i've just changed the number i know those two exist i'm pretty sure that 999 doesn't exist and what we're going to do is we're going to try and extract for multiple stores i.e. by changing that number various bits of information like the name address the geolocation the store type postcode and some other things we're not going to do all of these we're just going to pick a few of them so again i'm just going to run that to import my libraries and just an example of how this sort of works is before we go on to the test yourselves if i run this um no i'm not going to run that for a long time that would be the um the bbc site against standard requests there um um dear me seem to have missed a line out okay i am going to run that just to get the bbc site and that means first demonstration to make much different um having got the information back it's now in r what we do is we create a soup of that using bs which is beautiful soup and we need the text and then we can if we want to look at it we can use the pretty HTML to have a little look at it we won't bother looking at it all we want to know now is that soup contains all the information we need to extract anything we want from the the bbc web page and so we can actually the for loop we can say find find all anything which has a tag of img img is another tag like the a tag and so on and then within that tag i want you to get the value of the source parameter okay and then i just want you to print the url as that's come out so if i run that you can see here just by what's come out dot png dot jpeg what have you these are all of the um the images which are on that bbc page and the one which is probably more commonly done is to find all the of the href so again we're going to use the a tag and we're going to look for hrefs okay and you can see all of the links on that bbc page so when we um i should actually have showed you um what some big HTML looks like so this is a very simple HTML page which i've created and what the underlying h looks like is this so you can see here um the bottom i've got an example of an a tag with an href in it and at the top here i've got heading tags h1 tags some kind of something in it lists an older lists url li list item or well order lists they can be nested one within the other there are different ways of creating tables using different tags and again just looking at the tables you wouldn't know what they were so you need again to use the inspector in order to find this so how do we get the information we want from our tesco so here we are actually picking a tesco school tesco site 6367 which i know exists so if i run that and get that back um i know because i've looked at this before how do i get the title it's in h1 title and actually it's Manchester Oxford Street Express i want the store id as well and again the the issue here is finding the tags and potentially getting the text from the tag or getting some the value of some parameter within the tag and and once you got hold of the idea um this sort of thing is is relatively straightforward the complication occurs is that within an HTML page some of the tag or many of the tags will occur multiple times and it's the real difficulty is isolating what makes the tag unique to be the one that you want okay so here um get the address we're getting all of the h2s i don't have any of the word but then i want the text within h2 to be say address and once i've done that then i know that within the next span which is another tag i want the text and i know that is going to give me the address and the reason i know that is because um i went to the page and i used the inspector i found the address on i found the address on the page and i highlighted it and this is the slightly indentation or the path of of the um hml tags which i followed to find it or it selected for me and then i copied it and just read it off and worked out where it was various other ways of getting it um again here i used um actually i tunneled down to find the html i wanted but then there's other ways of doing it because there is a tag called div um which has this parameter set class is set to address and then from within there i can find the one which has height and prop equals address and text so again it's just a way it just all you have to do is find a way of making what it is you're referencing to be unique okay and then you can get all of it in a similar way the only complicated one is the image that's a bit more involved because the um longitude and latitude which is what we wanted is really only shown as the the the pin on the map so you've got to do something very similar again find the image have a look at what's inside there um get the url out which is a call to um google maps or whatever and then extract the longitude and latitude okay do not run this um and the reason i'm not going to run this is because this is going to get all of the code to four three thousand to four thousand that's a thousand times um i've put in here a sleep of five i don't know if that's strictly necessary but in the case of you don't want to overwhelm the um the server so be polite and just space them out so that's why i can't possibly run that with a far too well but don't worry i've got them already stored in stores here uh yes there they all are which i've done previously so having worked out how to get all our various bits we just need to start putting it all together but one thing we need to be aware of because we commented at the beginning that we're sequentially going through all these numbers and some of them won't really be real stores so what we're doing here we need a way of identifying a missing store okay um and what we've if i if i put that back to nine nine nine and run that test your store okay that's that is what is written in the header or in the title and because it starts with error i know it's not what i want so i'm going to miss them out the next thing we can do is put everything into a nice loop uh three thousand four thousand extract all the bits if you want we're going to put it all into um um a data frame we can look at and then we can um the last thing we're going to do is because and just go back up to there if i look at the store info of csv uh this is the final version so id name address lat and long came directly from the scraping the store type i've extracted from the name and the postcode from the address okay but that is now representative of all of the data that we've wanted to scrape this is just extracting the last two items and then finally what we want to do is put it all together so to potentially from our thousand files i don't have any actual stores or involved there i want to put this information onto a map so if i run that i get my little map whoops if i click on the points okay i'm going to try and play with it on there what i've also done is saved it to teskastores.html so if i go in and have a look at that teskastores.html i've got a nice clickable map which i can expand if i click on an individual one it'll tell me what the store type is and the postcode


Seven easy business ideas to make money in only 18 hours a month. Okay, so you got a little spare time, right? You could turn off Game of Thrones. You could watch one less movie a week or something like that. And you could pick up this 18 hours. And when you pick up that 18 hours, I want you to apply it to the seven things I'm gonna tell you about right now. By the end of this video, you're gonna have the potential to make a lot of money online. So stay tuned. If you haven't subscribed already, please do so. Hit that Subscribe button down there. Turn it from red to gray. And when you do that, a bell's gonna pop up. It's gonna look something like this. And what you wanna do is ring it. Turn on all bell notifications so you're notified each and every time I go live on one of these videos, or I upload one, either one. That way you'll know you don't have to go searching for it. All right, so let's jump right in here. And I wanna tell you about the first one. And the first one, number one, is drop shipping. Drop shipping, essentially, is selling a product that you don't own. Now, that may sound illegal. It's not. It's totally legal. All you have to do is find products out there. And you can do this through AliExpress. You can do it through all kinds of different avenues. I'm not gonna get into all that on this video, but bottom line is, somebody else has a product they wanna sell. You advertise or market that product. You send them the customer and you get paid a commission on it and they ship out the product for you. Now, this is a cool business because you could do it from anywhere in the world. You can set up a drop shipping business and you can travel the world. Literally, I've done it. I was taking six to seven trips a year before the illness struck. And I'll probably get back to that. But the cool thing is, I not only drop ship products for people because we have a warehouse and a shipping department and all that, but I also will market other products and they're drop shipped for me. So drop shipping is a great thing. If you think this is a fad thing or there's no way to make money at it, let me, look, come here. Let me share with you who some of the top drop shippers are in the country. See if you've ever heard of them. Amazon, Home Depot, Walmart. As a matter of fact, my products get sold at Walmart and I drop ship those products for them. So drop shipping has been around for years and years and years. There are some scams out there, but there's also some legitimate ways to do it. On my channel, you'll find a lot of videos on drop shipping. I do recommend that you look on the channel, search drop shipping and I'll show you how to do it. The next one, I love, love, love. It's print on demand. That's number two, print on demand. Now what print on demand is, you sell a product. Maybe it's a T-shirt, maybe it's a mug, maybe it's a shower curtain, whatever it is, and you personalize it or you print it with your logo or you print it with some saying on it. But you don't actually print it. Print on demand, and I have several videos on this channel on print on demand and you can find the recommendations of the best print on demand companies. But basically, once you sell a product, you send the order to this print on demand company and they print up your mug or they print up your T-shirt and they ship it out to the end user. All you do is market it to them. So you don't have to worry about packaging. You don't have to worry about inventory. You don't even have to invest your money in the product until you've already sold it and already gotten paid. That's a pretty cool way to make some money and that's certainly something you can do part time. Number three, you could be a freelancer. Yeah, freelancer is pretty good. You can go to something like Upwork or Freelancer.com and there's tons of them out there and you can offer your services. Maybe you make websites. Maybe you're good at graphic design. Maybe you're good at sales. Whatever it is that you want to do, there's probably somebody on one of these sites that wanna hire you and you could work as much or as little as you want. I have full time employees. I have part time employees and they all do various different tasks for me. I have people in India, people in Mexico. I have people in Pakistan. I have people in the Philippines. So we have people all around the world that are doing jobs for us and I don't need them to come to the office. It's not necessary. As a matter of fact, I rarely go to the office. It's not necessary. This is my office right here. And this, it can travel anywhere with me and that's the cool thing about selling online. Number four, start a pet business. Now, when I say pet business, that encompasses a lot of different things. It could be a pet babysitting service where you actually go to somebody's house and babysit their pet forum or a pet walking service or it could be pet supplies. So many people have pets nowadays and they need so many different products from tangible products to services. I know a lady in San Diego that we met when we first moved to San Diego and she had a pet sitting business where somebody would come to your home and feed your pet and do all that. And some of the people would actually live at your home while you're gone so that the pet always has somebody there. She had about eight or 10 people, something like that doing that for her and she was making a good living. When we left San Diego eight years later she had over 80 people working for her, 80 people. She didn't do any pet sitting herself. She just hired pet sitters and sent them out to different homes and she kept a percentage of the profits. Basically what they did is they charged what they charged and then she charged a percentage of that and took care of all the collections of the money and distribution and all that. And she did it from her home. She had an office staff by the time I left but she did it all from her home. That's another thing you can do. Pet business is awesome. Number five, start a YouTube channel. Hey, what do you know? I'm walking my talk, aren't I? I'm walking the talk right now because I have a YouTube channel. YouTube channels are awesome. You know, YouTube, I really can't say enough good things about them. They provide you a platform for free. All you need is a camera. All you need is to record something and you can upload it to YouTube for free. Why is this so good? Well, a YouTube channel can do a number of things for you. Let's say that you have a business and you offer accounting services. Maybe you'll want to start a YouTube channel and talks about accounting and things you can deduct and things you can't deduct and what you should do. And by doing that, people find you and maybe they hire you for the accounting services. But maybe you have a YouTube channel and you're getting a lot of views, you have a lot of subscribers. You can actually monetize that channel. My channel is monetized. And when they run ads on my videos, they give me a share of the ad revenue. I think that's pretty cool that I get that. And that ad revenue is coming in 24 hours a day, seven days a week. I make money on YouTube. Even if I don't make another video from this point forward, which I will because I want the money to keep going up. But if I didn't, I would still have that income off all those past videos I've done. And I've done over 1,000 videos right now. 1,000 videos, that's a bunch. And that gives me 1,000 opportunities to make money because they can run ad on all 1,000 of those. Now, as I increase that though, whatever my money is on 1,000, what if I had 2,000 videos or 3,000 or 4,000? Do you see what I'm saying here? There's a lot of money to be made. Now, that's not the only way to make money on YouTube. You could have links below your video that are affiliate links. So when people click on them, you get paid. You could have courses or products below your video in the description where people click and buy your products or services. There's so many things you can do on YouTube. Start a YouTube channel. I think everybody should have a YouTube channel. I really do. Matter of fact, when I meet people and they don't have a YouTube channel, I'm like, why not? Why don't you have a YouTube channel? I mean, even individuals, I think it's a great way to get to know people. Have a YouTube channel. Talk about whatever subject you want to talk about. It could be cooking. It could be cleaning. It could be, you know, like I talk about online, making money online, but pick a subject. Start a YouTube channel. You need to do that. Number six, be an affiliate marketer or do affiliate marketing. You could, yeah, there's a couple ways to do this. So let me explain to you what affiliate marketing is. When you send a customer to a business and that customer buys from the business, you earn a commission. It's really that simple. That's what affiliate marketing is. Now, I have a whole lot of different ways of doing this personally, but I am on a site called Share-A-Sale and I can put a link below for Share-A-Sale and you can sign up at Share-A-Sale and if you send people to my website, in other words, they click your link because that will identify you and they come to my website and they buy something, you get a cut of whatever they bought. So that's pretty cool. A lot of people send business to my website and they get paid for sending business to my website. They don't have to package up anything. They don't have to collect any money. They don't have to do anything but just send people to my website. So that's one end of affiliate marketing. The other end of affiliate marketing is what I do where I have products and I offer them out there to affiliates, right? And they send me business, but I can also do affiliate marketing where I send people to their business. Now, I have a list of survivalists, people who buy survival food. I have a list of people who wanna make money online and I can market other people's products to those lists. So that's another way I can make money with affiliate marketing. Affiliate marketing's really pretty awesome. I also do it through Amazon. If you notice in this video below in the description, you're gonna find a bunch of links to Amazon and all those links are my affiliate links.  That's a pretty cool thing to be able to do. My last one, number seven, run ads. Run ads. This is really pretty cool. If you learn how to run any ads on Facebook or YouTube or any type of search engine ads, people need you. They really do need you, especially small businesses. Maybe it's restaurants, maybe it's hair salons, maybe it's nail salons. All these businesses need more business. They don't have the time, energy, effort to go out there and run ads on Facebook. But if you've learned this, if you learn how to run ads on Facebook or YouTube or any of these other platforms, you can approach these businesses. And here's the best way to do this. Say, hey, I'll run an ad for you for two or three days and we'll check out the results. If you like what the results are, I'll go ahead and charge you and I'll run these ads for you all the time. They can't lose that way. You don't have anything really invested but your time. I know a lot of people will charge 1,500 bucks, 5,000 bucks, 10,000 bucks a month for one client just to run the ads and manage those ads for them. Now imagine you're charging $5,000 a month and you have 10 clients. So that's $50,000 a month. That's pretty good. And the way it generally works is you charge a fee plus a percentage of their ad spend. So that way, the more they spend, the more money you make but you also have your base fee. So that's another great way to make money online. And yeah, you could do this 18 hours a month. But here's the truth about this title of 18 hours a month. If you start doing stuff on the side for 18 hours a month and all of a sudden you're making more money than you're making working 40 hours a week on a job, you know what you might do? You might work more than 18 hours a month. You may work 20 or 30 or 40. And you may quit your job. And that's my goal is to get you to quit your job and work for yourself. You don't need an employer anymore. It's not necessary. Most people nowadays have the skills and the talent to make money online. They just don't know it.  You're gonna be pretty happy. I did it, right? I was working a job for many, many years. And one day I said, you know, I don't like my boss. I don't like what I'm doing. Maybe I'll try to sell stuff online. And within the first 24 months, and I have a video on this channel about that, I made over a million dollars. Within the first 24 months. Now the first few months I didn't, the first year I didn't do that well. But once I figured it out, once I learned all the little things that work online, and that's the cool thing about working online. This is the thing that gets me excited. Once you learn how to sell stuff online, it doesn't even matter what the product or service is anymore, you just know how to sell stuff online. And when you develop that talent, when you develop that talent, you're free for the rest of your life.  That's true freedom. Thanks so much for watching this video guys. I really appreciate it. Do me a favor, if you haven't subscribed, hit that subscribe button down there, turn it from red to gray. Don't forget to ring the bell. Turn on all bell notifications so you're notified each and every time I go live or I upload a new video. Thanks so much for watching. Give me a thumbs up if you don't mind. Put your comments below. What did you think of this video? Are you working online now? Do you want to work online? Do you have any problems? Do you have any successes? Put it in the comments below. Thanks so much for watching this video. Talk to you real soon. Hey, thanks for watching my video. Don't forget to subscribe to my channel and click that little bell right here so you can be notified every time I do a new video. Also, click on one of those videos there. Keep watching on my channel.


What does Tana Hill have? Takes the snap, throws left side, going for sharp. Tajay caught it. Touchdown! He took Davis to the woodshed and he whipped him. That was a professional throw, a professional catch. You can't do it any better than that.


Well with their backs up against the wall and win or go home game three at the BSU women's hockey team brought their best in the shutout over Duluth to advance to the final faceoff this week and awaiting the beavers though at Ritter Arena tomorrow the top team in the conference the entire country with scots the two just played in January a pair of extra period thrillers but also too bad your wins but BSU is looking forward to another shot at him and at one game winner moves on we have one game you have 20 you take it 20 minutes at a time you don't have to think about tomorrow or winning two games back-to-back or anything like that you just give it your all when we had our doer situation against Duluth we brought our best game we had all weekend so I have I'm totally confident in our ability to take on Wisconsin be able to play in March the month of March is a very very special accomplishment you know this group certainly isn't going to be just happy to be there great opportunity to to keep on playing you know we know we have a great challenge playing Wisconsin but just really really excited for this outfit this group to be able to experience this and you know our senior class to be able to do it twice is pretty special so one game survive in advance puck set to drop at Ritter Arena tomorrow afternoon 207 if you enjoyed this segment of Lakeland news please consider making a tax deductible contribution to Lakeland PBS


Welcome to a three-part series on factoring. So today we are going to start with the greatest common factor, which is also known as the GCF, which means the largest factor that both terms have in common. So we're going to have to factor some of these numbers. I start with numbers and then I move into some expressions here. So the first one we want to find the GCF of is 16 and 24. Now some of you may be able to look at 16 and 24 and pick out the greatest common factor. That's fantastic, but I'm going to also walk you through then how you can do this by hand. So let's look at 16. So 16 can be factored as 1 and 16. That's always the easiest one to forget. It can also be factored as 2 and 8. It can be factored as 4 and 4. So those are my three different ways I can factor 16. Do the same thing for 24. I'm getting more room on that one. 24 can be factored as 1 and 24. It can be factored as 2 and 12. It can be factored as 3 and 8. And it can be factored as 4 and 6. So comparing these two, so let's go ahead and list them out. So 16, our factors are 1, 2, 4, 8, and 16. 24, our factors are 1, 2, 3, 4, 6, 8, 12, and 24. So comparing these two, it looks to me like the biggest number that is on both of these lists is 8. So the GCF of 8 and 16 is 24. Okay, now if it's difficult for you to come up with these factors on your own, we can use the calculator. So let me bring up the calculator here. What you want to do then is what I described down here below. If you want to factor a number, you want to go to your Y equals menu. So click on Y equals, type in your number. So let's say we want to factor 16, so 16, and then divide that by X. Okay, now what this will do is it will factor this number. So if we go to second table, you will see all these different factors. So zero error because nothing times zero will give you 16, so ignore that one. 1, 16, 2, 8, 3, oh isn't this one lovely, 5.33333. Okay, ignore anything that has a decimal. We're only looking for whole numbers here. Then we have 4 and 4, 5 and 3.2, 6 and 2.66667. So those we're going to ignore, and then you can keep tagging down to get more values. But obviously, so here's 8 and 2, so things are going to start to repeat. So just toggle up and down your table in order to see what those factors are, and that will help you then with your calculator. Okay, right, fantastic. Okay, now let's go to some expressions. So these have some X's in them. So we're going to do the same thing. I'm going to write these out. So let's write out 2X squared. So factors of 2 are 1 and 2, obviously. Okay, so 1 times 2 times X squared is better known as X times X. And you don't have to put this 1 every time, by the way. I just put it in there. I guess to be consistent with what I did up above. Then 8X cubed, let's go ahead and write out what that looks like. So 8 can be factored as 2 times 2 times 2. Just to break that one down into prime factors. Then X cubed is X times X times X. So there's 3X's there. Alright, and then again let's circle what they have in common. Let me do that with a different color this time. So I see a 2 in common with both of them. I see 1X and I see another X. Okay, so my GCF of this one then is 2X squared. So you'll notice we're doing something different with our coefficients than we are with our variables. So with our coefficients you want the biggest number that goes into both of them. With our variables you want the same thing, but you notice it's actually the smaller of the two exponents. Okay, so something to keep in mind for the future. Okay, building up a little bit more, we've got 3X times X minus 5 and 7 times X minus 5. Your first instinct might be to multiply this out. Don't do that. It's already factored. So to list out my factors, I really don't have to do anything. So this would look like 3X. Yeah, let me not even put an equal sign there. That doesn't make any sense. All right, so this would just be 3 times X times X minus 5. I'll put that in parentheses. And this one's just 7 times X minus 5. What do both of them have in common? I see an X minus 5 in both of them. So let's go ahead and do that. If you were to multiply it out, you'd have to refactor it and then you get right back to where you started. That doesn't make any sense. So for this one, the GCF is X minus 5, which is technically a factor of both of them. Okay, all right, moving along. So now let's put this idea of the GCF into some actual, they look more like expressions, so they get a little bit harder here. Okay, so 36X plus 16. So I'm looking at 36X as one term and 16 as another term. So I want to think about what is the biggest number that goes in a bull 36 and 16? So we'll deal with the coefficients first and then we'll look at the variables next. So the biggest number that goes in a bull 36 and 16, leave is 8. So, oh no, 8 doesn't go into 36. It would be 4. So we can factor a 4 out. And you notice this one has an X. This one does not. So these don't have any variables in common. So our GCF is just going to be a number. Okay, so I put my parentheses here. Let me make that 4 a little bit better. Okay, so I got a 4 here. So 4 times what will now give me this 36X? Well, 4 times 9 is 36. And then I need an X there to make my X. And I got my plus sign. Okay, 4 times what will give me 16? Well, that's a 4. And then this thing is now factored. Now, how do we know we're done? How do I know that 8 that I said originally wasn't correct or was not my greatest common factor? Well, that's where you've got to look at what's left inside of these parentheses. So 9X plus 4. Do 9 and 4 have anything in common? No. So that's how I know I'm done. That's how I know I did pick the greatest common factor to start with. So what I'm saying is if you saw a 2 in common at first, you could take the 2 out. But then you'd have to notice that whatever you have left has another 2 in common. Okay, so that's where you've got to always go back and double check what you did. Okay, and another way also to check this is you can multiply this back out like we did with the last set of videos. So you could multiply this thing back through to check our answer. So this is our check. So when you multiply 4 by 9X, you get 36X. You multiply 4 by a positive 4. You get a positive 16. So it checks. Okay, good. Next one. This one's got some Y's in it. So again, we want to look at 9Y squared is one term and 3Y is another term. So let's look at the coefficients first. So 9 and 3, 3 goes into both of them. And let's look at the variables next. I see a Y squared and I see a Y. So they both have a Y in common. It's a smaller of the two exponents. Okay, ask ourselves the same questions. So 3Y times what will give me the 9Y squared? Well, 3 times 3 will give me the 9. And then Y times Y will give me a Y squared. I need a plus sign here because you don't want to lose your terms. You notice here I had back in this number 4, I had two terms and I end up with two terms of my parentheses. This one I have two terms. I better have two terms of my parentheses. So that way when I multiply them back out, they will check. So 3Y times what will give us 3Y? Well, 1. Again, multiply it back out and check it and make sure that it works. Okay, next one. So this one looks like 2X times X plus 5 minus 3 times X plus 5. Again, do not multiply this back out. You're going to be doing too much work. So look at this as the first term and this as the second term. So what do those two terms have in common? What's their GCF? Well, like we did up above, I see an X plus 5 in both of them. So let's factor an X plus 5 out. Now you'll notice I put all that in parentheses and that's because there's two terms with that one instead of just the one that we had up above. Okay, now what do we have left? So if I take an X plus 5 out of my first term, let's cross that off, I have left a 2X. If I take an X plus 5 out of my second term, what do I have left? A minus 3. So that is my factored form. Okay, again you can multiply this back out and make sure you get back to where you started, but you cannot be careful when you multiply this one back out. You may have to multiply both of them out to make sure they are actually equivalent. Fabulous. Alright, last concept here is what's called a difference of squares. So I put over here some perfect squares from the book and how they got them. Okay, so these are kind of good to talk in the back of your mind. You don't have to memorize them but just know. And then they say, so what you have to do is you have to identify the values of A and B. So basically what, looking at number 7 here, what when you square it will give you X squared and what when you square it will give you 100. So that will give you your A, that will give you your B. Okay, so this is technically A squared and B squared. And then 2, so it is a difference of squares. You better make sure you have a minus sign in here and then kind of follow in this pattern. So what when you square it will give you X squared. Well that's just an X. Set up my next set of parentheses here, an X. What when you square it will give you 100? That's a 10. So we know we're going to have a 10 on each one.  It doesn't matter what order you go in. So minus and plus. Okay, then when you multiply this back out, you'll notice that you get X squared plus 10 X minus 10 X. So those middle two terms cancel, which is why this works for a difference of squares. Okay. All right, next example. So this one gets a little bit trickier. So we got to think to ourselves what when we square it is going to give us 4 Y squared. So again, let's take the number and the variable separate for the coefficient. So what when I square it will give me 4? Well, that would be a 2 because 2 squared is 4. Looking at what they have up here on our chart. What squared will give me Y squared? Well, that's just going to be a Y. So that means my A in this case is 2 Y. So that's going to go with the front of both of these sets of parentheses. Okay, do the same thing for the next one. This one's just a number, so that's a little bit easier. So what squared will give me 25? And that would be a 5. 5 times 5 is 25. Give 1 a minus, give 1 a plus. Doesn't matter what order, you're factored. Again, how do I know I'm done? Well, a 2 and a 5 don't have anything in common. So I can't factor either of these two terms anymore. Okay, then the last one. So I see 16 m squared and 49 n squared. So what when I square it will give me 16 m squared? Well, what squared will give me 16? That would be a 4. What squared will give me an m squared? That would be an m. And then doing the same thing over here. So what squared will give me a 49? That would be a 7. What squared will give me an n squared? That will give me an n. Okay, so going to throw these into the equation Oh, did anybody notice what happened with this one? So you notice number 7 had a minus. Number 8 had a minus. What does number 9 have? It has a plus. So this is a sum of cubes or a sum of squares. Not a difference of squares. So this one does not factor. Man, so we just did all that work for nothing. I just wanted you guys to see how this one would maybe, you know, it looks perfectly innocent. I see squares everywhere, but it's got a plus in there instead of a minus. So this one does not factor. If we had tried to factor it as 4 m minus 7 n and 4 m plus 7 n, if you multiply that back out, you're not going to get back to where you started. So, you know that one's wrong. Okay, fantastic. So we'll do more factoring then very soon. Thank you for listening.


All right, I guess that's started Hello everyone, I'm Nate Becktold. I'm an enterprise architect at EBSCO information services and I wanted to give this talk because We are we've deployed open stack. We are in live with open stack We've been that way for a little while now and we're we have an interesting story because we're kind of your average large enterprise, but we're not really at the Scale of a lot of the the hyperscale companies you see presenting at summits So I wanted to go over kind of the practical issues that we encountered of an enterprise our sides getting to live with open stack and how we wound up solving them so EBSCO we are a Discovery service provider we provide kind of online research content to mostly libraries academic Academic organizations If you've gone to if you've gone to college or university, there's a good chance you've used our product We also serve electronic journals ebooks From the back end perspective we serve up about 300 million back end searches per day So we do we do serve a fairly reasonable amount of content So what did we need so around two two and a half years ago? We started looking at infrastructure automation solutions and of the typical story There was interest in public cloud interest in self-service provisioning environments the development teams and full-stack automation So we kicked off the typical investigation effort and at this point in time what we wanted to do is really just focus on increasing the productivity of our teams and Focusing on automation We want to lower our costs by going with open-source solutions And one thing we were really looking for was Finding a solution that integrates really well with other products and allows tools to integrate with it with really good quality integrations So what I mean by that is we have there are a lot of solutions out there where their idea of an integration is a plug-in to a user interface And when you're talking about automation You're looking at API's and those those types of integrations are almost useless So we were trying to find a good solution to kind of abstract our underlying infrastructure and we wound up settling on OpenStack So why did we settle on OpenStack at that point in time? We were looking very strongly. We still are to public clouds and OpenStack gave us a very easy to consume API that Kind of had a very nice methodological alignment with AWS You know, you don't have API here really good API compatibility But from a methodological standpoint and the objects lineup you have an you have an instance aligns to an EC2 instance EBS aligns to a volume and the same kind of actions are capable in both environments. So You're reduced you're solving a lot of the same problems that in both of them A Keep a component for us was abstracting our underlying infrastructure Because if you try to if you try to automate say a traditional virtualization provider what you wind up with kind of are a lot of leaky Abstractions you get a lot of changes in the hardware configurations or say data store storage Configurations that really bubble up to consumers and whenever you wind up changing them it affects your automation So we wanted a stable platform that could really abstract the underlying implementation from our people building automation from our development and operations teams We wanted a standard interface to be able to get compute network and storage And one key that was good for us was that when something integrates with OpenStack and integrates a lot more consistently than a lot of other Solutions, so if you look again back to kind of the traditional virtualization side You'll find things with VMware support or other hypervisors But whether you can actually leverage that solution in your data center is another question How your VMs get IP addresses you do use an IPAM solution DHCP all that can vary from environment to environment So if you're looking from an for an end-to-end solution You know you don't often get as much out of the box as you do with a solution like OpenStack And also we wanted to build an infrastructure as a service platform. It was fit for our live services and without we could hand out to say a diverse set of teams and Be kind of confident in the level of Project level isolation that they wouldn't be able to interfere with each other It's a big tenant for us to be able to say to our consumers if you break it It's our fault or it's somebody else's fault. It's not your fault. Don't be afraid to use the product I much rather tell them that then hand them a product and give them a long pay a large page of Best practices and things they should never ever ever do Otherwise some admin is going to hunt them down and say you shouldn't be doing that Nobody likes that. No one wants to use a product like that So what's our current scale and these numbers are a little bit still a little bit old but right now we have Three separate OpenStack clouds one backing our development resources and two backing our live environments We are approximately well at that point in time 1259 running instances that was a snapshot point in time. Obviously it goes up and down all the time One interesting metric is since we started OpenStack We've had almost five hundred thousand instances created and destroyed since we made it generally available We find that a very interesting metric to track About 68% of the workloads are concentrated on development environments because that's where we have a lot of development environment a lot of development teams and They have had access to the environment the longest and they spin up a lot of infrastructure to support their development efforts Around one-third of our virtualized workloads are currently on OpenStack So our design philosophy We really wanted to build a platform to run our production applications and so what we mean by a platform is we're really providing this platform service and Part of it a significant part of it is OpenStack, but really the platform encompasses all of the tools and integrations you need to bring an application From development and host it in a live environment. So that brings in a whole bunch of other tool sets as well we wanted to Have a solution that was multi-tenant at its core so we could be able to kind of safely give a development team and an operations team a Different project on the same physical infrastructure and not be concerned about them creating security issues or affecting each other in any way All the tools and everything that was part of our platform had to be highly available and it needed to be production grade so Good enough for development, but not for production is not an acceptable for a permanent state in our in our case and there's a lot of reasoning behind that philosophy because developers and People building automation they don't want to code to a state that they can't use in live You get your value in life so if you start making solutions that kind of work in dev and Then you transition to this other completely different solution in live. That's not a good strategy So we basically said everything we put into this platform everything we put into OpenStack Every project has to be something we at least have a path to live with We didn't want to put any toys in it we want to make sure all of our All of our offerings were really built for general purpose and customizes little as possible Kind of provide a menu of infrastructure offering so once you get OpenStack out there suddenly you always wind up getting people saying oh well I want this specific configuration of core and memory and you don't have any your flavor model. Can you please build me in that flavor and That what you wind up if you go down that path is you get a lot of different flavors And you get flavors with application names in them and all sorts of crazy things You don't want to be able to support so we drew a hard line and said everything we put in here Should be general purpose if you found a flavor that you know We don't support and we find a good general purpose for it. Yes, we gladly add it But we won't add one with your application name in it Same thing for all of our storage offerings and everything we've done through OpenStack The other part is we wanted a solution With good safeguards that would encourage experimentation Development against any kind of any kind of platform is a lot easier when your developers can actually experiment on it They don't need to worry about bringing down the entire environment It makes them go a lot faster because they don't need to fully research everything They're going to do they can experiment and try things and find out what works and that gets you a much faster feedback loop So what does our current architecture look like I'll go into I'm going to go into this a little bit more detail later But right now we have OpenStack for our monitoring solution on top of OpenStack one up leveraging Zabrex For kind of an operations Dashboard we use Rundeck. We use that to kind of power a lot of the Automated jobs we built to operationalize OpenStack For some of our dashboards for metrics you use Grafana And then we use all the core components you see on the presentation there from OpenStack And also we use a product called avi software to find load balancer for all of our load balancing with an OpenStack And Here's some of the things that we learned so Then we had quite a few problems to solve and they kind of fell into a Munch of broad categories of skills in training Selecting vendors and integrations the actual deployment of OpenStack adoption and productionizing it to get it to live So first part and you'll probably hear this all throughout the summit all throughout the summit OpenStack skills are very hard to hire They are very hard to hire And if you don't get the direct OpenStack experience Direct OpenStack hire you need somebody with very good and Linux administration experience and that seems to make sense but What usually winds up happening and a lot of organizations kind of like ours initially is Your virtualization team is what winds up taking over the OpenStack POC and you get a lot of skill sets for VMware, but not necessarily Linux administration so that can be a gap that has to be plugged And inexperienced administrators on top of OpenStack can do large amounts of damage When you're running as admin the safeguards are off Safeties are off you can do terrible things So you need to understand the system very well to be an admin How do we how did we kind of get around these we? Decided we're going to develop Through our POC Basically a set a core group of subject matter experts We're going to be very hands-on and become experts and kind of train the rest of the team We decided very early on we we actually attempted to go this way originally But our advice is don't waste learning opportunities by over reliance on professional services If you want to use professional services, it's a good idea to use them as a architectural guidance But it's not a good idea at least in my opinion to have them come in and actually install OpenStack for you You miss a very big learning opportunity for your team and ultimately they just kind of get handed this thing over the fence that they Really don't understand how how to operate or how it was installed to begin with so you lose a very big learning opportunity by Not letting them do the installation the first time means it's going to be a little bit harder But everything you learn is going to be directly transferable to their job is administering it When you look for new candidates look for people with strong Linux backgrounds Networking virtualization Python skills don't look for a direct OpenStack experience if you get it great But we've had we've had a much better Success looking for people with a strong foundation and then training them on OpenStack Give your give your team the opportunity to experiment and learn how OpenStack works people learn by doing and If you don't have a safe sandbox environment for them to experiment on Then they're gonna have to experiment on real environments and potentially cause issues And vendor support so if you go with a vendor you get support then that's going to lower the amount of expertise You need to get to production you always have that that phone to call if some issue arises and you don't know how to Deal with it so it means you can get to production Sooner and with less people at least that was the approach that we took Vendors and integrations, so there are a lot of OpenStack a lot of vendors who integrate with OpenStack today tons of vendors and varying Degrees of quality as to how good those integrations are there are lots of established vendors and Ultimately now with whatever vendor you pick with a direct or indirect integration with OpenStack You need it to be approaching it with the philosophy that everything under our platform is going to give our development or operation teams the tools they need to deploy and manage a highly available application so Our preference we strongly prefer Products that integrate with OpenStack's multi-tenancy model. It makes it really easy to get integrations So and a lot of them will align with the OpenStack project model So their existing OpenStack login will get them access to all of these other services that they need that's a really clean and nice model Focus on vendors who are building for cloud natively rather than trying to integrate and tack it on to a product that wasn't built for it Look at areas everywhere to improve your stack reevaluate all your product decisions There's high value when a product integration is done correctly under OpenStack It can work really well and increase your adoption And also you're never going to know how good an existing vendors integration is until you actually try it There's a lot of hidden landmines with missing support or API capabilities things like we've encountered sender drivers it didn't support snapshotting functionality and from your develop from a developer's perspective if you were relying on snapshots it's a breaking API change So you're not going to really know How good the integration is until you actually get it up and kick the tires So a good case study we when we first approach we said all right, let's let's integrate with our existing load balancing system The existing vendor they had kind of a limited open stack knowledge a to bare bones implementation But we tried to integrate it under Elbas We encountered issues we encountered a lot of issues We actually encountered a bug in their product that wound up a kernel panicking it many times And when we went to go approach support, we got this interesting answer back For now to avoid failover. I would recommend to program the open stack not to delete IPs So what that told us OpenStack wasn't really a first-class citizen for this vendor They really didn't even know what it was from their support perspective Or the fact that the code that was running there was their Elbas driver In our case Elbas v1 at that point in time. It was very limited We didn't think it would cover very many production use cases at all And fundamentally our load balancing product if we weren't using Elbas It really didn't support safe multi-tenancy was really hard to kind of give somebody access and say hey You can't break it. You can only break your own things. I know there were all these shared resources Where if they weren't knowledgeable enough they could cause an outage It was a prolonged evaluation period ultimately Resulting in rejection after about six to eight months and we went back to the drawing board so this is when we brought in Abby and This was a really nifty product because it was built for a cloud from the get-go So our the installation product process was basically deploy the controllers point them at your open stack And it provisions and manages all the load balancers underneath automatically So that allowed us to move really fast we Were able to get basically a production grade load balancing Solution in our development environments sent out a general availability announcement to our develop to all of our teams Hey, you've got access to load balancers all of you have an access to open stack You've got access to load balancing now within a week of actually purchasing the product So we were moving really fast It's multi tenancy model aligns with the open open stacks project model So there is a product There's a tenant inside of the abbey product and aligns with the tenant inside of the open stack product So when we give somebody an open stack project, they've got the ability to Do everything they need from infrastructure and load balancing perspective automatically. There's no Giving them a separate login or giving them access or permissions to different systems. It just baked in And that was really powerful. The other part was because we want this product It has a very strong kind of insight and analytics module to it that the development teams really enjoyed and this helped us Made people want to move to the new platform want to move to open stack and want to move under Abbey because it was naturally incentivizing they really liked the The functionality they were getting so that was a good example where we kind of dethroned the existing vendor in place for one that approached Clouds as a first-class citizen All right problems to solve on the deployment side Deployments on open stack are take a long time and they tend to be very complex The story is getting a lot better than it used to be but for the first time approaching it It is still a very complex problem And a lot of a lot of functionality and open stack isn't ready for production and it's not always obvious What is and what is not especially to a newcomer? So from our experience when you're going to do your deployment one of the biggest things to do is online all of your resources all of your manpower for storage networking data center teams make sure that Supporting this installation is a top priority in troubleshooting issues top priority for everyone in that in that tier or anyone who is allocated to this project because open-stack requires really this tight integration with all of your infrastructure components and You get a very slow troubleshooting And it's very slow troubleshooting the feedback loop basically when you starting countering issues with say networking or storage access An account doesn't work if you have to go back and submit a ticket That is going to make your deployment generally take a very very long time And people are going to lose context while they're sitting here waiting for the storage team or the network team to respond to that ticket So it's going to directly affect the time it takes for you to deploy And it's going to take directly affect the quality of the product at the end of the day because People get there It's hard to keep your mind in the right place when you get those constant interruptions You need to understand what deployment Choices are difficult to change afterwards and make sure you get them right. There are certain ones like core networking drivers some storage decisions Where whether you want an SDN platform or not that you really get one chance to do it right at deployment And if you want to undo that decision you have to redeploy a whole open-stack cloud And there are some that that are easy to change afterwards So it's important to identify which ones are going to be very hard to change And make sure you've done your due diligence on them before going forward And assume it's going to take you multiple tries to get a production ready configuration You're probably not going to get it on your first go On the adoption side adoption is one of the most critical elements to really success in a any private cloud So what I'd strongly recommend to have a really close relationship with your early adopters They're going to help you you're going to help them by giving them access to something that they want and They're going to help you increase the resiliency of your deployment go up there speak to them regularly in person Help them understand open-stack help them learn it and be open to them when they tell you about problems So when you go through that'll help you understand when they say oh, I tried to provision some instances yesterday And they failed now your team can go in find out troubleshoot Why did they fail was it because maybe somebody restarted Nova computer at a certain time or maybe you have an issue in In your messaging tier That feedback early on in that close relationship will help you build a much more resilient product at the end of the day Because they are going to tell basically do the opposite of what a lot of people do to have them tell you every little hiccup This system has early on that will help you a lot Get the deployments into users hands as fast as fast as humanly possible There's only so much you can do in a poc environment once it starts taking real Real workloads and you have real people on it. That's when you start seeing how well it's working and what you need to fix And this one was very important for us. Don't stall getting into production teams Do not want to code to an API that they cannot use in production So yes, we got a lot of early adopters and then we had a lot of people sitting here waiting saying Yep, this is really interesting to us, but until you can give me a path to production with this You know, I'm not going to commit any development time and that is going to limit your adoption so Just merely having the environment in your production environments available will Increase your adoption rate considerably from our perspective was almost exponential growth from the second we put it into live I think our environment started doubling approximately every six months from that point on a literal exponential growth and Early feedback is really critical to this process so productionization Open stack provides a lot of Building blocks, but some assembly is required to actually build a product out of it That's ready for production Monitoring and common operational tasks are really not solved out of the box So open stack kind of gives you all the little Lego bricks to piece together But somebody or something on your side has to actually piece them together to get make them work So one of the biggest things that we found that was successful was monitoring open stack by actually using open stack provision instances provision volumes attach them to instances exercise the functionality Because Open stacks a complex system trying to approach it from the bottom up and actually figure out what the impact of an error is really hard When you have that top-down monitoring when you say alright an instance is failing to provision You know what the impact to the customer is you know at the impact to your users are and Now you have something that you can troubleshoot a lot easier So that's where we had our most successful monitoring Was from that to your that was what would always pick up issues and help us troubleshoot Open stack is really complex finding the effect of a failure is a difficult problem It's important in order to get adoption that you find these issues before your users do if your users start finding these Problems and you don't know about them until they report them to you They're gonna lose faith in the resiliency of the product Another important bit automate completely automate common operational tasks in our case things like taking a compute node or a control node out of service Restarting open stack services some elements of patching any any common operational task automate it in our case we used run deck to To be the the kind of single pane of glass that people can go to to run these tasks But that will lower the barrier to entry for people actually administering open stack and give you a It kind of helps plug the gaps and kind of assemble those Lego box into a cohesive story This one might be a little bit controversial, but I would say open stack ha is complex and it is needed for all environments We went out on our development environment originally We did not do an ha installation and what we found is a lot of tasks and troubleshooting that involved restarting open stack services or Testing out configuration. It was almost always disruptive. It was almost impossible to make a non disruptive change When your environment isn't highly available Also, you want to make sure you have an adequate twist testing environment for any changes you make because An ha environment is a lot more complex than a non ha environment and they act very differently So from our perspective, we said everything has to be ha From the ground up and that includes development and every resource our development environment We treat exactly like a production environment So what did we actually do? What did our process look like beginning to end? So we started out in Havana We did a prototype The usual dev stack single all-in-one machine learn the basics validate direction Keep it a disposable environment because it will probably go down. You will probably trash it So make sure anyone using it understands that in our case We did blow it up many times and then we transitioned to what we called our interim environment Which was we broke apart compute and troll and started getting experience in a distributed environment got feedback from our users and determine the desired configuration and that's when we went with a highly available environment on Juno at that point in time and We treated it exactly like production we announced it was generally available for all of the development workloads and Started to determine the tasks we needed to do to actually take this thing into production and then finally went to Production after we figured out the problems we needed to solve But what went actually happening is we had this big delay that I went talked about early in the presentation between Dev and production It was there was a lot of reasons for it one issue was a critical team member Matt left and we spent too long looking for somebody with an open stack skill set To backfill that position that we lost a lot of time So we wound up Going with somebody with a strong Linux administration experience Because that just set us back way too long looking for somebody with open stack talent We figured out that additional work had to be done for monitoring and operations before we were confident having this host production workloads And a lot of the required skill sets to do that actually weren't part of the open stack team at that point in time So our solution was create a focus squad And what we did is we kicked off a basically a six-week effort with a cross-functional team everyone we needed to get the job done they thought the Requiring it was anyone on the team would have to focus 100% on this project didn't matter what else was going on Director had a good quote where he said set your email to out of office if you have to do that But it had to be top priority for all of the members and the focused effort was incredibly efficient We the feedback loops for travel shooting were incredibly reduced There were very few blocked tasks and when they were blocked they weren't blocked for very long gave us a higher quality implementation at the end of the day and I think within It took us Several weeks to a month to get the first From first installed to get the first dev QA out there and we had both of our live environments done I think in a couple of weeks Over here and that was with the with the enhanced Productivization and more documentation tasks. So it was that focus was incredibly important The focus squad what they did was it created a reliable monitoring solution based on Zabax And we wrote a Python framework to execute checks on open stack. That's the part that does the Does the instance provisioning our reports on health? It one thing we had to do is say if an instance provisioning fail fails It's generally important to identify what part of the workflow failed So if you have a lot of these compound actions, for example, if you want to test volumes if you want to test sender You we have to provision an instance provision of volume attach it But if Nova is having issues and you can't provision that instance We don't want to reflect that as a sender failure. So we had to put in a lot of effort into making sure we Returned the correct component of the failure in these in these kind of compound monitoring tasks We created automated recovery for issues we discovered in the dev QA environment Things like automating compute node of value Evacuation on a fail on a failure. It's essentially one API call to evacuate a compute node when it fails But something has to make that determination and something has to make that API call It's not going to be open stack. So we had to put that into Zavix and We automated some of our failed open stack service recovery Workflow so every now and then an agent or a service in open stack would either lose communication with a message queue Or get into a bad state need to be restarted. We automated those restarts and kind of made open stack a self-healing environment We also increased visibility into the environment and tried to make it as publicly public as possible with Zavix and Grafina Automated common operational tasks with Rundeck And we deployed all of this infrastructure. So we deployed these weren't pre-existing systems We deployed Zavix Rundeck open stack and did all this within the six week time frame So tracking success This one is really difficult because it's critical to getting ongoing commitment But it's really hard to track how successful was your open stack to plan was open stack the right solution for the company And that's really hard to Get kind of a a metrics-based analysis behind so what we wound up doing was we counted some KPIs like how many instance Are how many instances are on it how much resources are allocated to the environment? The number of teams that are leveraging open stack one interesting thing We found was the number of instances that were created and deleted was a very useful metric because it wound up being an Indicator as to whether teams are actually using the environment correctly where they provisioning Elastic resources where they provisioning kind of a fair moral systems or do we have a lot of long-standing VMs And they were just kind of using it for self-service provisioning And I think that's pretty much it does anyone have any questions I'm sorry. I can't hear How big is our team our team? our team is Between three three and four people I'd say three people really proficient on the open stack side For storage we use a net app for storage We found the net app integration was actually pretty high quality. It worked. It worked very well out of the box So we have a lot of net app and we just stuck with it. So that's our sender provider We also did go the route where we do use kind of a shared instances mountain and we do support live migration So that's backed by net app as well That was one of those things where we didn't really want to do it But we did it to get faster adoption, but we put forward we said Live migration is kind of a best effort live migration rather than a guaranteed Guaranteed solution so we do it to minimize the impact of the environment, but it's not guaranteed Thank you very much you