The Cube is where we try to extract information and share it with the broader communities and get out there. And we have a long history talking with Q-Logic and we've had a number of Q-Logic people at previous shows in the past. And so first of all, I believe you're new to Q-Logic, correct? And this is your first time in the Cube. It is, yeah. All right. And if you're extracting information, don't go too hard on me if you don't mind. It's all right. So give us a little background in yourself and where you come from and what brought you to Q. Yeah, so I've been in storage business for a while. So probably 20 plus years I've been in the storage business. And just recently I was in the business with Quantum as an OEM channels guy. Sure. And just within the last few weeks I've started at Q-Logic. Great opportunity for me and really excited about it. Excellent. Great. So we're here at HP Discover and obviously a very strong partner. We were just talking about virtual systems and the FlexFabric Virtual Connect, which is built on Q-Logic technology. I wrote last year that it actually killed the protocol wars. It's fiber channel and ethernet are converging together. That's built on Q-Logic technology partnered with HP. So what are you doing here at HP Discover? Where's the relationship with Q-Logic and HP? Well, it's a great opportunity for us just to network and work with a lot of the sales and the channel partners that we work with every day. I think Q-Logic is a bit of an unknown secret underneath a lot of the chemistry that makes the HP products great. So all the things we're focused on around FlexFabric, the things we're doing around virtual connect, these are all projects that we've worked on for years with HP, a lot of development work, cooperative work that went into those. And I think one of the things we want to try and do is just make sure that we're doing everything we can to make that an extension of their sales efforts and make them successful. Right. So as you said, Q-Logic is one of those embedded pieces. Q-Logic also has components. So most people, if they're familiar with Q-Logic, think about fiber channel host bus adapters, something that might not be the sexiest thing, but that device driver doesn't work. My storage, my server can't talk to each other. So a lot of value, mission critical, running in all the data centers there. And Q-Logic, many years ago, through the leadership of HK to PSI, really pivoted to the embedded space. But maybe from the sales standpoint, where is Q-Logic from standalone products, embedded, channels, OEMs? Where's Q-Logic in the ecosystem today? Yeah, you mentioned something interesting. It's kind of the heritage of the company. It's one of the things that attracted me to Q-Logic. This is a company that's built on a technology basis. A lot of engineering brain in this company. These guys are very capable, very sharp engineers. And that's kind of our heritage. And it's one of the things that attracted me. I think in our business, we need to be technically differentiated. We need to be pushing the envelope a bit. And that's one of the things that I think has helped us develop such a strong position in fiber channel adapters, what everybody knows Q-Logic for. I think the interesting thing is there's more to the product line now. We've done a lot more in recent years, stuff that people are not as aware of probably. So a lot of the converged infrastructure that we're focused on right now, it's a big extension. As people look for flexibility in the protocols and the way that they connect their storage, we see a big advantage there. We've got a strong history in storage. We know storage better than anybody. And as those protocols move forward and as people need flexibility in the environments, as they build fabrics out, it's one of the places where we offer a big advantage today. Yeah, and I have to say, I believe HP is a very strong partner from Q-Logic. Last year, John Furrier actually broke the news when there was the big split between HP and Cisco over switches and Q-Logic was the beneficiary from the fiber channel switching standpoint. So you talked about some cool and interesting technologies.  Well, I think in relation to what happened a year ago, those things are happening throughout the industry today. There's still a lot of, the storage market is consolidating. You've got a lot of people that are becoming part of bigger companies. I think one of the things that you'll always see from us is that as we develop products and as we move forward, flexibility is going to be a big part of what we're focused on. I think if you look at between left-hand and three-par and EVA and MSA, all the storage technologies that HP represents in the market today, giving customers the flexibility to adapt and use tiers of storage where they need it and use one fabric and have a common connectivity structure with those products is really important. And I think that's something, looking forward, we look at things we're working on technology-wise. It's to push that forward and do more with it. If we're able to give people, I think the big things we're focused on are gonna be performance and scalability. Obviously it's important in a storage world, especially as people put more on less storage systems. It's gonna be the flexibility of those systems being able to adapt to different protocols. And simplicity is a big thing. Simplicity, management, all those things. So Rob, I mean, we talk about convergence. Q-Logic plays at the storage level, the server layer, and the network layer. Network's really kind of the core. So final question I have for you is, big trend waved towards Ethernet being more prevalent in storage. You're the sales guy there. How's Q-Logic gonna take on the likes of Intel and Broadcom and your traditional competitors like Emulex? How does Q-Logic come through this transition being a market leader? Yeah, I think the thing you look for is where's our heritage number one? We're a strong storage player. We know the fiber channel and the storage world extremely well. I think converged infrastructure. There's a lot of pluses to that, but there's also challenges because you've got critical data expertise and experience in storage is important. Making sure the stack's proven that you've got technology that's been through the battles and is hardened, that's one of the advantages we bring to the market. So as much as storage as a part of that converged infrastructure, we see a really strong role for Q-Logic in there. And I think that's of all the things, all the data we're moving across network, storage is one of the toughest. It has the highest requirements, the least tolerance for interruption, all of those things. The fact that we have strength there is going to be an advantage to us. I absolutely agree. Storage is mission critical. Can't loot, no unavailability, no data loss. So Rob, first of all, best of luck in your new role with Q-Logic. You are now a Q-Balloon, so thank you for sharing with the community. That's my first time on the Q-Balloon, it was fun.


Hi everyone. Thanks for joining us. It's 4 p.m. I'm assuming people can hear me. If not, then please, similar to last week, let me know very, very quickly. Excellent. I'm Dermot McDonald. I'm a research associate at the UK Data Service with my colleague Julia. We've been developing some live coding demonstrations, particularly for social scientists, but for people from a wide disciplinary background. Basically, if you're new to programming, new to web scraping, new to Python, we've got stuff to help you. So we've got some live coding demonstrations today. We're on week number two. So we're focusing on collecting data from the web. So we're going to use a technique called web scraping. So we've got about 25-30 minutes of a live coding demonstration that you can follow along with as well. So Julia, my colleague will post the link into the chat. I'll see if I can do it as well. No, I can't. So Julia, if you could please post the link. Excellent. So if you look in the stream chat option on the right hand side of the stream, you'll see a link. If you copy and paste that link into a new tab in your browser. So I'll do it here. That will launch the Jupyter notebook containing the Python code we're going to use today. Fantastic. Excellent. So it's working for me. You should see something similar. It will take maybe 10-20 seconds to load up. So yeah, if you have a second screen or if you want to split your screen, then use this link. We'll give it a couple of seconds if you want to get set up. And what you do then is you can enter slide show mode. So you can see I'm hovering my cursor just up here. It looks like a bar chart. Basically the whole document reads like a notebook. So it's an electronic notebook. It contains some text and it contains code and it contains the results of code as well. So you can work through the document as it is, but it's easier to enter slide show mode. It just cleans it up a little bit and we can begin. Excellent. So just some introductory material. So today we're going to focus on using Python to collect data found on websites. So this is more colloquially or informally known as web scraping. So we're trying to scrape data from a website. And we're also going to try and just develop your computational thinking skills through some coding examples. So specifically when we say computational thinking, we're not really talking about technical ability. We're really talking about defining and solving problems. So it's using computers to try and solve human problems in a sense. And the problem we have is there is data that's really interesting. It's found on a website. One way of getting that data is to copy and paste manually. That doesn't really scale up. It takes a lot of work on your part. So we're going to look at how computational methods can help us collect data. If people are getting... Oh, I'll just take a quick... Yeah, so people are getting a binder inaccessible. That's strange because it is working for me. This is not my personal copy. Let me just try again. So I'll move it onto my screen here. Copy and paste. It seems to be loading for me. So I'm accessing that notebook through the web. It's not on my machine. Yeah, keep posting messages, unfortunately. Oh, I ignored the error message. Okay, I haven't seen the error message. So somebody's just posted a solution. If you ignore the error message, perfect. People are figuring it out. Excellent. Thank you. You're much more tech-literate than I am. Fantastic. Well done, guys. Okay, back to the presentation. Yeah, so the two aims, we'll look at how Python can be used to collect data from the web, and we'll just try and define what we're doing in terms of problem solving. So briefly today, I will make myself a little bit smaller so you can see the slides. And we'll make this a little bit smaller as well. So this has been designed as a kind of self-taught or self-directed teaching material. I'll go through almost all of it today, but there's a little bit more in the notebook that we don't cover. So you've got the link. You'll be able to work through this yourself. It takes about 30 to 60 minutes. Today, I'm assuming no knowledge of Python, no knowledge of web scraping will take you right from the beginning. But we did do a session last week specifically on Python. So you may find it useful to work through that at a different time as well. It's for anybody, so thank you for tuning in. And yep, today you'll understand the general approach for collecting data from web pages, but you specifically will be able to use Python for getting a web page, parsing its content, so that's understanding the structure, extracting the data you need, and then saving that data to a file for future use. So if you're new to Jupyter notebooks, as I said, it contains a lot of text. It contains results of analysis. But the meat and drink of it is the code. So you can run the code that's contained in the notebook. So basically all you need to do are look for cells that on the left hand side have this I n square bracket symbol. And then all you have to do is run or execute those cells. So now that we're in slideshow mode, I'm going to use the keyboard shortcut. So for me, that's shift and enter at the same time. Or you can use control and enter as well. So here we have some really basic Python code. So I do shift and enter, and it executes the code. So it asks me for my name. I give it my name, and it gives me a little gentle message to say good luck with what you're doing. So straight into it, what is web scraping? So it's a computational technique for capturing information stored on a web page. Computational is the key word, because as I said, you could manually go to a web page. So you could open your browser, put in the web address, and then when you see the content appear, you could start copying and pasting. You could highlight the text, you could right-click, copy, etc. And then maybe you paste that into an Excel file or a Word document or a TXT file. I'm sure you can understand the considerable disadvantages collecting data from the web manually would bring. There's the potential for human error. It's so time-intensive. One of the examples we'll see today is to collect data about charities. There are 160,000 in England and Wales alone. If I wanted all of their data on a certain topic, yeah, it's not really going to happen. So what's the general approach? So no matter what programming language you use, be it Python or Julia, PHP, there's so many that you can use, there is a general logic, there's a general approach for collecting data. So the first thing we need to do is, well, we need to decide on what we're interested in collecting. So let's say we can think of a web page. So the UK Data Service, for example, has its website here. So that's the location of the UK Data Services website on the web. So this is known as a link or URL or web address. So this is the location of the UK Data Service website on the internet. So we need to know that. Once we do, then we need to find information that we're interested in on that web page. So for example, there might be a paragraph that we're interested in extracting. There might be a table of statistics. There might be lots of images or videos. There may be links we want to collect on a web page. So we need to know where that information is stored. And that's a visual process, as we'll see. There's not really a computational or programmatic way of doing that. You do manually inspect the web page. So you need two key pieces of information, the web address of the website and the location of the information that you're interested in collecting. So then we're into the doing stage. So what we want to do then is request the web page using its address. And we're going to do that in Python. And then we're going to parse the structure of that web page using Python. So then we can actually work with its contents. And I'll explain what that actually means when we do it in practice. So once we actually parse the structure of the web page, then it's about getting the information we're interested in. And most importantly, we want to save our work so we don't have to redo the web scrape as well. So these kind of six basic steps that we've laid out is called pseudocode, or in a very simple way, it's an algorithm. You know, it's the rules we need to follow in order to collect data from the web. So we're going to look at two examples today, one real social science data example to do with charities. But to learn the techniques, we're going to use a much simpler example. It's a real website, but it has been designed for practicing web scraping. Okay, so where does this, where can we find this website? And so the web page is here at this web address. We can take a quick look again, if I was to open this in a new tab, I can get a look at the web page. So it's just a really simple text web page. It contains an extract from Herman Melville's classic English literature work, Moby Dick, and it's just I'm not sure which pages is from but it's certainly a section from Moby Dick. We can actually load that website in Python. Oh, yes, and if you're wondering how to I should have said at the beginning if you're following along on the binder link, it's the space bar to go forward and it's shift and space bar to go back. So I've just went back and space bar goes forward. So apologies. You can use the directional keys, but it's a bit tricky. So as I said, if I want to take a look at that website in Python, I can load in the module I need in Python. So there's an iframe module that we need. Then I can use the iframe method. I tell it the website I want to view and I just want to adjust the width and the height. So for example, just to show you that we can edit this, you know, now the width is not as large. So it's more difficult to see the web page. So this is a useful feature of Python and Jupyter notebooks that you can embed videos and images and other websites into the notebook. So good. So we've done the first thing we've said, okay, we know where the website is, we know its web address. Now we need to find the information we're interested in. In this example, it's really simple. We just want this big long paragraph here. This is what we want to scrape. So we want to ignore this header here and we just want the text. Okay, so we can visually see where that paragraph is. But that's not really what we need. So what we need is the underlying code that powers a web page. So that's called HTML. It's hypertext markup language. And that's basically a very simple programming language for creating web pages. And what HTML does is it describes the structure of a web page. And it does that by defining a series of elements. So paragraphs, tables, headers, images, links, all of these elements together make up a web page. And then each element is uniquely identified by what's called a tag. So a paragraph is identified by a p tag. A table is identified by a table tag. A top level header is identified by the h1 tag. This link here has just some extra quite useful information on HTML. So when you're trying to find information on a web page, it's not what you visually see. And when you load the website. So the fact that you know, this is the first thing I see doesn't actually tell me where that header is on the web page. What I need to do is actually look at the source code. So we need to do this manually and visually. So I'll open this in a new tab just to make it clear what I'm doing. So what we need to do so I'm using Firefox as my web browser. So what I want to do is I want to right click anywhere on this web page. And I want to go to view page source. What that does is it shows us the underlying code powering this website. And so this is what HTML looks like. It's a series of tags. So we've got a HTML tag saying this is the beginning of our web page. We've got some head tags. Usually within this you'll see metadata about the page. And for example, if you have Google Analytics tracking views of your page, there'll be some code here about Google Analytics. But for scraping information, what we're interested in is everything within the body tags. So there's an opening and a closing tag. And within that is the information we need. So we can see the header here, Herman Melville, Moby Dick, it's an H1 tag. And then the information we're interested in scraping is contained within firstly within a div tag, and that means section in HTML. And then the actual information itself is within these two p tags. So it's a really simple webpage. Everything we need is within this p tag here. So let's see how we can use Python to get that information. So we visually inspected. If you're using Chrome, it's similar, it's right click and view page source. Safari is a little bit different. I think there's a developer option you need to enable first. So I have a link here of instructions if you're on Safari. So again, I've shown you the actual source code as well. But I've just put it in the notebook. And here's a full HTML webpage. Excellent. We have our two key pieces of information, the web address and the tags on the webpage relating to the information we want to collect. So let's request the webpage. So we know how to do that using our browser. I've just shown you you copy and paste the link in you press enter, and the webpage appears in your browser. Python can do that for you. But there's just one quick preliminary step is we need to load in modules that help us do the web scraping. So Python comes with lots of functionality built in. So for example, if you wanted to do calculations, you know, right out of the box, and you know, Python knows what multiplication is. If you wanted to, you know, print a message, we saw that earlier. And you know, the print command already exists in Python. But lots of other commands and functionality need to be loaded in to Python. So we need these three key modules for scraping a webpage. So we need the OS module, which is good for working with your operating system. So it's to navigate, you know, between file folders to create new files, these kind of actions. For scraping webpage, we want the requests module. So that mimics, you know, copy and pasting the link into your web browser, and then beautiful soup, which we'll call soup. And that's for actually parsing the webpage. So soup tells Python, hey, you're not just working with text, you're working with an HTML file. And so I run this code, and then I print a little message just to say, you know, the modules have been loaded in. So let's actually request the webpage. So I need to tell Python, okay, where's this webpage found. So I define a variable called URL. And its value is the link to the webpage. Next, I use the requests module. From the request module, I use the get method. So you can see it's quite, you know, easy to understand in Python. If I want to get a webpage, I use the get method. What am I getting? I'm getting this variable here, which earlier we've defined as this web address. And then there's a little option to do with the get method, which is if we get tried, if the website tries to redirect us to the correct website, we allow that, but it'll work. It'll work without that so we can get rid of it. And then the last line of code here basically says, tell me if the request was successful. So a 200 code in response to requesting a webpage means, yep, we've sent you the webpage, everything has gone as expected. I'm sure just by using the web, you've come across 404 errors when you try and request the webpage. So that's an unsuccessful request. That means something on your end has gone wrong. So you haven't done it correctly. If you get a 500 or a 501, a 502, there's something wrong with the website itself. So maybe it's down for maintenance. Maybe there's an issue, etc. So we're looking for a 200 code. Good. So the request has worked, but you may be thinking, where's the webpage? I thought you said, we've requested it. So we have, but first we can just take a quick look at some of the metadata associated with our request. And so there's an attribute of the response variable called headers. And this is what the webpage gives back to us when we request it. You don't really need to know much about this. And there's a handy date field, which, as you can see here, tells you the last time that webpage was updated. And the content type can be interesting. You can see that we're working with a HTML file. It's possible to use Python to download files from the internet. So this might say something like Excel, or it might say JSON, or it might say a different file type. It might say PDF, for example. But in general, you don't really need to know much about the metadata. It's just important to know it's there. It just proves that the website has communicated with you. And it's talking back to you saying, here's some information about your request. So what's the actual content of our request? So we've successfully gotten the webpage. But where's the content? Where's the title? Where's the paragraph that we requested? So we've got something called the text attribute. As you'll remember earlier, we stored the results of our request in a variable called response. Now we could call that variable, whatever we want, scrape underscore results, chicken, it doesn't matter, it's just a variable name. But it stores all of the results from the request. So it stores, you know, whether it was successful or not, it stores the metadata as we've seen. And it also stores the webpage itself. So this is what we've actually requested. So give me the HTML behind that webpage. So we can kind of see, you know, the structure of the HTML, you know, we can see the opening HTML tag, you know, we can see the heading tag and we can see the paragraph tag. But it's a bit messy, as you can see. So Python understands that it's text and we can view the text. But it doesn't recognize it as HTML. And this is the next crucial step. This is what we need the beautiful soup module for. Beautiful soup then tells Python, Hey, this isn't just a blob of text. This is actually a webpage. And then once you know it's a webpage, you can start extracting the information you need. And this is what we call parsing the webpage. It's just basically, you know, identifying the structure of the webpage. So we're going to use the beautiful soup module and really good open source module for Python. And it provides a systematic way of navigating the elements of a webpage. So let's see how it works in practice. It's a one line bit of code. And so we're using the soup method. And into it, we're saying, Hey, take the text that we requested and parse it as HTML. So tell Python we're working with a HTML object. And then we're just going to view the new variable that we created. And voila. So we can see that now Python recognizes that a it's a hierarchical structure. So HTML is hierarchical, there's opening and closing tags, and the content runs from the top to the bottom. So now we can see Python understands that. You know, if we went back just to the raw text, you can see that Python doesn't understand that there are tags, it doesn't understand that there's a hierarchy, et cetera. But now that we've parsed it as a beautiful soup object, now Python knows we're working with HTML. So that's not just an academic, you know, piece of knowledge. Now we can actually use Python to navigate and extract the information of interest. So you'll remember that what we want is the text contained within a paragraph. And a paragraph is identified by the P tag. So what we're going to do is use beautiful soup. We're going to use the find method. And we're just going to find the P tag on the webpage, which we've called soup underscore response. We're going to save the results of our search in a variable called paragraph. And then we're just going to view the contents of that variable. And again, now we can see, okay, we've, you know, cut out the rest of the HTML. And now we have a variable called paragraph that just contains the P tag and the information within it. So we're almost there. So now the next thing is we know that there's a P tag we're interested in. How do we get the text from within the P tag? So again, we use the find method on the super response variable to capture the P tags. And now we need to extract the text like so. So again, the new variable we created paragraph that has a text attribute. And so I'm just going to take the text from within the P tags and store it in new variable called data. We'll just take a little look at the data right now. And that's actually almost all there is to it. Certainly with this example, so we've, you know, we've requested the webpage. We've parsed it as a HTML page in Python. We filtered out all the other elements of the webpage just so we have the paragraph we need. And then we've extracted the text contained in that paragraph. And as you can see here, now we have a new variable called data, which just contains the information from that paragraph. So we might want to save this for future use. So we'll do that very simply. We'll just create a brand new TXT file. And the first step is just create a variable that defines where the file is located. So basically, I'm just going to save the file where it currently, where we currently are. So this script is running somewhere online. And basically, I just want to create a new file at the same location as this code. Basically, in the same folder we're working in, create a new file called mobidickscrapedata.txt. So I want to do is I want to open this file that I've defined here. So open the file, open it in write mode. So write means save in computer science language. We just want to call that file F for short hand. That's all. And with that file, write the data to it. So we didn't get any outputs. We haven't asked Python to print anything to the screen. So what we need to do is check if this actually worked to see if I'm not just tricking you. So the simplest way to check is whether A, the file was created and B, the actual contents were written to it. And this is a good test here because I'm currently working with the online version. So the file is not on my machine. I can't go looking for it on my machine. So I need to actually use Python to go find it. So there's a method called listdir. So list directory. So basically this lists all the contents in the current folder. So this will check if the txt file was created. And yes, yes, that has. So that's really good. So we know Python has at least created the file. But remember the next step was opening the file and placing the data into it. So we're basically just doing the same as saving the file, but this time opening the file, but this time in read mode. So in importing mode, read in the contents of the file, store them in a variable called data and then print the variable data. And voila, we get back to the original data that we scraped. And again, just to show that with, you know, Python variable names, you know, can be whatever. I can store the results in something called this. So you've seen I haven't created that variable before. I'm not, you know, I'm not showing you what I've made previously. Again, I've opened the file. I've read the contents of the file, stored them in this variable. And I just want to look at the contents of this variable here. And I can just zoom out a bit. You can see it's, you know, it has collected. I have scraped all of that paragraph. I have saved it to a file. And I've had and I've been able to import the file once more. And voila, that's how we have successfully scraped a web page. So now for the rest of this, for about five or 10 more minutes, I'm going to show you a slightly more complicated example. It uses all the same techniques that we've done. But in reality, you know, trying to collect research relevant data from a web page, it carries a few extra tasks, basically. So the web page we were just working with was very simple. It had one paragraph. It made it very easy to find the data we were interested in.  There'll be lots of data spread across the web pages that we want to collect. Or there might be multiple pages we want to scrape as well. We had one p tag, but there could be multiple p tags on a web page. We only want one of them. So how do we filter out all the irrelevant p tags and just get to the one that we actually need. And of course, there's lots of other tags we could work with as well. And lots of other, you know, potential issues. What happens if the web page is down? Does your script break? You know, can you control for that? Et cetera. So we're going to look at a piece of social data that makes it slightly more complicated to collect data. So for one of us, myself, web scraping is a tool for collecting data that I just can't get any other way. So I'm interested in UK charities in particular. And so I want to scrape data from charity web pages. You know, that data is not, you know, released in a file. It's not on an open data portal. For example, some information about charities I can only get from a web page. So it's a necessity to create research data for myself. So for this specific example, I'm interested in which policies a charity, you know, reportedly has in place. That sounds a bit dry. Why it's interesting to me is you can link that data to observed organizational outcomes. So we know which charities go bust. We know which charities, you know, break the law. That's all open data. So we've got an open data set. And if we can collect information, you know, on policies, for example, then we can explore correlation.  Or maybe the purpose, the point of having a risk policy in itself might be a big step, but it forces you to think about risk, maybe those charities last longer, for example. So for my research area, there is a real need to collect this data. So we go through this example, I'll go through it quicker than the previous one because we're employing all the same steps, but I'll just stop at certain points where you'll see something different. So again, we need a couple of extra modules, but really the same ones as before. We need requests and OS. We need beautiful soup. And this is just the iframe module for embedding a webpage. So the only new module is called pandas. It's a data set module in Python. It's similar to the tidyverse in our, for example, if that's what you're used to using. So what we want to do, we want to identify the webpage I want to scrape data from. I'm going to focus on an incredibly well known charity, Oxfam, so we're going to try and get their data. And so basically the web address looks like this. So here's Oxfam's public data on the Charity Regulators webpage.  And then some information about what the charity actually does. So here's its charitable purposes. So the relief of poverty overseas aid and the groups that it purports to help and the type of activities it engages in. The information I'm particularly interested in is in the documents tab and way down here. Here we go. There's a little section called policies and under which are listed the policies this charity says it has. And again, you know, we can use Python to actually embed that webpage as well. So we know where it is visually. There's a documents tab. There's a heading called policies. But again, in terms of the source code, where is that list of policies? And it's under a section called PCG dash charity details, etc. So there's a div tag. The div tag has an ID or a class attribute with this value here. There's a header three. And then there's a series of span tags. The text within contains the policy we're interested in scraping. So let's do this bit quickly. We now know how to request a webpage. It's the exact same as we've done before. And, you know, define a variable capturing the web address, request the web address, and just check if it's successful. Fantastic. It was a successful request. Good. So Python now has the text of that webpage. Now we need beautiful soup. I'll just make this a little bit bigger. Again, we want to take the text that we requested, parse it as HTML. And because this is a more detailed website, you'll get much more information. So, you know, the webpage itself is more complicated. There's more code, as you can see here. It's not oceans of code, but you know, it's a more, it's a more complicated webpage. But again, we can see with Python, we've requested the webpage, and we've parsed it using the beautiful soup module. Okay. So we want to extract the policy information. And this is where it gets a little bit more complicated than we saw with the previous one. So we know that there's a div tag containing the policy information. We know that the div tag is identified using the class attribute and using this value here. The problem is there are multiple div tags with this class attribute. So we can't just use find. We have to use find all. So first, we need to find all the div tags, and then we need to filter through them. So this is similar to what we did before. Before we just used the find method. Now we're using find all. So find all div tags that match this condition. So find all the div tags where the class equals this value here. And then I'm just going to view how many sets of tags are returned. So there are seven div tags with this class attribute. And so one of those we know contains the policy information. In case you're wondering what the other ones are, I mean, this is one of the sections. So one, two, three, four, five, I think six and seven. And I think this has a different class attribute. So seven div tags that we need to work through. If we want to actually see the div tags themselves, remember, we've stored the results in a variable called sections. We can take a look at that variable itself. So we can see that there is a list of div tags. So the first one I found, yep, refers to the governing document. And the next one it finds is yet the one called other regulators, etc. So we know we've got a little bit more of a challenge, which is basically finding the correct section. So what we do is, okay, we define a search term. So I know that information I want is contained under a header called policies. So what I want to do is I want to define that as a search term. And then I want to loop through all of the div tags. So we have seven div tags. And for each tag, I want to check if that search term exists in that tag. So that's what we're doing here. So for every section in sections, again, this could be for for chicken in sections, it can be called whatever. So this is the variable name we defined earlier. But this is just the placeholder. And I can show you an example of what I mean in a second. So for every div tag, if the search term is found, basically tell me which div tag you founded in and show me the location of that div tag. If you if you don't find the search term, just keep going through the loop. So for each div tag, check if the search term is found, if not, move on to the next div tag. Probably easier if I just show the results of this. So I created a new variable here called policy section. And it contains the div tag I need. So that piece of code up here loops through each div tag. And when it finds the correct one, it creates a variable here containing the information. So I'll just spend a quick moment explaining what I mean by a what I mean by looping through a list.  There we go. Okay, I'm just trying to create a new cell. Let me just exit slideshow mode a second. Yeah, here we go. So I'll just show you really quickly how we can use a list. So let's say I have a list of fruits. So let's create a very quick list. So I have a list of values. So let's call it apple pear. And I might misspell this. And then yeah, there we go. Okay, so I've created a new list called fruits. And I can have a look at its contents. So you can see this list has three items. Three words are contained in this list. So if I wanted to figure out which fruit is listed first, now I know I can visually see that, but imagine the list had hundreds and hundreds of items. So I could say, let's comment this out, fruits.index pear. And that returns the value one. So what I've asked Python is tell me where in the list the value pear occurs. And that might seem like a strange result that it says one, even though I can visually see it's the second element in the list. The issue with Python is it starts counting from zero. So zero is the first item in the list, one is the second item in the list, two is the third item in the list, etc. So that's where Python gets a little bit tricky. So for example, where is Apple in the list? It's at position zero. And this is what's going on here. So we're looping through each of the div tags. And once it finds the correct search term, it finds the location in the list where that search term is found. So for us, it's found in position five. So it's the sixth div tag in the list of div tags that we downloaded. And then we extract that div tag based on its location. So I realized that's slightly, slightly trickier element that we hadn't that we didn't need to do previously. But now we know the correct div tag. So we now have a variable called policy section. And we know that that contains the information we need. Okay, so now we have the correct div tag similar to before. We know that that div tag contains span tags. Each span tag has the policy information. So again, we want to loop through each span tag. So for every tag that we can find, we want to extract the text and save it in a variable. And then what we want to do is we want to combine the charity name with the individual policy and then we want to append this information to a blank list up here. And again, once we run it, I think it will show much more clearly what I mean. So it gives me what looks like, you know, a long format data set.  So a row is not uniquely identified by Oxfam because Oxfam appears multiple times. A row is uniquely identified by the combination of the charity and the policy in question. So that's what I mean by a long format. So you can see we've gone through each individual span tag. We've picked out the text and stored it in a variable here. We've combined the charity name and the policy and we've added all of that into a blank list up here. So that's really good. We know in Python that we have the information we need. As social scientists and researchers in general, we probably want to actually put that data, you know, into a file into something a bit more friendly that we can load into SPSS or Stata, for example, or if that's what you're using. So we want to save the results. So the new thing we're going to do is use the pandas module. And so that's similar to the tidy verse in or it provides methods for working with data sets in Python. So what we want to do is we want to use the pandas module. We want to use the data frame method. And what we want to do is create two columns and so two variables called charity name and policy. And under each of these columns, we want to put the list that we created earlier. And again, it's probably much easier to see this in terms of the results. So now we've created something that looks a little bit more like what we're familiar with. So we've got a data set. So row one, we've got the charity's name and the specific policy itself. So now we want to take that data frame that we've created, and we want to save that to a file just like we did earlier. Exact same process, a variable that stores the location and the name of the file we want to save to. And this time we don't have to, you know, open the file and do, you know, a couple of lines of code. We've created a data frame and we can use the two CSV method. What are we doing? So taking the data that's stored here and putting it into this file and right here, I will just do the same steps as before. Check whether the file was created and check whether the actual contents exist also. So I'm going to list all the files and folders and we can see here. Yep. And hopefully, again, you'll believe me that if you go back in the code when we run, we run that command previously, you'll have noticed that that file didn't exist. So again, I'm not tricking you. It's not all set up to work. This is, you know, live coding. We are, you know, scraping things and saving things in real time. So we know the file exists. And again, we can use the pandas module. It's got a read underscore CSV method. So again, go to this file here, read in the data. And these are just some options about how to read it in. You don't need these. And here we go. So read in this file and store the contents in a variable called data. And here we are back where we began a moment ago. Voila! So what have we learned? I was going to say what have I learned? What have we learned? Excellent. So those were two fairly involved examples. So thanks for sticking with me. I always say it's a half hour, but it's always a little bit longer. But I think you have learned some really practical skills. So firstly, you've learned how to import modules into Python. This is crucial for social science work, you know, for computational social science. You'll need to download and install modules on your machine and then import them into Python. Thankfully, when you download Python itself, most of the modules we've used come as standard. You just have to import them into your Python session each time. Again, if you've used or this is similar to the library command for loading in functions. We've learned how to request and parse web pages. So two steps. Can we actually request the data we're interested in? And then can we parse the request that we've made as a HTML file? So beautiful soup. No idea why it's called that. Is the module you use to parse contents? And then we've done the non-trivial, slightly boring but really important task of reading and writing from files. This is a task you're going to engage in lots doing computational social science. And hopefully, you know, you've learned as well how to structure your code and to do this in a very kind of efficient and clear manner. Maybe I haven't gotten it across in a very clear way.  So this was a very, you know, very focused on the live demonstration. And we do have a web scraping series of webinars. So we did three and there's a couple of notebooks supporting what we've done where we go into much more detail, you know, about the ethics and the limitations of web scraping. So for today, we just focused on the practicalities. But, you know, so great power comes great responsibility. So you have to start thinking about things like data protection, website terms of service. So what do you actually allow to do with the website? That's actually a contract between you and the website, the terms of service. So it's important to read that. And the Charity Commission website we use today is the information is available under the open government license. So we're allowed to scrape that data, which is good. But there's lots of murky ethical issues. Should you scrape personal information? For example, people. So this is really interesting. But, you know, here are the 12 trustees on the board of Oxfam UK. So, you know, we don't get their age or the date of birth or their address or whatever. But we do get the other charities they're on the board of so we can then maybe start to pin down who these people are exactly. So, you know, there's some ethical issues as well about scraping data. But it's usually exciting. It's usually powerful means of collecting data.  And as I said, we've got some free training materials that we did a couple of weeks ago. I'm sure some of you were probably on that. So thanks for coming back. Again, so, you know, we've got some, we've got three webinars that you can watch. And we've got some Jupyter notebooks as well that go into much more detail. So this websites one, you know, covers a lot of the ethics and limitations. Yeah. So value of web scraping limitations, ethical considerations. And there's an example where we capture COVID-19 data as well. I quite like this free book. Automate the boring stuff with Python. That's how I learned a lot of mine a couple of years ago, my Python abilities. Yeah. Chapter 12 is quite good on web scraping covers a lot of what I've covered, but there's, you know, a couple of extra cool techniques as well. And it's actually really good for working with files. So it's really good for using Python to work with PDFs, Google spreadsheets. And there's some, like, you know, some interesting things if you want to automate the sending of emails using Python, which is quite interesting also. So that is the 30 minute demonstration elongated to 45 minutes. So apologies. But again, hopefully you found that really useful. I can see a long list of questions. My lovely assistant Julia has been tackling those, but now I'm going to jump in as well. So I'm going to take first questions. So with less static pages. So yes. OK. So there's a really good question here. So there's a difference between a static and a dynamic web page. So this is a static web page. What that means is when you request it, all of the information in that web page is is is shown to you at the same time. So all of the source code is loaded in when you request the web page. Staying with the charity field. So Republic of Ireland Charity Regulator. So this is a different website. So some web pages are known as dynamic web pages. What that means is when you request the web page, some of the information is loaded in. But then as you click through the web page, additional information is brought in as you click through. And I'm sure we've, you know, we can think of examples where if there's a search function, for example, when you click on search, then, you know, a list of results magically appear underneath the search button. That's an example of a dynamic web page. So there's an Irish charity called Goal Sports Charity. Watch what happens the web page here. So see that? So some of the content is being dynamically loaded in. So when I initially request the web page, I'll do it again just to show you. Let's just do it one more time. So I'm going to look for this charity. Cool. I want to scrape its data. It will request the web page, but then it loads in some extra information. And this only occurs because I'm using web browser. So if I was to use Python, it wouldn't load in that extra information. So sorry. So this is a long winded way of saying, yes, there are ways of using Python to collect data from dynamic web pages. There's basically two approaches. One is you bypass the web page itself and you try and connect to the database that is providing the dynamic information. So that's possible with the Republic of Ireland charity regulator. There's actually an online database the web page is pulling information from and I can actually bypass the web page directly to the database. The second approach is to use a module called Selenium in Python. Basically, this mimics the launching of a browser. So this tricks the web page into thinking you're actually working with a browser and that you're actually clicking on elements on the page. So there are two solutions to working with dynamic web pages. It's not particularly difficult. I suppose it's just recognizing that it is a dynamic web page and then you have to find a solution. Cool. Oh, and yeah, automate the boring stuff is brilliant. Yeah, that's it's a it's an online course as well. Fantastic. And the actual. It's not free as a PDF. Perfect. OK. So there's an extra question here. So if you're interested in UK government data stored on web pages, as you'll probably notice the UK government website, you know, is is highly consistent. So, you know, all the charity data, all the web pages look very similar. And so is there a way of easily automating the scraping across many sites with different HTML structures? And yeah, so if all the if the underlying source code is quite different from page to page, yeah, then you have to tailor your code to each specific website. So for example, if I was to search for a different charity here, let's say I'm looking for any. So let's just click the Arts Foundation here. We can see it's web page looks the same. So I know that I can loop over a list of charity numbers, request the web page and I'll get the same elements every time in the web page and I can extract the same information. If the page looked different, then yes, you're I suppose your code does have to be different for each individual web page. There's probably elements that will remain the same. So the process of requesting is the same. The search term you're looking for might be the same. But no, if you've got different structured web pages, you need code tailored for each individual web page. We've got a very cheeky question is yes, you could scrape the automate the boring stuff web site if you wanted. I'm sure you could. That's quite clever. Yeah. And then as someone's pointed out with the UK government website, there's an online database known as an API that you can connect to instead. So that's really good. Yeah. Any other last questions? Oh, OK. Yeah. A very, very good question. So how would I go about scraping data from multiple links? So let's say a web page has a link has a list of companies and each company has a link to further information. Yeah. That's a really good question. Yeah. That's really possible. It's really doable. So for example, let's try and do it really, really quickly. Yeah. So let's yeah, let's request Oxfam's web page. Yeah. So we're here. We've done all that. Yeah. So let's request Oxfam's web page and yeah, parse it as HTML. It might take a while. Thankfully, I've run it on my machine. So let's do that really quickly. That was good to have a backup. So basically what you're doing is you're asking to find all of the A tags instead of the P tags. So A stands for link in HTML. So let's show you how that's done. So let's parse the web page and then let's, let's say let's find links instead. So let's create a new variable called links and then I want to find all the A tags. And I'm not interested in the differences between A tags just yet. I'm just trying to find all the links. Yeah. So as you can see here now instead of scraping paragraphs or sections, I found all of the links on the Oxfam Charity web page. A lot of these links, you know, aren't necessary. So this is a link to a survey, you know, that the regulator wants you to fill out. You know, there's lots of internal links to the web page. But we've got a link here. So on the Oxfam web page, there's a link to its record on company's house, which is the company regulator. So that information would be as we're gone from Oxfam. So let's just show you again. Yeah, perfect. So this is the link here. So we've found a link that looks like this and it corresponds to this link here, which takes you to the company house and regulators website. Voila. So yes. Yeah. So if you want to scrape data, if you want to scrape links from a web page, yeah, that's easily done as well. Now that we have a list of links, I could say for every one of these links, extract the information contained in the href tag. And for each of these links, I would use requests dot get and request each of those web pages in turn. So yeah, somebody's just said it. You can write some code that loops through a list of links. And in the web scraping for social scientists, course that we've done recently, yeah, we show you how to do that. And we show you how to write loops, which is good. So we're coming up to an hour. I'm happy to stay on for another couple of minutes if there's more questions. You can contact myself and Julia to do with anything to do with this work. So where do I put my contact details? So we both work at the University of Manchester. I think our details are on the Twitch page itself. But if you're just looking for me, you just kind of need to know how to spell my very Irish name and then you can find me on Manchester. And yeah, Julia has just posted some our Twitter links. So we check our Twitter links fairly often. So yeah, so if you can find my email address, I'm more than happy to discuss your individual, you know, scraping projects, what you're interested in doing as well. So on that note, I'm going to say good evening. Thanks for joining us. Thanks for giving up your time. Also, thank you to Julia as well, who's been fantastic. And next week, we'll be looking at how you collect data from an API. So that's instead of scraping a page, there's an online database. So there's a formal means of getting the data, but the code is slightly different. So I've been Dermot, you've been great. Thanks very much. And I will see you soon.


Hi and welcome to the session. I am Purva and I will help you with the following question. Evaluate the definite integral dx upon under root 1 plus x minus under root x where the limit is from 0 to 1. Now we start with this solution. So we denote this integral by i. So we have i is equal to integral 0 to 1 dx upon under root 1 plus x minus under root x. This is equal to integral limit from 0 to 1 1 by under root 1 plus x minus under root x. So rationalizing this we get into under root 1 plus x plus under root x upon under root 1 plus x plus under root x dx. This is equal to integral limit from 0 to 1 under root 1 plus x plus under root x upon. Now we know that a minus b into a plus b is equal to a square minus b square. So we can write this as 1 plus x minus x dx since a minus b into a plus b is equal to a square minus b square. We cancel this plus x and minus x. Now we get this is equal to integral limit 0 to 1 under root 1 plus x plus under root x dx. This is equal to, now we can write this as integral limit from 0 to 1 under root 1 plus x dx plus integral limit 0 to 1 under root x dx. This is further equal to, now integrating under root 1 plus x we get 1 plus x to the whole power 1 by 2 plus 1 upon 1 by 2 plus 1 and limit is from 0 to 1 plus, now integrating under root x we get x to the power 1 by 2 plus 1 by 1 by 2 plus 1 again the limit is from 0 to 1. This is equal to 1 plus x to the power. Now 1 by 2 plus 1 is 3 by 2 upon 3 by 2 the limit is from 0 to 1 plus x to the power 1 by 2 plus 1 is x to the power 3 by 2 upon 3 by 2 again the limit is from 0 to 1. This is equal to now 1 upon 3 by 2 can be written as 2 by 3 into 1 plus x to the power 3 by 2 limit is from 0 to 1 plus again 1 upon 3 by 2 can be written as 2 by 3 into x raise to the power 3 by 2 limit is from 0 to 1. This is equal to 2 by 3 into 1 plus now upper limit is 1 so we will put 1 in place of x we get 1 to the power 3 by 2 minus 2 by 3 into 1 plus now the lower limit is 0 so we will put 0 in place of x we get 0 to the whole power 3 by 2 plus 2 by 3 into now again the upper limit here is 1 so we will put 1 in place of x and 1 raise to the power 3 by 2 is 1 so we will get only 2 by 3 minus 2 by 3 now the lower limit is 0 so putting 0 in place of x this will give us 0 now this is equal to 2 by 3 into 1 plus 1 is 2 so 2 raise to the power 3 by 2 minus 2 by 3 now 1 plus 0 is 1 and 1 raise to the power 3 by 2 is 1 so plus 2 by 3 and 2 by 3 into 0 is 0 so cancelling this 2 by 3 and this 2 by 3 we get this is equal to 2 by 3 into 2 raise to the power 3 by 2 we can also write this as this is equal to 4 by 3 into root 2 so we will write our answer as 4 by 3 into root 2 so this is our answer hope you have understood the solution take care and God bless you


It's now time for Member Statements. The Member from Perth, Wellington. Thank you Mr. Speaker. I'm pleased to inform the House that the fit contract for the Conestoga Wind Energy Project has been terminated, effectively ending the project. This is nothing less than a victory for those who do not want industrial wind turbines imposed on their communities. It's a victory for the grassroots organizers who work tirelessly to preserve their neighborhoods, farms and their way of life. They prevail over a multinational wind company, as well as the Liberal government, whose process was rigged against them. That's extraordinary. It's also a victory for the municipality of North Perth and the township of Perth East. They stood with their constituents as unwilling hosts. I was proud to serve them at every step. I have repeatedly spoken up for them to two premiers, three energy ministers, two environment ministers and to in energy itself. I hosted and attended public meetings and wrote to the government about 500 times. Tragically, the victory came at a cost. Individuals, families and municipalities have spent considerable time and money. They will never get it back. All this could have been avoided if the government had just listened to us in the first place. We know the Liberals Green Energy Act has been an absolute failure. This legislation put the Conestoga Wind Farm proposal in motion, fighting our community and pitting neighbour against neighbour. Now it is time to heal the rifts this project has caused. I look forward to working with the community to help make that happen. Thank you, Mr. Speaker. Thank you. Member statements, the member from Windsor to come see. Thank you, Speaker. Speaker, tis the break before Christmas and all through this house. The members of Thinking of Turkey, not gross. Up in the press gallery, all wager a bet, described, are all gone. You'll find no ink stain wretch. Some members, some ministers, are either in hiding or making their way back to their writings. My supposition that here in the opposition we're sticking it out, still firmly believing we actually have clout. And Speaker, we do, as I'm sure you have heard, we usually find a way to get in the last word. Tis a wonderful season in this legislative place when we set aside our differences and actually are seen with a smile on our face. Oh, we may hoff and puff and pretend real rages, always being careful not to scare our young pages. Yes, we can be persistent, but we'll really be lost. We'd really be lost without our legislative assistance. Speaker, even you, sir, you're quite the agreeable fellow, but you'd be lost if it wasn't for clerk Ms. Debbie Dumber. Not done yet. Members will know I live near the American border, but I haven't yet heard the member for Windsor to come see will please come to order. Order? Being here has quite an education. I'm learning more than I ever expected about time allocation. So I better wrap up and be on my way. I'll see you on that February, Tuesday after family day. Yes. Members, where I'll be, yours truly the member from Windsor to come see. I can agree with most of what you said. Members, David's the member from Cambridge. Speaker, it's hard to top that one. I'm wondering if the member from Windsor to come see can sing the following. Children laughing, people passing, meeting smile after smile. The lyrics of the traditional song, Silver Bells, capture the spirit of the three annual Santa Claus parades in my writing of Cambridge. Hundreds of hours put in by many spirited volunteers make these much loved parades continue to be a highlight of the holiday season for Cambridge and North Dumfries families. The Cambridge Santa Claus parade held at night is a magical experience for children. It's organized by volunteer Dave Howell and over 35 dedicated community members. Parade goers assist others by bringing food for the Cambridge self-help food bank, Canada Post letter carriers picked up the children's letters to Santa. The village of Air Parade is organized by volunteers from the North Dumfries Lions Club. The Air Paris band, a favorite for over 125 years, kept the beat as I handed out the candy to the kids. Hesbler Santa Claus parade marshals Peggy and Russell Bygrave volunteers for over 30 years stand out in white top hats and suits. Speaker, a sponsor summed up the reason that he volunteers. It's everybody every business's duty to help support local traditions and build community. This is a great cause to support. Besides, we can't say no to Santa, can we? For generations of children and families in Cambridge, they would certainly agree. Thank you to all the Cambridge parade volunteers. Thank you for the member statements, the member from Chatham, Kinesis. Thank you, Speaker. Speaker, in 1954, the Chatham Good Fellows embarked on its first annual campaign to ensure there's no child without a Christmas. Through donations and volunteer efforts, the Good Fellows have helped brighten many on Christmas morning ensuring there are toys for children and food for a holiday meal. And recently I was privileged to speak at their 60th anniversary and to see the large number of people in support of the work the Chatham Good Fellows do. It was very encouraging to say the least. But on a sad note, Speaker, last month $3,700 worth of toys slated to be given to local children in need as Christmas gifts were stolen from the Chatham Good Fellows. But on a happier note, and in the spirit of Christmas, volunteers were not discouraged and they have received an outpouring of community support. Chatham Kent residents donated over 50 banana boxes full of toys and non-perishable food. And they raised, Speaker, over $35,000 in only a couple of hours during the Chatham Good Fellows' 60th annual porch-like campaign. For one night, a small army of volunteers walked door-to-door collecting donations. Well, Speaker, back in the early 1960s, my father, Fred Nichols, was a Good Fellow and he worked closely with his good friend and then President of the Chatham Good Fellows, Jay Vickers. I wasn't even a teenager when my father recruited me. I remember standing on the corner of King and Fifth Streets just outside the then Royal Bank selling good fellow papers to help raise money so that the good fellows could buy toys and put food in their Christmas baskets for the less fortunate families. Yes, Speaker, at a very young age, I learned how important in life it is to offer a hand up to those in need. Personal values were being instilled. I also remember, though, how cold those wintry snowy days were while I stood outside. But what I would do is I would go into the Good Fellows' trailer parked on King Street to enjoy a nice heated drink of, yes, Werner's. Yes, I said heated. You should try it. You'll like it. It was good to throw a cinnamon stick in while you're at it. So during the first 60 years of operation, thousands of volunteers from the Chatham Good Fellows have helped thousands of families. Despite the growth and success of the Chatham Good Fellows, the premise has always been and will remain no child without a Christmas. Thank you, Speaker. Welcome. Thank you. Member Stiglitz, the member from London West. Thank you, Speaker. Housing-first approaches to ending homelessness are founded on the principle that people are better able to move forward with their lives if they are first housed, then provided with the services and supports they need. Evidence demonstrating the effectiveness of housing-first is based on research, research primarily focused on men. But women's experience of homelessness is different. It is often invisible and frequently involves children. Women who experience homelessness have higher rates of mortality than men and are less likely to access resources and services. When they do access these resources, women are less successful in maintaining stable housing with success rates of less than 50%. To address these gender differences, an innovative collaboration is underway in my community of London to develop a housing-first strategy that responds to women's unique needs for relational supports rooted in the neighbourhoods in which they reside. Called Homes for Women, this two-year project is funded by the London Community Foundation. Partners include the Canadian Mental Health Association Middlesex, My Sister's Place, Women's Community House and Health Zone Nurse Practitioner-led clinic. The project will provide housing for 50 women with a variety of units scattered throughout the community. A housing stability worker will provide individualized supports in a 24-7 hotline. A housing coordinator will ensure coordination across the units and connection with community partners. Not only will homes for women improve women's ability to find and maintain stable housing in London, but it will also generate a new housing-first framework that can be shared across Canada to end homelessness among women. Thank you. Remember from Mrs. Saga Brampton South. Thank you, Mr. Speaker. Mr. Speaker, recently I was at the opening of Mrs. Saga Bus Rapid Transit Ways Phase 1 at the New Dixie Station in my great riding of Mrs. Saga Brampton South. Phase 1 is the end result of extensive planning and preparation by all levels of governments. Our government has invested $65 million towards resolving one of the most pressing issues, gridlock. Over the next three years, by 2017, 18 kilometres of transitway from east to west with 12 stations will become a reality. The transitway will be a great two-way artery facilitating movement of people, vehicles, commerce, in and out of Mrs. Saga. It is a complement to existing transportation routes and systems. It will integrate suburban areas with urban and encourage more use of public transit. I am proud to be a part of a government that is taking action to solve the problem of gridlock. And Mr. Speaker, this is a perfect example of what communities can achieve when all levels of governments work together. Thank you. Well, very good. Remember statements from the members who lead Brampton. Thanks, Speaker. I rise today to provide the legislature and the people of Ontario with an end of session report card on the Liberal government. With regard to working together, the Liberal government gets a failing grade. Way back when this Parliament first started, the government house leader asked for some cooperation between the opposition parties on getting consensus and getting some bills passed. And initially, it was four bills that the government won past. Then it went to six and then it went to seven. We on this side were going to allow a number of bills to move forward quickly in the legislative process, but there were a few that needed public input such as the tow truck operators, the blood plasma companies and also independent daycare operators. But what we learned was the government house leaders request to work with us was just a shallow promise. In fact, the corner office, the Premier, shut us down at every opportunity to have those hearings. I can't believe that we can in this legislature sit down and get a few public private members bills passed. In our caucus, we had Ryan's Law from Mr. Eurek, bill 20 that I think could have been easily passed from the NDP, bill 17 in the name of the member for Hamilton East Stony Creek for child performers. There were a number of Liberal bills as well. There was a bill on Hispanic Heritage Month for the member for Davenport, Ms. McMahon, Ontario Bike Month, Mr. Quadri regarding rate on awareness and Mr. Potts on his tip-out bill. There was a lot of bills we could have cooperated on. As far as I'm concerned, the government received an F. One of the members said that they believe in transparency at only the right time. Well, you know what? The right time has passed.  Thank you. I'm sitting with a member from Scarborough Agent Court. I'm pleased to rise today to recognize a young talented Scarborough Agent Court resident, Satchel Patel, a former student at Stephen Leica Collegiate in his fourth year at Ryzen University, who have known since 2006. For his final thesis project, Satchel and his fellow students at Ryzen School of Media are producing a Snow Queen 2014, a community benefit to support 519 Church Street Community Center. Snow Queen is a live, 60-minute holiday-themed production with drag and musical performances at Winter Garden Theater that will take place tomorrow, December the 12th. For over 35 years, 519 has been working with the LGBTQ community to build a healthy, welcoming spaces to meet, participate, and celebrate together. The production of Snow Queen received significant sponsorship from Ryzen University President Dr. Sheldon Levy and notable LGBTQ organizations. The students themselves also raised additional funds on their own to help to cover the cost of producing the show. Mr. Speaker, I'd like to congratulate Satchel Patel and all his classmates for producing Snow Queen 2014. Their dedication to the LGBTQ community in Toronto is an inspiration to all residents, especially young people in my riding of Scarborough Agent Court. And tomorrow, Mr. Speaker, I'm looking forward to seeing the production of Snow Queen 2014. Thank you, Mr. Speaker. Thank you, Mr. Speaker, and it gives me great pleasure to stand up as the last member statement of this section and to talk about something that's most appropriate, the Christmas tree lighting ceremonies I've had to enjoy. Recently, Mr. Speaker, I participated in your own Christmas lighting. When we lit up lights across the country at a certain time, and I was able to bring greetings on behalf of the Premier, I was delighted to do so. And then two weeks ago, two weeks ago, I was down at Danforth in Victoria Park at the Eastland Park where the DECA, Danforth East Community Association, had its own tree lighting ceremony.  Angela Matich set up that facility. We had the Pegasus Studio, Zero Gravity Circus, lit up the lights. It was most inspiring. And that same day, Mr. Speaker, I went down to the foot of Ludi Avenue on the boardwalk in the beach. And there, we had an incredible ceremony where we were to light up the Ludi Lighthouse. And of those of you who have seen my Christmas card, that's the Ludi Lighthouse with those wonderful trees in the background. Mr. Speaker, the DECLUTE real estate family for the last 10 years have been lighting up the beach and it was an unbelievable event with Lena Boyd, the first lady of guitar. She and I singing Here Comes Santa Claus. And finally, there I was at Kew Gardens last week, Mr. Speaker, with the Beaches Lions Club. What an incredible organization doing such good work in our community and around the world. And there, we had joined by Mr. and Mrs. Claus. And I'll tell you, Mr. Speaker, all of these tree lighting ceremonies have certainly filled my heart with seasonal joy. And I'm looking forward so much to sharing this time with my community, my family, and I want to wish all members of Merry Christmas, seasons greetings, happy Hanukkah, and to all, a good break. Thank you, Mr. Speaker. I thank all members for their statements. It is now time for repeat.


What I'm going to be talking about is how you can get insights by joining two maps But before we go there just some basic bookkeeping things in case you're tweeting these are the hashtags You probably want to be using the bike on India hashtag my hashtag my idea is a son and zero You don't need to worry about the slides. They are online. I've already posted on Twitter The link to the slide deck this slide deck that you're using But if you desperately do want to take notes, then one small suggestion Research has shown that taking notes on pen and paper is much better than taking notes on laptops if you want to remember stuff Or on mobile phones, so this was a discovery for me In fact, it was my discovery of the year and I'm following it diligently. Do give it a shot if you want to take notes Let's dive in the story begins at the Karnataka elections in 2018 see about an eighth of the voters are Muslim and both the Congress as well as The Janata the JDS were trying to get their support while on the other hand BJP was taking potshots saying both of them are just trying to Peace the community The Hindu newspaper wanted to write a piece about how large a factor this is and where all the Muslim vote Is strong? You see here. We have a problem the thing is that the proportion of population by religion is available only at the district level or the Village level depending on where you get the data from and this is from the census Unfortunately Elections are not conducted by district elections are conducted by constituency and these are two very different maps so I have data in one map which shows me how many Muslims exist in a particular region and I want to see how many Muslims live in a different region on another map and even though they overlap There really is no direct way of getting the data from one layer on to the other So we literally don't know how many Muslims live in a constituency. So how do we solve this problem? well, the logical way is you could take one district and a constituency or a set of constituencies and Let's say the district has a population of 100 out of which we know that 13 percent are Muslims And we want to split it evenly across a bunch of constituencies. We could just overlay them So one district could cover multiple constituencies one constituency could cover multiple districts And there is a many to many mapping between these there is sometimes full coverage sometimes partial coverage So this district for instance covers at least one constituency fully and maybe this takes up about one third of the total area So I can say Approximately one third of that district's population, which is that red area lives in this constituency Or let's take another constituency which overlaps with this district So now only a portion of this constituency overlaps with this district So in this area, which takes up maybe about one fifth or about 20 percent of that district population I can say that population lives here in other words We are simply making an assumption that within a district, which is the lowest level of data that you have or if you have village data That's far more granular the population is uniformly distributed. That's the basic assumption now What we can do is fragment each of these districts and constituencies by overlaying them and creating an intersection Out of those and reassembling those and this is a process that I call reshaping the map How much of this can we do in Python? Can you switch over to this side? Okay, there is a library called reshaper that we put together The reshaper library is something that's very work in progress by the way You can find it at github.com slash grammar slash reshaper. It does exactly what I'm about to show you right now So let's give it a shot. I'm going to open up the ipython notebook. The library that we are going to be using for this Primarily the core library is geopandas for those of you who have been working with data Pandas is pretty much the de facto library to use for any kind of data processing Geopandas is becoming that kind of a standard for any shape file So if you have any shape file and you want to do any kind of geospatial processing an easy way of doing it is Geopandas and an easy way of installing it is through conda using anaconda rather than trying to do a pip install by yourself It's a little more efficient on most machines. So let's import geopandas now I have a shape file which has the karnataka census data Which will eventually appear I'm gonna load it once it appears on the screen. You're just taking a long time Okay, there we are back again so GPD has which is abbreviation for geopandas has a from file function that lets you load any shape file Now the other question you'll have is where am I going to get these shape files from we'll come to that in a bit It's not as difficult as you might think. Let's say you've downloaded the shape files This particularly is the karnataka census shape file. And what does this look like? Geopandas has a plotting function which lets you see what the Map looks like. So if you look at these constituting at these districts, this is a pretty large district This is be there and so on Well, let's take the area for these geopandas offers an attribute called dot geometry Which has an attribute called dot area which gets you the overall area of each of these regions And if you want to look at what that data frame looks like each of these regions Corresponds to one row. So the bugle code district is one row Bangalore rural district is one row and so on all of the data In the shape file also comes in here. You have a column called geometry, which has the additional geometry details This is a pretty large column You probably won't be going into the details of it and we've just now added one column called area Which is out here and this has the area of each of these regions and at the very least you can figure out Which are the larger regions which are the smaller regions. Let's do the same for the constituencies data set So here we have the constituencies. There are more. These are parliamentary constituencies, by the way Not assembly constituencies the difference being if you're electing someone for the parliament It's or an MP then it's a parliamentary constituency if you're electing them for the assembly, which is an MLA Then it's the assembly constituency parliamentary constituencies are bigger. So you'll notice that out here There are multiple parliamentary Constituencies that sit in the same region that this district sits in but it's not a perfect match Again, let's take the area and see what this looks like We have a bunch of these parliamentary constituencies like Gulbarga, Bijapur, etc and their respective areas now Geopandas has a function called s join which lets you take two shape files and Create all of the intersections around those shape files the fragments that I just showed you Out here Yeah, creating all of these fragments is what the s join function does so if we do that Then what it's now done is created a new data frame called merged and that has all of these shapes Let's validate that So there are 30 districts and 28 constituencies, but when you overlay them it turns out that there are 147 fragments each of which represents an intersection of a district and a constituency Now given this it should be possible to just take any metric Like the percentage of Muslim voters or the number of Muslim The size of the Muslim population from the district data into the Data that you have on the constituencies, but it turns out that it's a little trickier than that so you have to do a little more calculation and That's what's available in the reshaper library. You can take a look at the code What it does is moves the metrics from one layer to another in a way that is seamless so ultimately all of the Somewhere a lot, okay No, so it was not the function the function is merged out by the file is merged out by and This moves all the metrics from one layer to another Once we have this the final result is an excel sheet that kind of looks like this It has all of the attributes from both the layers So it says for instance that this particular assembly Constituency is broken up into three regions each of which maps to different districts So some of it overlaps with more on some of it with Shiva saga some of it with Therap In fact, these are across different states And what is the area of each of these along with a variety of other? Metrics that you can calculate and the proportion of area that is overlapping Once you have this kind of a data set what can we do with it? So let's revert back to our story what actually happened in the Muslim vote Well, this is the constituency wise Muslim voter population In Karnataka. This was used by the Hindu to publish an article around where exactly the bulk of the voters are concentrated So there is a chunk here. There's a chunk here. There's a chunk here Now what was happening at this particular point was there was a fight for an alliance the AIMIM which is a Muslim party whose name is very long and I can't even say it fully but they had won a number of seats in Telangana and We're looking to also participate in the Karnataka elections They had planned to contest in 60 seats now to make sure that they get the Muslim vote both Janta Dal and the Congress were vying for an alliance with the party and in April 2018 AIMIM decided that they would not be directly contesting in the elections, but instead would be supporting Janta Dal now we have the results of the elections By constituency we know the voter population by constituency Let us see what happened to Janta Dal Turns out that where there were more Muslim population Janta Dal actually got lower votes So you can see the net result of this election and the alliance Congress on the other hand had a mildly higher voter share and Where there was a significantly larger voter population it turns out that BJP was the one that gained the most Now while I'm moderately okay at Python. I'm terrible at electoral analysis. So I have no idea what this means Okay, I'll let you figure it out the Elections in Maharashtra and Haryana are also coming up and it turns out that Congress is allying with AIMIM and Let's just leave it at that So what can we do with this what kinds of data sets exist and what is the potential of being able to join? Data sets across two spaces. That's something that I'm pretty keen on he turns out that in India There are broadly three kinds of geographic hierarchies. There is a political boundary hierarchy a postal boundary hierarchy and an administrative boundary hierarchy By political boundary, I mean the state parliamentary constituency assembly constituency going all the way down to polling booth this has all of the results of the elections and One of the important aspects of this is that policies get made to a good extent at this level because the MPs and the MLAs are focused on their respective constituencies The second is the postal code boundary. There is a zone within which there's a sorting district within which there is a post office And there is a pin code and there are about a hundred and ten thousand of these in total The third is the administrative boundary hierarchy. So there is a state. There is a district There's also something called a division, but we leave that aside Then they could be a sub district block or village if it's a rural area or it could be municipality zone and ward if it's Township Now this apart there is one other way we can create our own hierarchies By the way before that in case you're looking for shapefiles for many of these The easiest way to get the shapefiles for India is to search for data meet maps data meet is a group that It's it's discussion forum and there is a lot of active discussion on various kinds of maps pretty much any kind of map There's a decent chance that you'll find it on data meet And if it's not there on data data meet ask the people they might be able to post something and if not It probably just doesn't exist But you can also create your own boundaries if you have a single location You can look at the area that is closest to this particular location than any other location So for example, if this were a network of let's say schools Then what is that region that is closest to a particular school than any other school? So if I take this particular point as a school then this red region Represents all of those points which are closer to this school than any other school This particular process is called a boronoi desolation and is something that comes out of box with QGIS It's something that you can create with command the command line problem again using the reshaper library But what that means is that now you can take literally any point and convert that into a region and The potential for that is quite high So if I look at the kinds of data sets that you can create with location boundaries, right? So there's Take all the hospitals take all the schools take all the bank branches take all the petrol pumps take all the locations Where crimes have been reported take any address? Take all the telephone towers take all the locations where there are stores of a particular of a particular brand All of these are data sets for which you can get an address and an address can be geocoded into a point if it can be Geocoded into a point you can convert that into a region and for each of these you naturally have some kind of data For schools, you know how many teachers or how many students there are for telecom towers, you know, which is the Organization that runs that tower potentially the telecom organization will know how many calls are flowing through it If it's healthcare data, you know, how many facilities that hospital has how many patients how many doctors all of these are data sets That can be that can be added to that particular cell In in your respective region What this means therefore is that if we take any of these data sets that which you can create from location boundaries Or that often already exists by administrative boundaries And this is a pretty powerful set as well senses gives us demographic data Asset ownership who owns laptops internet connections TVs car fridges social and religious data economic indicators Well income household indicators is the house made of a mudroof brick roof Do you have a toilet in the house not have a toilet in the house practically every government scheme is tracked this way So how many? How many people have benefited from the National Rural Employment Guarantee Act Banking data is reported this way health data is reported this way So effectively anything that the government runs is reported by administrative brown boundaries anything that the corporate sector runs by and large is Reported by locations so between these two there is enormous potential But there's also the fact of how decision-making happens Ultimately political boundaries are owned in some sense by an MP or an MLA and of course there is also the associated How? The IIS equivalents who usually run it by administrative boundaries So if I wanted somebody on the political side to make decisions then I could take any of this data and Put it on to the constituency boundaries If I wanted an administrative official to make a decision then I could take any of this data and put it on to a district If I wanted a manager or a principal of a school or the CEO of a hospital to make a decision I could take all of that data and put it on to their geographic boundary For example, one of the things that the Hindu again did was found that the Congress is doing much better in the Agrarian areas and they did that by taking the census data which had the percentage of farmers and Mapping that on to the voter constituency regions If we took for example census demographic data and school data, we can answer a question Where should we open new schools so that students don't have to travel far or where there is a reasonably equal distribution of students across schools if we took economic Indicators how well the country is growing versus Bank branch data then we can answer questions like are the bank branches Distributed based on population that is does every person roughly have equal access to the bank or Based on wealth does every rupee have roughly equal access to the bank or if it's in between How close is it from one to the other? We could find out whether increasing a district's wealth Leads to more theft so that means people get richer So does that mean to that does that lead to increase in crime or does it lead to less theft? Because the people are richer and they don't need to steal therefore and these are data sets that are available and can be joined Similarly with health data does poor health lead to an increase in the number of pharmacies that are set up in that region because the pharmacies Can sell more vice versa if you actually set up more pharmacies Does that have a positive impact on the people's health in that particular region? Now the reason these questions are trivial to ask but nearly impossible to solve today is because Merging the data across different kinds of layers of maps is non-trivial But both conceptually and technologically is quite an easy exercise What can we do to solve problems like these? Well me personally? I'd love to see more of these hidden insights come out But there are a few things that you can do literally right now first If you have an idea Take a look at these data sets any of the data sets that you know raise an issue on this particular repository on PyCon And I'd invite all of you to share this with people. It'll be great to see what kinds of ideas can be Solved using these problems and I'd like to crowdsource this to a number of people on the administrative side on the NGO side And on the corporate side to create a repository that says here are things that we can do If you want to try solving One of these and discovering your own insight To build your own portfolio to share some useful knowledge then start by finding a map Like I said data meet is a good place where you can find a map You can find the reshaper library on github.com slash grammar slash reshaper the links are again on github.com slash grammar slash PyCon 2019 This is the one link that you need to remember and if you find something do share it on Twitter Please tag me my ID is sRN0 I'd love to share it at least with the media and Get some people to understand the power of geospatial joints if you want to contribute to the library right now It's it's in a terrible state or if you want to learn more I'm planning to organize a series of workshops on geospatial joints do drop me an email My email ID is s.anand at grammar.com and I'll mail you the workshops if nothing else if you just enjoyed the talk And you've learned something about it then tweet about it the tags are PyCon India 2019 my ID is sRN0 More than anything else. I'd love to see some insights come out by joining data. Happy mapping So basically So my question is Basically, I belong to North is part of India. That's why I'm from Assam so what happened In terms of the the documentation for this geographic data and so those are always kept in a sort of in a Register we call registers or something which are all in a So how we use that image processing and all like to enable the Those things into a more of a like a public space Okay There are broadly three ways in which you can get this kind of data out The first is big borrows team somebody in the government may have this data So for example, if you go to the survey of India They sell these shapefiles, of course I've been trying to buy one of these shapefiles for the last six years now and I've failed And I've tried it through the Prime Minister's office and I still failed But it's actually easier to just walk over to the survey of India office in Deradun and give them a USB stick So depending on how you approach it Prove relatively straightforward On the other hand sometimes the maps don't exist So for example, most interesting anecdote was the former Head of the postal college of Mysore was trying to create a postal map Region of all of the pin codes. It turns out that Nobody knows what the region is that a pin code covers So he created that he uploaded that into ISRO's booban and then after about a year and a half realize that People have permission to upload into ISRO but not download from Booban So after one and a half years of putting all the data, the data is locked. It's not even there So today what is the best source of getting pin code data? Turns out that what people did was took various locations Geocoded them. They said this locations at this particular pin code this location that this This particular pin code let's draw a region around it using the concept of overnight polygons and publish it So the second possibility is to create such maps The third possibility that you talked about which is can we use image processing to detect it? Some features can be detected that way So for example, if you want to detect urban regions or constructed regions That's possible using satellite photography if you want to locate water bodies and whether they are growing or shrinking So for example in in Chennai is the Chamberambakkam Lake actually drying up That's something that you can draw a boundary around using image processing and that's a straightforward method But the thing is I don't think a single method will work for the wide variety of datasets Which is why we have many of these but the biggest lesson that I've learned is that 90% of the things that we want Somebody else has usually wanted and has managed to get their inputs So I find that the most efficient ways to ask and data meet is a pretty good place to ask if somebody already has this data Thanks a lot Anand thank you very much for your talk I've got a question regarding shapefiles. I had the requirement of using the map of India a few times And I suddenly realized that our External boundaries in a lot of places are in dispute and the kind of shapefiles that we get are not matching with what politically we Want our boundaries to be so is there any official place from where we can get these shapefiles because the only shapefiles which are available are those distorted shapefiles and I finally had to Change the shapefiles myself to use it. I couldn't find any official place from where to get the shapefiles So the official place is the survey of India which claims to sell these maps Like I said for the last five six years now. I've been trying to buy these maps It's actually not possible, but there are people who have succeeded and Okay, this talk is being recorded, right? Okay Let's just say that if you go to data meet maps, you will get unofficial but correct maps That's what so since you are in this field shouldn't we have a system of getting Correct official maps. Isn't there a process being put in place or something? I tried talking to a couple of people at the Prime Minister's office and Suggested this they put me on the phone with the inspector general of surveys or some such high ranking official Who said yes, absolutely connected me to some person who connected me to some person who connected me to some person Who was exactly the same person I talked to in the first place So I don't know. I'm sure there is a process. I Don't know it well enough