Hi everyone and welcome to the early of the mid-career keynote for the research software engineering Asia-Australia conference. I am delighted to introduce Jess Ma. And I will read the very official and very formal introduction for Jess. Jess Ma is an associate professor at the Australian Institute of Bioengineering and Nanotechnology. Jessica spent six months in the United Kingdom working for the European Bioinformatics Institute before heading to Harvard University in Boston to undertake her PhD. Later she started her own lab at Albert Einstein College of Medicine in New York, USA. She has won the Metcalf Prize in recognition of her leadership in stem cell research. Her research group focuses on the development of bioinformatics methods to understand how regulatory processes go right in human diseases. The explosive ability of big datasets coupled with the speed at which sequencing technologies have advanced have created an exciting environment for the current state of computational biology research. The Ma group looks to modern tools and statistics such as Bayesian methodologies and machine learning algorithms to make sense of biology from big data. So that is the official introduction. The personal introduction is that I've known Jess for a very long time. I have even had to look at her code at some point. I will tell you that story when we are not recording. And that's just the R part. It's not that you're okay. You're okay. It was actually quite easy to understand. And I really, really, really grateful for Jess today and I will hand over to her to start her talk. Thank you so much, Jess. Thank you, Roland. Thank you, everyone. It is really wonderful to be here. Thank you for that really kind introduction, Roland. It's been put to me that my official bio is a little bit stuffy, but you made it very approachable, I think. And yes, Roland and I go back a long way. And so it's actually really quite meaningful and interesting to be here giving the mid-career keynote, but more importantly, listen in on some of the discussions and topics that you're going to discuss in this conference. I think having met Roland early on in my career and having gone through different transitions all in academia, I think, where I'm certainly at an interest in crossroads and thinking about how do we create more opportunities for people in this area with this sort of capability. And also, you know, there are no easy answers. So I would love to hear your thoughts and questions and have a great discussion independent of just me talking. So I'll leave a lot of space that we can talk together. So hi, everyone. I just also wanted to flag that I am coming to you from the Gold Coast in Queensland. And I'm actually running my own conference here at the moment. So it's not the most ideal setup. I'm in my hotel room and I hope the internet can sustain us. So let me know if there's any problems though. I gave this the title of my talk just really because it overlaps a lot of with the kind of research that we do in my group. So it's a bit of a statistical pun. But I think also that there's a little non normality in everybody. So I hope that there are things that that you can take away or I certainly have a lot to learn from all of you. So I just want to start by shouting out some kudos to the research software engineer visibility is a really meaningful concept. And I think that's probably why one of the reasons why we're all here today. Funny story. Roland approached me a while ago and he said, I'm putting together this conference and would you be interested in giving us talk and I had a look at the conference spiel. And it, it, this is this is it Roland did not pay me to say this but I actually had no idea what a research software engineer was and in reading the paragraph and I thought it was so well written and I'm sure Roland Apollo you guys put that together but it is, I think such a wonderful delineation of the kind of people that I've worked with, and also someone who I think I've become as in working in bioinformatics. And I think this part you've all captured really well as this trouble of defining their role and value within academia and academia is is a very well defined hierarchical space. And I sort of see over the years, people who would subscribe to this label of being a research software engineer, sometimes getting lost and sometimes not getting the kind of promotability or visibility that is is congruent with the skill sets and the contributions that they bring. So I think this is a fantastic idea to get people together to talk about some of these issues. You're a big believer and you can't be what you can't see and, and I think in reading this paragraph my jaw just completely dropped open and I felt really seen. You know, and I hope many of you feel this way that this conference has given you a platform where where you feel seen and recognized to because I think that's a really important step to doing the great work that we do. I'm going to talk a bit about I'm not sure again I'm not at my office so I don't have my dual screens is the zoom panel blocking the top of the slide. I don't know what it's fine okay cool because I don't know what I see is what you see but anyway, I'm actually blocking it for me so I'm going to move it a bit okay. So just just a highlight is as really mentioned, I am a research academic at the University Queensland. I run a research group that largely does bioinformatics research. I'm guessing I saw in the chat that there's some fellow bioinformaticians out there and I certainly recognize some names and faces there so it's really great to see you. And most of you who aren't aware bioinformatics is a discipline that combines statistics and computer programming with the goal to really solve complex questions in biology. And one of the other things that I think bioinformatics makes bioinformatics such an attractive capability is that you learn to work with big data sets. And science in the last few years has become a very strong buzzword. And it can mean a lot of different things to different groups of people. But certainly, I think the real thinking and engagement with the messiness of big data and trying to use statistics, but also using programming because the data is so large to arrive at actionable conclusions. So I can move forward in science. I think that's really capturing what we do in bioinformatics. So in my career. I've been working in research for quite a while now. But the kind of contributions I've made have been in different areas of biology so stem cells, aging cancer and neurological diseases. And everyone alluded to I used to love coding so I started my career as a undergraduate in statistics and mathematics. And I used to really love the fact that you could just write lots and lots of our code and get insights into data in ways that other people really, really couldn't. So having said that I am a group leader I'm an associate professor at a university. And I'm based at a research institute so full disclosure I have a fixed term contract in three years time and be looking for another job. And so what that really means is that I now love mentoring people who love coding and sadly, the amount of code that I write these days is very small and for embarrassing applications like exam questions and not for research. But that's okay I mean I really love working with the people that that work in my group it's it's been really exciting and challenging but also stimulating to try to help them and get them set up in their careers and I think some of the points that were alluded to in the discussion about creating sustainable jobs. I think is a really, really important one and one that I think we need a lot of input and action on quickly. So, just to go back to some of the work that I do. The title of my talk is is really inspired by the kind of research that fundamentally my research group has been focusing on for the last 10 years. And it's essentially stepping back from this question of assuming things follow a certain way. For those of you who know data will know that that takes about five seconds before we throw that assumption out the window, but really effectively made a career out of asking what's, what's not normal, you know what if it's not normal. I'm talking about the statistical distribution, the normal distribution. And so just to remind you, I'm sure you're all familiar with this bell, bell curved shape. But in statistics there are actually other kinds of distributions that we don't often talk about that these have similar have the same sorts of mathematical kinds of definitions and statistical properties that we all know and love the normal distribution to have. But as you can see by the diversity of shapes, there's different ways to model data and there's different things we can get out of these distributions some are really easy like these bimodal distributions where we can get different sorts of different sorts of sub clustering. But others like the log normal and the Pareto are really good for modeling asymmetric data. So, in bioinformatics, all the way back in 2011. This was really the first paper that set my group apart and and ahead. The idea of instead of looking at the average expression, which is what most people do when they look at gene expression data is to think about using the variance so while variance is a property of the distribution it means instead of looking at the center location we're now looking at the width of a distribution and proposing using this as a regulatory parameter to understand how diseases work. And this was a proof of principle kind of question, working with actually some of Roland's former colleagues, and we found some really interesting things and and from then on we started to look at this question more closely. What happens when we don't assume a normal distribution of the data. And I've had various students who have taken this idea further. I had a wonderful MD PhD student Dr Daniel PK, who was looking at bimodal gene expression in breast cancer patients and found some really interesting subgroupings there that helped us identify some new regulators. I've got a current student Malindri Dalmaratni who's about to finish her PhD. And she's been looking at the prevalence of different sorts of shapes and single cell RNA sequencing data. And also what that means. This is also quite a big departure we don't tend to think about modeling gene by gene individual distributions, and she's finding some interesting things that really challenge the assumptions that we all make. I've been working with a wonderful student Ebony Watson, who's just had a paper published in briefings and bioinformatics where we're trying to understand what does distance really mean and in the context of these high dimensional maps where we're trying to understand data that has tens of thousands of axes. And what can we learn from that because again there's lots of assumptions there at play and we're finding that, depending on the data structure there are different conclusions that we can make. But I'm really here to talk more about how do we empower research software engineering communities. And so I thought it might be insightful to share some of my origin story, in terms of how I came down this path. I wanted to start out this part by saying that I really slept my way through a first year undergraduate engineering math course. And that was because we were taught as probably many of you were, or maybe nowadays they do it really differently, but learning maple and MATLAB and so these really, you know, mini tutorials on how to learn how to code for the first time. And that's why I slept my way through because at the end of this, I did not know how to code I didn't even know I didn't know how to code. Yeah, and it wasn't until I went to we hi in Melbourne in 2000. And I was fortunate to get this summer internship working with Jean Yang. I can't remember exactly the project. But she effectively gave me like a set of box plots and said, you know, I've got the software and are that does a sort of analysis but I've got these box plots and I, I need you to basically flag when there's like dots, like outliers in the box plot, or where they are how far they go or something like that it was super super super super simple question like you could see the dots. And it took me a few weeks to learn our to get it to do what I needed to do to get it to do it correctly. And then complete that loop in a timely way. And I remember those few weeks were the most exciting, you know, weeks of my career at least at that stage. And it really kind of reflected how in insignificant or unsubstantial as budget that's the right word. The undergraduate training had been, and it was a real contrast to actually have the chance to sit down, focus on a specific problem with coding and orientate my learning around that. And after that I really caught the research bug in a really big way so I went on to a new for the following summer, and what was Professor Sue Wilson there. And it's also this idea of being lucky and also being open and available for opportunities. And so while working at a new they had struck up a collaboration with a startup company bilateral in Sydney and so these guys were basically doing the first set of bioinformatic courses around the idea of learning how to analyze at the time of microarray data. And so they invited me to come along with their teaching team and so for a year. I was really fortunate to be able to teach courses around Australia so effectively on how to use our to do basic statistical analysis of micro and microarray data. And that was both a really exciting way to engage with researchers and understand the kind of problems that they wanted to solve. But for also, for me who was really I was still an undergraduate then it was a really formative sort of experience and learning how to teach. And at the time, I remember one of my colleagues who was part of bilateral who I was co teaching with and, you know it's quite nervous and insecure being an undergraduate lecturing in these courses when we had participants who were professors elsewhere. And he said you know you really just need to know the right amount to teach something. If you know too little obviously that's not going to work. But actually if you know too much sometimes that that doesn't mean you're going to be a better teacher and I think that was a really important message that I've carried through my career that you just knowing stuff doesn't make you a good teacher and that there's other ways that are actually really important to pay attention to to get that information across. After that I went for six months to roll and mention in the intro to EBI in Cambridge England. And I worked with Albus Brasmouth there working on some software. I then went to Harvard to do my PhD and I went there specifically to go work with Robert gentlemen who I'm not sure people are familiar with who he is, but he is the co creator inventor of our. And it was my dream to work with him and being at Harvard also meant I could do some other things so I taught our courses at the Harvard School of Public Health. And also at Cold Spring Harbor in New York where if you know genomics is basically the mecca of molecular biology and genetics and genomics. So that was a really fun sort of experience. Along the way as many dreams sort of don't come true, Robert actually left to move to Seattle, but in the process I was able to transition my PhD to John Quackenbush who is also a huge person in terms of software development as well. And ultimately then finish my PhD in 2008. So I think well obviously there's a whole other chapter that went on afterwards I think for me, sort of developing the kind of skills that that I needed and sort of mindset to do the research I do. I think all of this here is really what sort of was very formative. And I think for everything that I've learned. Sorry to keep moving this around I hope it doesn't annoy you. For everything I've learned I think the conclusion, the one conclusion I can stick to is that science needs everybody. And along the way one thing that I've really enjoyed doing is to to run hack the buns. And I think they're really inclusive kind of environments that bring out a lot of the fun and creativity. And that is inherent in the kind of software engineering research that we do. And so I ran a stem cell hackathon, a few years ago before the pandemic. And that was really focused on trying to get more people, especially people who, for whatever reasons may have shied away from programming to get them involved and and we use stem cell research as an anchor to get people kind of focused on a specific scientific topic. And the real goal was to break down barriers to data science capabilities to try to, you know, break down that sort of phobia people might have about programming, and just develop that approachability accessibility so that people could see that it is actually a lot of fun to learn how to program to use programming, when you've got a really central important question to solve and a team around you. So that was that was really, really great. Before that, when I was still based at Albert Einstein College of Medicine in New York, we also ran a hackathon there but that was based more around aging. And again, it was themed around getting more people to have a go at programming to come along to work with with different kinds of disciplines of research. So it was a huge success. So, I haven't run a hackathon since but I would definitely be open to it. I know it's become a much more popular medium. Since I first tried in 2017 and so it's really great to see that becoming more popular, especially in our community. Oh, sorry that there we go. So I think also another thing I've been involved in is trying to see what age we can foster sort of this critical thinking and data science mentality. And a few years ago, I did a pilot experiment at the ABN where I basically sat one on one with kids to see if I could either get their curiosity up around data. To see what they thought about data science related questions and also get them to try out some preliminary skills and in data science, whether it was making plots or writing codes or coming up with other ways to engage with data and as you can see with all the pictures here it was a really great success. So I'm in the process of trying to expand that out to a high school program. It's a little bit old because we weren't actually able to debut something in 21 we did get funding actually to take this to the next level but because of COVID we weren't able to take it out to schools and the funding program weren't happy with that so we weren't able to move forward unfortunately but I think maybe there's a possibility now that things are starting to come back. So I think that research, you know, regardless of the kind of specifics and the technicalities of the research you do, for me in my career, research has been a really wonderful outlet for curiosity. And so it's really not just about the results that you generate or the papers you write or the software programs that you output. For me and I think for everyone like it, it's a really powerful kind of learning experience. And it's really built you know that sort of personal growth. It gives you networks that move your career forward whether it's intentional or serendipitous. And it also develops or really challenges you to grow your own self awareness. My career is a tough environment and I think understanding yourself understanding the limitations and strengths who bring a really one way to keep moving forward. And I think regardless of what career path you go down whatever job title you end up with. I think developing this, this outlet for curiosity developing these sorts of properties that you have. I think are really important so I put this together actually for undergraduates a while ago and I thought it was a kind of a cool bingo card to to keep track of you know I sometimes in the busy day to day lose sight of some of these questions but I think it's important to keep them close.  What do you really enjoy the most I think as we get older as as we've become more grown up. You know we send tend to lose sight of these things that are fun or enjoyable or inspiring. You know so I think there's value in just making note of what stories resonate you. You know who do you talk to that you're really intrigued by. Is there a gap or disconnect that you really really really want to learn more about and using those sorts of questions to guide where you're going internally. There's always value in that. So, I think the other thing that it would be irresponsible to to acknowledge is that, as I said, academic academia is a tough environment and because of that researchers need to be resilient. Roland was talking about triggers earlier and I think there's, you know, not enough acknowledgement of that or not enough conversation perhaps around that. But nevertheless, you know there there is a lot of kind of resilient people walking around. But I highlight that not not because I really want to hop on that but the fact is we often don't talk about the fact that researchers need a lot of other skills I think to make science goes smoothly. And so we think about you know research being built on very precise set of skills and programming is certainly one thing there's no wiggle room around you know calling a function or working with with a data variable. But I think sometimes we lose sight of, especially in our trading of others is that things like compassion patients, you know effective communication, respecting others being empathetic and understanding that you know people may be on a different path. I think if we could bring more of that into our research front and center I think in many ways some things would improve. And to me, I've got two models that I really like to sort of lean on in various times of my career and they both come from a remarkable artist, Marie Andrews Andrew. And she's got an Instagram and she's got various books and things like that. But you know and and I'm sure you've seen sort of variations of this and some sort of capacity but this idea that you know what you see is the tip of the iceberg what you see sharing being shared on Twitter or, you know, for people that are doing well it's it's and and we certainly celebrate and applaud those efforts. But behind that there's all these different things. Yeah, and you never know for certain people how deep this iceberg goes right or what parts of the iceberg are really cutting into someone's life and I think it's, it's it's hard. And that certainly, you know affects how effective we are at work. This other idea of resilience, you know, and, and developing strength and self awareness and all that sort of superpower that we call upon, especially to be leaders, you know, I heard those talk more about how to empower senior overseas and I think as we move up in in the career ladder, you know, we do take on bigger fights. And I think we often focus on sometimes so much on on the negativity of those fights without stopping to celebrate how far we've come and the kind of fights we've we've actually been successful in navigating. So let's see. I know that time wise for sort of running short and I don't want to, you know, slow down the time too much but I think if anything we can sort of talk together about how do we support our RSEs and and I think that's what the afternoon sessions are also for and I apologize I can't be part of that because there's no difference here but it sounds like you're all set up to have a really great discussion but some other things that I've been thinking about is, you know, academia is a bit like being on a roller coaster and in some ways the chaos is a bit like being at an amusement park. And so not not to get too deep about it. But, you know, some of the questions that I've been wrestling with lately is, are we in an environment that supports and values problem solving. You know, I think the kind of work that we do is is is so important for developing solutions. But sometimes that's not the same thing. Right. In terms of what's valued in our day to day jobs. Obviously what is the future of academic positions specifically research positions in a university framework. How do we celebrate achievements you know I think with other industries that are more established there are certainly you know more prestigious kinds of traditions that people follow that allow us to celebrate quite naturally you know achievements from our colleagues.  It's, it can be hard right and I know Roland's trying to create different prizes and things like that and I think that's also really important, both for celebration but also visibility. One thing I wrestle with a lot is how do we develop critical thinking skills, especially with different generations who have different sorts of training sets and and and kind of ways to think about the problem and I think they're all valuable. But how do we sort of corral that to to develop, you know, excellence in critical thinking. How do we build identity and credibility as a group, I feel like the attention span for so many things I mean we saw that with data science that can be difficult, I think to engage into finding control. And for grant writing as a research academic. This is my, you know, greatest bugbear and and how do we navigate those cycles for those of you who have been grant writing in Australia for the last two years. We also understand that the road has been exceptionally bumpy, and it's hard for us but it's also hard for our ECRs you know I saw that the ASC just opened their DECRA call for next year but my understanding is the existing call has not even been announced like the those people have heard the outcomes. So I think that's hard like how do we mentor our ECRs to do these sorts of things when so much is at stake like contracts, research, all that sort of stuff right so there's a sustainability issue grants in Australia really in these three years, five years of fellowships. And when we think about that with the cycle of a PhD student right, it's just really hard to create you know critical mass with those kinds of timelines. And there's also an equity issue from lots of different dimensions how do we make this grant writing experience, you know one that is more aligned I think supporting our RSEs. I think a big topic is reproducibility and research. And this last one here is how to foster mentorship you know other, other disciplines have sort of got different programs and different streams of developing a mentoring kind of community and I think you know all those have pros and cons and and I think it's interesting to talk through, you know the limitations and the successes that we can sort of mimic. I wanted to end with this sort of last I know that side was a bit negative. But I wanted to end with this slide so as I mentioned at the beginning of my talk, while this term of research software engineering is is quite new to me. I feel like you know it's something that I've been entrenched in for my career and it struck me so recently I've been mentoring different kinds of students supervising different students and a medical student who wanted to do a research project. And he actually had worked at a software engineering company for quite some time. And it's really interesting because he is asking lots of great questions, you know about some of the research that we're doing. And a lot of it's based on, you know, but we don't have that we don't have this standardization and we don't have this framework that's version controlled or we don't have this. And it's really interesting because all those are really important points. You know I've got another student who's an undergrad CS major and he comes in and he says well you know we don't have version control but maybe we could do it this way or we don't have this but you know maybe we could you know check this one and that one and he's just just brimming with ideas you know and how do we navigate these kinds of problems and I think to me that sort of seems like the value that fostering this kind of community brings because you have to demonstrate out of the box thinking and be creative with the solutions that that you you put forward because it's different from a formal or I don't know what the other word is a commercial software engineering environment where you know there are much clearer sort of safeguards or things that are put in place. And I think that's really great because I see people getting really paralyzed right and not being able to know what to do. And I think having that kind of research mindset means that you see those lack of safeguards and you see the goalposts and you figure out a way to get there. And so one thing I've also seen which is I think really commendable is this commitment to building new research infrastructure right there. You know we need to get to point X but if it means we have to build a boat and and the order to get us there then then we'll do it right. And it takes a certain special individual I think to to see that and also be willing to do that yeah. And obviously there's a sense of fearlessness that that comes you know when everything is not completely defined because I think a lot of people do get scared when they see that you know it's like well what are the variables how do we not know you know this and this and this but when you think about it you know life research is is completely defined right you're never going to be in a situation where everything is locked down and documented and you know controlled and tested so actually it's much more realistic I think to be in this type of setup where you have to solve these kinds of problems. And I think over the years working with different kinds of research software engineering people. They've taught me taught me a lot of things about you know the importance of reproducibility and everything else that has to come around that. But this commitment to standardization you know I worked with some of the people who did all of that early foundational work in bioinformatics with setting up the microarray. And this idea also of accessibility you know we live in a time now where open access is almost a given. But I still remember you know lots of journals like behind paywalls and things like that and so data and not being shareable and things like that so I think it's really important to not lose sight of that. And also longevity you know how do we create resources that will stand the test time will be there for other people when our jobs are not there or we have moved on to other things. And I think the thing that I've seen the most in my career and watching other people as well as that it is a training that requires a sort of style of apprenticeship right so we mentor people we bring them under our wing we give them projects to to work on skills, and I think that very naturally lends itself to this idea of mentorship. But then how do we take that on a bigger scale and build a really robust supportive community. So with that, I'm going to end my talk and sorry for going a bit over but I have to acknowledge a lot of people my career. I lived in the US for 16 years and through that process had lots of really people are really wonderful people that I worked with. While being in Australia for the last six years I've also been really fortunate to have some really amazing researchers to work with and also and hopefully able to hear from you and any thoughts or ideas or questions you might be having more than happy to answer them and address them. So, thank you very much. Thanks Jess.


This is Chipping Norton Soccer Club, South Park. This is where it all started for me. Follow me and I'll give you a look. This wall, after before every game, pretty much one touch, two touch. I don't know my brother, my sister, some teammates. So there's a few memories, just smacking it against the wall. The senior field on my right and the minis on my left pretty much, where I played every second week in the under sevens and under eights. My dad was the coach as well. He didn't really have a set position. He was a natural defender I think, from a young age. But yeah, this is a special place for me. I remember also being a young kid, always wanting to hang around the canteen and that as well. So I used to help clean up and pick up rubbish and my reward would be a free 50 cent bag of hollies. This club is massive for me.  And this is legit where it all started for me. So I'll never ever forget this place. This is the field where I played pretty much all of my games for the under sevens and eights. I think a few goals on here, I remember one. It was a pretty long rated shot, just inside halfway. So that one sticks out. It's a long way to come to play here and to play my first games here and now to be involved in my first ever Sydney Derby on the weekend, representing my club, representing Western Sydney. It's a huge honour. I think it's going to be a cracker of a match, a lot of passion, an intense game. It's a Derby, so an East versus West. It's a big game, it's probably the biggest Derby in Australian sport. This is the change rooms for the club, where we used to get ready, the home and away, brings back memories. This is the former president, Ross Anderson. At the time that Bujo was playing here in the minis, it tended to be set up that you'd have a few outstanding players who you'd always see in the figure of the action. Bujo was probably one of those. Brendan came in and did a session for us last year when he was injured with one of the teams and the kids were just... Not only did they do everything he told them to do, which is exactly opposite to what happened in the cage, but they were buzzing about it for weeks afterwards. It's tremendous to have players who give back. To still be in contact and have such a good relationship with the club now, for me it's something I'll never forget my first football club. You know, it's a special place here. The bike makes for a city of subcontinent tier and being at the Derby and being on the pitch and seeing the passion in the crowd as well as on the field. It's a big thing, everybody. The banter is heavy. I know our banter in the last couple of weeks has been heavy all through to trick Bujo up as he runs down the tunnel. Saturday night, I'm proud of him. He's a Chippo boy. And that's what I love, the fact that he still thinks he's a Chippo boy that he's said to move before. So to have somebody that proud to be a member of that club is a great thing.


What's up everyone? Takedown here. Welcome back to another comic book video. Today I'm going to be focusing my attention on some X-Men comics and sharing those with you. If you guys remember a while back, it did purchase over 2,000 comic books, and I'm going through them slowly and sharing what I find with you guys on the channel as well. I purchased them to resell, but let's face it, you guys know I'm a collector, I love comic books clearly, so some of it is going to be staying in my personal collection. So mainly what I collect for myself whenever it comes to comic books is comics from the 80s, mainly Marvel, and that includes stuff like X-Men, Fantastic Four, Spider-Man, and other stuff like that. And luckily that's what we're dealing with today. Mainly all of these are going to be from the 80s and they're all X-Men, so let's just get right into it. Up first, we have issue 190. That tells you how long X-Men has been around for. That is an amazing cover. I absolutely love this comic.  I believe that was 84. Next we have issue 203. It almost looks like an Elvis or something. Not 100% sure who that is, but that would be very interesting to read. We have issue 204, and that looks so cool. It looks like they're in Japan, something like that, based on the fonts here in the building, but that's a really cool comic. Now I should mention that most of these are not boarded. They're all bags, but if there's any that I'm keeping for myself, I'm likely going to board them and change the bag just to have something that is good in my collection, something that will last and not get any damage. We have issue 216. We have Wolverine over here in the corner. That's really cool. I like that. Absolutely love X-Men. You guys know this. 218. I'm not sure what's going on there, but definitely going to have to check this one out and give it a read. 219. Don't know what happened there. That kind of looks like Storm. Can't really tell. Face looks a little distorted. Definitely not Storm now that I see it. 220. Very interesting stuff. Absolutely love these, and they're all in really, really great condition. 223. That's cool. I like the fangs there. That's going to be an interesting one to read. This one has a nice classic car on the cover. 224. Absolutely love that. That's going to be cool. 225. Colossus right there. Absolutely love that. I don't have many comics with him in it, so I love getting them where he's on the cover because that's really cool. And we have 227. The Fall of the Mutants. That's going to be a nice one to dive into. That one might have to stay in my collection. Now I did pull out some X-Men for my personal collection. Not sure when I'm going to show you guys what I'm keeping in my collection, but that is really cool. We have 228. There's a lot of doubles as well for the X-Men, so I'll be able to pull some from there and keep and also sell the other ones. 229. Huge battle scene here. You have Wolverine. You have Colossus, Storm. That's cool. 230. This one's bagged and boarded. Some of them, actually most of the ones left for this video are going to be bagged and boarded. That's really cool. Not all of them in the collection was. We have 231. Colossus right there. 232. That's going to be really cool. I love the artwork of these. This is 86. That's really cool. This one's not bagged and boarded, but that doesn't matter. 233. Yeah, 233. Wow. Just throwing a car at Colossus. I'm not even sure what that is, what is going on for the story. So that's going to be really cool. 234. It kind of looks like whatever creature that was is taking over Wolverine. So definitely going to have to read some of these before I sell them and see what is going on. We have 236. I can't really see what it says here with the fonts, but that's really cool. We have 237. Logan is on a train. Logan in possibly Rogue. All aboard the mutant train. That's going to be very interesting. I wonder if it's like a Holocaust scenario where they're trying to get all the mutants into one camp. That's what I guess just looking at the cover and seeing that's a name. 238. Huge fight scene. It does look like soldiers they're fighting. So maybe I'm right. Going to have to definitely find out. No spoilers please. Issue 239. We have Sinister. Cool. That's going to be nice. 241. Ooh. A son for the Goblin Queen. Hmm. Honestly not familiar with that, so I'm going to have to check that one out. The originals are back. X Factor. 242. I like the purple. That just makes it pop. I like Beast. I don't think I have many with Beast in it or Iceman. So that's definitely going to be interesting for me. 243. Huge fight scene happening. Going to have to find out why. Really, really cool. 245. Clauses looks shocked. Logan doesn't really care like normal. That's definitely going to be cool. They have all guns. It looks like tanks are pointed at them. Very, very cool. And the last comic for today's video that I'm going to be sharing with you guys is Issue 246. His name is slipping my mind right now. Not 100% sure. Comment down below who this is. Just slipping my mind for some reason right now. But that is an amazing comic. So like I mentioned, I do collect mainly from the 80s. Absolutely love these comics. These ones here I think I've already gone through and pulled out the ones that I want to keep in my own personal collection. And there was some doubles that I will be keeping my personal collection. But for the most part, these comics I will be selling. Now in my area, people sell X-Men comics for around $5 each. I'm not going to do that. Likely I'm going to sell them between $2 to $3. Likely $2 because that seems really fair. And that will let people know that I want to move inventory. They don't know that I'm making profit even at $2 each. So they'll probably just start purchasing them. I might lot them together all 50 together. Or I might separate them and sell them individually. Say $2 each. Something like that. But hope you guys enjoyed this video. There's going to be more comics to come. I have a lot more comics to go through. There's going to be some more X-Men. Some more Spider-Man and a bunch of other stuff including some of the most expensive comics that were included in this lot. So hope you guys enjoyed this video. I'm going to leave this one here. Please take care. Peace.


Good afternoon and welcome to our webinar on computational methods for collecting data from the web. This is part of a larger UK data service training program and it's focused on new forms of data that social scientists may find productive in their research and teaching. So some upcoming webinars, we have Getting Data from the Internet and two more on web scraping and we have some past webinars that focus on agent-based modelling, which is a simulation-based method for social scientists. First, I hope everyone is well under the current circumstances and thank you to those joining us live during this challenging period. If you have any questions during the course of this, as Gillian said, type them into the question box and then at the end I'll read out every question and answer it to the best of my ability. You may have noticed that due to an understandable scheduling conflict, Professor Alistair Rutherford is unable to join us. Luckily for you, as you notice, I am one of the co-authors on the paper we will be discussing, so you are in capable hands. So the aim of today's webinar is to show you how we employed web scraping techniques to generate a new, linked administrative data set to evaluate a regulatory intervention. Specifically, we looked at the causal impact of attempts by the fundraising regulator to incentivize charities to contribute to the cost of regulation. So this is quite novel in the charity sector. Usually government funds regulators. In this case, the charities were funding the regulator. We wanted to test how effective that incentive was. And you can get the paper which is recently published at this DOI and the materials underpinning the paper. So the web scraper and the data analysis scripts are all available in a public repository also. Let's take a look at what we're going to talk about today. Of course, we'll describe what web scraping is as a social science research method. We'll discuss why it's valuable for social science research. Specifically in the context of our paper, we'll see what problem it solved. We'll also look at how we actually implemented it in practice. Which programming languages did we use? How did we actually run the script? And of course, most importantly, we look at the results it produced for our social science project. And we'll also make some general reflections and critical points about web scraping itself as a social science research method. I'll answer your questions and I'll point you to some further learning and teaching resources in this area also. So what is web scraping? Firstly, it's a computational technique for capturing information stored on a web page. Computational is the key word here because it is possible, as I'm sure you're aware, to manually collect data from a web page. You can highlight the words, the paragraphs, you can right click on the image and save to your machine. However, performing this task manually carries some considerable disadvantages, as I'll show you in our specific example. Web scraping is generally implemented using a programming script. By that I mean some executable code that's written in a programming language, so OR or Python, Java, C-sharp, etc. So there are software applications on the market that you can use to do the web scraping for you. In a future webinar, we'll show you how you can use Excel to perform a web scraping task, but to have full flexibility and power, you're going to use a programming language and you're going to write your own code. It is relatively simple to implement web scraping techniques using open source programming languages. You do not need to be highly computationally literate, you don't need to write screeds of code. Web scraping is quite a common task in computational social science with lots of documentation and lots of examples for you to learn from. So if you're feeling apprehensive at the outset, web scraping is a great entryway into computational social science. So I'll show you an example from our research area, which is the charitable sector here in the UK. We have a regulator of charities, the Charity Commission for England and Wales, and every month they provide a data extract. So this is a census of all registered charities. It also has their financial information and lots of other organisational characteristics. Now that data download is useful and it is used for productive research on the sector, but there is an abundance of extra information available elsewhere. So I can show you. So for example, the Charity Commission has a public facing website. So I'm just looking at a sample charity here, which is Oxfam. I'm sure it'll be a familiar organisation. We have an overview page. We have some financial information. We can see the charity's documents, its annual reports that it submits. We can get a list of trustees and we can see where that charity operates globally. Now why do we need this website? Well, in the data extract, we do have a lot of the information that's available here in the overview. So we can get its unique ID. We can figure out what type of activities it undertakes, and we can understand who the charity helps also. However, some really valuable information that we might be interested in would be the trustees of the organisation. And as we can see here, it also lists which other organisations a trustee is on the board of. And that's a fantastically interesting piece of information. We can look at networks of trustees, what do the networks look like amongst the most successful charities, charities that get into trouble, etc. And the financial information is also available in the data extract, but it actually documents themselves or not. So if we wanted to perform some text analysis of how a charity reports on its annual performance, we would need those documents themselves. So we've got a public website. We've got a data extract. So if we want to build a complete picture, a complete understanding of the charity sector, we need to collect data from multiple sources. So as I said, we could collect this data manually. We could go to Oxfam. We could right click on each of these links to the PDFs. We download them to our machine and we could do some content analysis or some sentiment analysis. But what if we're not just interested in Oxfam and we're interested in all humanitarian charities in England and Wales or we're interested in all 160,000 registered charities in England and Wales. Therefore, we need a different approach and that's where web scraping comes in. So why collect data from the web? So I've hopefully given you some reasons to do it. Web pages themselves are a really important source of publicly available information on social phenomena of interest. Unfortunately, the current COVID-19 public health crisis is an excellent case in point. The UK government has a dedicated website with some daily updates on actions individuals can take. The World Health Organization has a very informative website also produces daily PDFs known as situation reports. So web pages themselves do contain very, very rich and research valuable information that we might want to get a hold of. And as I've shown as well, web pages can store a range of different data types, including files, text, photos, videos, lists, etc. All which may be collected and marshaled for research purposes. And finally, once collected, the data can be reshaped into familiar formats. Data that's stored on a website will be in an unfamiliar format. It'll be in something called HTML. But it is possible using programming techniques to reshape that into a CSV file, for instance, and then link it to other sources of social science data. So the research problem we had that we thought web scraping would help us solve is the fundraising regulator for England and Wales. It's a statutory regulator of fundraising activities. So a charity that operates a shop and sells goods to raise money for the organization. Or if you give money by direct debit to a charity, those activities are now regulated by the fundraising regulator. And this is a fairly new development since 2016. It came in the wake of some prominent fundraising scandals that some of you may remember. These were aggressive targeting of donors. There was questionable data sharing practices amongst the larger charities. So there's a need for statutory regulation in this area. As I said previously, the regulator is partly funded by the charities and organizations it oversees. This is quite novel globally for charities for the charity sector. And of course it opens up opportunities for regulatory capture also. So organizations are not compelled. So there's no legal mandate to take money from charities to cover the cost of regulation. However, the regulator does expect certain types of organizations to pay a fee. So those organizations are the ones that spend £100,000 or more on raising funds for these for their organizations. So if you think of something that the largest most successful fundraising charities, you've got Cancer Research UK, Macmillan, for example, they spend quite a lot of money justifiably raising funds so that the charity can carry out its activities. So what the regulator did was say, for those charities that it expects to contribute, they would threaten to name and shame those organizations if they didn't pay this voluntary fee. So they were trying to put some pressure, they were trying to incentivize certain organizations and to pay that fee. So in our research project, we were interested in evaluating how successful this attempt to incentivize payment of the fee was, which presented us with our core challenge, collecting data on which charities have and have not paid this fee. So first, method, which is requesting data from the regulator directly. And they were very helpful. They provided us with a PDF file of all the charities that had paid the fee at the time we made the request, which was a couple of years ago. So however, the list already contained over 1000 records, which would require considerable manual input to extract the list of charities from the PDF and put it in a more tabular data format into a CSV file. And it's also meant that our list was essentially out of date once it was transferred to us. Therefore, we needed a different method to get a hold of the data. So the fundraising regulator operates a public website, and it lists a directory of all the charities that have paid this voluntary fee. And they also list expect to pay the fee, but haven't. And these are listed in red. So for example, alphabetically, we can see these are charities here in the UK, who have paid the levy. And they're across a range of different sizes. So there's small charities, they're large charities. So Aberystwyth University here has paid the levy fee as well. And how this is structured is we have multiple webpages as a list of charities that have paid the fee. And each individual charity has its own page also. So for this organization here above and beyond, we can see it's a large charity. It's paid. And the crucial second piece of information we need is its registered charity number also, because then that allows us to link back to the open data that's made available by the Charity Commission for England and Wales. So the problem is relatively simple to map out. We have a list of webpages. We have certain pieces of text information on those webpages. We would like to scrape and we would like to export into a more usable format, link to existing data, and perform our analysis. So let's take a look at the logic and skills behind web scraping as a general technique. So we begin with a web page that contains information we are interested in collecting. Therefore, we need to know the following. We need to know the location, which is also known as the URL or web address, where the web page can be accessed. So as I showed you previously, the directory can be accessed at the fundraisingregulator.org.uk forward slash directory web address. Once we know that, we then have to locate the information on the web page that we are interested in collecting. So where on that directory page was the information about whether it was a large charity, whether it had paid and its charity number. So then we need to find that information as well. Once we know those two pieces of information, then we need to do the following. We need to request the web page using the URL manually. That's equivalent to you just going into a web browser, copy and pasting in the URL and pressing enter. And the website then should appear in your browser. But we need to do that using a programming language. Once we request the URL, we need to parse the structure of the web page so your programming language can work with its contents. As I said previously, a web page is written in a language called HTML, which uses tags to delineate the elements that make up a page. So for example, there are tags that identify tables on web pages. There are tags that identify paragraphs and links and images etc. So you need to tell your programming language that we're working with a website. Once it knows that, then it can navigate the tags and it can extract the information that you are interested in. So for us with the fundraising regulator, we are looking for a paragraph tag or a P tag and we want to extract the text in between those P tags that give us the registered charity number and its levy payment status. And finally, we want to write this information to a file for future use. So we're going to extract the text. We want to format it in a tabular format and then we want to export that, save it as a CSV file or an Excel file or a .txt file that we can use at a later date. So with these general principles in mind for any web scrape, specific to us, we needed a programming script that will do the following. So we needed to iterate or loop through a list of web pages and extract the charity number and its levy status. So we needed the unique idea of an organization and whether it had paid or not. We needed to perform this task on a routine basis. So for us, we wanted to schedule this programming script to run on a monthly basis so we could get up-to-date lists of who had paid the levy. So we can take a quick look at the code itself and as I said, we will be releasing this code in a public repository quite soon. Make that look a bit bigger for you. So what we decided to do was write a web scraper in the Python programming language. Python is a general purpose, easy to learn, open source programming language. By general purpose, you can use it to create a web scraper, you can use it to create a website, an app, host a website, it's a general purpose programming language. It's easy to learn because it's a high level and what that means is that it uses its English language based in terms of its variable names, its function names, how you write it is a bit more like how you would write English, which is good and it's open source, it's free to download and it's free to change as well if you're so inclined. So our scraper specifically, as you can see up here, it only requires three separate packages or modules so it doesn't require lots of downloading additional packages, it uses a set of core Python packages to perform this task. Here's the pieces of information we need to know, we need to know the website and where the information can be accessed from and then we need to create a... So what we do then next is, once we know the URL that we're scraping, we do some project administration so we create a new output file, so we create a new CSV file containing these headers, so the Charity ID, the Charity name, what kind of regulatory type it is etc. So at the beginning of our scrape we set up an output file that we can write all of the scraped information to. Once we open that file what we do then is we construct the web address, so that's a base URL plus the web page we're interested in scraping and as you can see on the fundraising regulators website you know page one had 20 or so charities, page two had 20 or so charities etc. So we loop over those page numbers. This very small piece of code here is the command that actually requests the web page so that's again equivalent to you going into your browser and saying you know give me this web page. So Python does that for you in a very simple concise way. Then we need to parse the web page so basically here we say take the web page that we just requested and tell Python that it's an HTML structure and once it's an HTML structure then we can start accessing the tags and the information that we need. So this will be shared with you at a future date so we don't need to go into it in too much detail but essentially we're going through elements in the page and so we're looking for an element called charity title and then we're scraping the text from that element so that gives us the charity name for example and this is doing something similar if we find a list if it's a registered charity we cycle through the list and we pull out the information we need and then right here at the very end once we've collected all the information we need and we use a writer command in Python and we export all that information to the file we created earlier. So what we get at the end is a CSV file with one row of power organization that we find on the web page and we have its charity ID, its charity name and whether it paid the levy or not. So I realize it's better obviously to run this code so in the future webinars we will be showing you how to actually execute this code and you can follow along but for now just while we're tidying up this project we said we'd give you a look at the code how we did it and but you'll also be able to see for yourself quite soon what we did and we saved and shared all of this work on a publicly available repository which I'll show you in a moment as well. So what was the point of all of this? Well we had a social science research project we had what we thought was an important question the whole point was to generate data that can produce some insightful results. So we took that scraped data in the CSV file and we linked it to financial data from that data extract I showed you earlier and this gives us about 4,000 organizations on which we have detailed financial information. So these are the 4,000 that spend a certain amount that therefore we know exactly what they spent on their fundraising costs for a given year. What we did then is we exploited the sharp threshold by fundraising expenditure so remember the regulator said if you spend 100,000 pounds or more you're expected to contribute to the cost of regulation. What that gives us is a research design known as regression discontinuity and which is a quasi-experimental research design and regression discontinuity is suitable when a cutoff point is used to allocate units to a treatment or an intervention or not. So 100,000 pounds was set as a threshold. If you're above that threshold you're expected to pay. If you're below that threshold you're not expected to pay but of course you can if you want. What makes it a quasi- experimental research design is charities could not manipulate precisely which side of the threshold they fell on. Very wisely the fundraising regulator decided to use the 2014 set of accounts in order to set the threshold so charities could not go back two years into the past and amend their accounts to ensure they don't fall on the 100,000 pound side of the threshold. What that means is that close to this threshold charities around this threshold are essentially very similar with one difference which is one fell on one side and one didn't. So for example an organization that spent 90,000 pounds on fundraising is very similar to one that spent 110,000 pounds with the crucial exception of the latter being expected to make a voluntary contribution and the former not expected. So we conducted the analysis in Stata using the RD robust package by Kalaniko et al and so I encourage you to check that out. It's a very good package and very well documented. So you can see in this project we've combined Python and Stata quite easily but out much extra work at all. So what kind of findings did we produce? Well first there's a clear correlation between what you spend on fundraising and the probability of paying the voluntary fee. So if you look at the x-axis if you spend between 0 and 100,000 pounds the probability of paying that fee is about 20%. So about 20% of organizations within that fundraising expenditure band paid the fee even though they were not subject to being named and shamed by the regulator. But we can see at the threshold there's a considerable jump or discontinuity in the regression line. So above the threshold about 60-65% of organizations paid the fee and it's that discontinuity between the two regression lines that gives us our estimate of the causal impact of the threat to name and shame. If we expected the threat to be ineffective what we would see is the trend line left of the threshold carrying across over the threshold and onto the right-hand side of the x-axis. But instead we see a big jump just at the threshold which corresponds to the regulator's threat to name and shame. So we do get some quite convincing evidence that the regulator's threat to name and shame was very successful in getting charities to pay this voluntary fee. And how successful was it? Well we could use the results of the regression discontinuity to make predictions of what would have happened if they didn't threaten to name and shame. So the top gray line is exactly what they raised from charities. It was just shy of 1.7 million in its first year and we can see that the probability of paying increases again the more you spend on fundraising. If we made a very naive assumption that just at the left of the threshold where the line cuts the vertical line that's about 20%. If we carry that across we would have expected the regulator to raise about 400 000. That's obviously unrealistic because we know that the probability does increase with how much you spend. So basically if we can carry the upward trend line on the left-hand side over to the right we would have expected the regulator to raise about just shy of 800 000 pounds. So we conservatively estimate that this threat to name and shame covered about half of the regulator's budget. If they didn't do that they would have had half the money available to them for regulation in its first year. That introduces lots and lots of interesting substantive reflections. Is this an approach that will be successful going forward if every year the regulator has to say you can be named and shamed for not contributing to the cost of regulation I'm not sure that's a successful long-term approach but as a one-off intervention it does look to be a very successful approach. Charities do seem to be concerned with their reputation therefore they will pay this voluntary levy to avoid negative reputational damage. So let's summarize some of the pros and cons of web scraping in our context but also just more generally for social scientists. So I think hopefully one of the major pros and it's probably difficult for you to divine just me showing you some code but when you actually get to interact with code as we'll show you in future webinars it is a skill that's relatively easy to learn and web scraping can be performed in you know 30 40 lines of code all of which is very accessible and available online. Web scraping is a very easy to routinize or automate methods so this is particularly important when data are continuously updated so in our example and if a charity stopped paying the levy it would disappear from the website and we wouldn't have we wouldn't have a longitudinal data set where we capture the charity's participation it just simply would not appear on the website anymore. So for us we schedule the script to run every month and but if it was particularly important if we were talking again unfortunately about the coronavirus health crisis it's incredibly fast moving if you wanted to collect data you might have to run daily versions of your web scraper as well. It's not it's not too difficult to do that and we can talk about approaches for scheduling programming scripts. Just thinking about social science in general there's lots of public sector bodies lots of charitable bodies and shared data through their websites so if you want to get a hold of annual reports kind of monthly statistics and figures you're probably going to have to go to a website and you probably won't be lucky enough to go to a data portal and to get some nice clean formatted open data ready to be imported into Stata or SPSS or R for example and so while that poses a challenge the good thing is there's incredibly rich information out there stored on websites that can be productively used in your research and it's also quite easy to format data scraped from a web page it looks messy it looks like paragraphs and images and tables etc and but as our example showed it can be easily formatted to permit data linkage you might be interested in local authority statistics you could go to your local authorities website you could scrape some of that information and then it might be possible to use a postcode field to link to a large-scale social survey for example of course it's not all singing and dancing there are considerable ethical issues and particularly around personal data so as you saw in our example we can scrape information about trustees and that is publicly available information but of course thanks to GDPR it means even if it's publicly available as soon as I scrape and download personal data I have a responsibility for how it's processed and so that does bring me into the realm of you know data protection and of course getting ethical clearance for doing such research web scraping may also contraven the terms of use of a website lots of websites don't want you scraping and they might not explicitly stop you but it'll be in the terms and conditions of using a website you may not scrape this information repackage it share it onwards etc so you do need to be quite diligent and you do need to read the terms and conditions on a website and to avoid getting into trouble probably another obvious con is if the structure of the web page changes your script can stop working currently actually with the fundraising regulator they've just made a change and we've had to update the script hence why it's not available on the public repository we have just now we need to get it working again because it's been two years since we first collected the data of course as well some websites are up to quite up to speed and they can actually blacklist you so they can ban you from scraping the data so your computer has a unique ID when it connects to the internet it's known as its IP address a website can just block your IP address and not even let you request the page never mind interact with it you won't even see the web page so some websites are advanced enough to do that others may allow you to scrape the web page but they may put some kind of throttling in place where you can only request the web page once every 30 seconds or you may only be able to interact with it in a certain way it also requires good internet connectivity for extended periods of time there are ways around that but I'm sure like me your university laptop is tightly controlled by IT services it can be difficult to leave it open for more than 20 minutes before it cuts into sleep mode for example there are ways around this you can have your own personal server for example but it's worth bearing in mind that if you have a very intensive long running script you need to have a think about your computer setup as well so thank you very much for joining us I really appreciate giving up your time particularly at a period like this so I can see some questions coming in so I'm going to read those out one by one because I'm guessing you can't see those so the first question is there an alternative package in R? yes there is that's a really good point you should be quite agnostic about the programming language you want to use if you're an R user absolutely you can do all of this in R you can do all of your text mining in R you can do all of your connecting to the Twitter API in R absolutely for us we began with Python so we've continued to use it but the techniques are very similar so absolutely I'm not advocating Python it works well but if you want to use R you absolutely can we had a question as well about which part of your code was for scheduling that's very good in the snippet I showed you we didn't have this the scheduling code what we did is we had a slightly more advanced version of the script that script is kept on a Raspberry Pi which I'm sure some of you have heard of that's basically a very small portable personal computer you can leave the Raspberry Pi on all day every day and so that's what we did we moved the script to a Raspberry Pi and then we basically scheduled the script to run once a month as well and we also had on the side of this we had a Twitter bot which after every scrape would start saying these are the new charities that have just paid the levy as well so you can also connect your script to Twitter or Facebook updates if you want to do some kind of kind of knowledge exchange or some public information using your scrapes data as well and we have a question about to what extent does the speaker rely on UK copyright law to scrape and reuse and store the data that's a very good question as I've mentioned GDPR applies when you're scraping personal data but of course copyright law does apply as well to the scraped information which brings me back to my point that websites will have a terms of use and if you read the terms of use then that should give you an idea what further use is and repurposing you can make of the information provided on a webpage so yeah as a general principle don't create a web scraper and then figure out whether you're allowed to do that read the documentation on the website us specifically because we contacted the fundraising regulator to get data in the first place we did clear our web scraping activities with them so they were fine with the use of data we have a question about can you scrape all kinds of websites like social media pages for example excellent question tentatively yes if it's a if the website is available at a URL so if you can type that web address in and it'll appear on your browser then yes you can write a scraper with social media web pages they provide a different approach which is to use an online database and then you're allowed write code that connects to that online database so you don't actually have to scrape the page there's a database that contains public available information that you can write code to access so that's a really good point as well you don't always have to use web scraping and being brutally honest and it always helps to contact the website owner first and say can I actually have this data as a zip file that might save you quite a lot of trouble in the long term as well we have a question here do you advocate using a Jupyter Notebook since you are using both Python and Stata yes so for those of you who are unaware a Jupyter Notebook is an electronic document that allows you to mix live code the results of that code and some narrative information as well so it brings together basically your Stata syntax file which is your journal article to put it simply in the next two webinars we'll be covering how to use Jupyter Notebooks to perform web scraping so hopefully if you can join us in April I'll actually provide you with a Jupyter Notebook that you can run yourself to perform your web scraping but yes as a general principle it's good to keep all of your work in one portable transparent document that you can share I have a question here about what about MATLAB what about it no of course I haven't used MATLAB for web scraping so I can't say for certain if it allows you to perform that functionality I'll have a look feel free to contact me directly my details are on the page and I will look that up for you that's no trouble out of curiosity how long did it take to collect the data through web scraping for this paper I'll interpret that in a broader sense of how long it took to write the code two years ago when universities were on strike about our pensions was the two-week block when we learned how to do this so it took us about two weeks to learn Python in terms of the scrape itself it's reasonably quick if we're talking you know if we're talking about a thousand organizations split across 25 web pages you're talking about maybe a script that takes a minute to two minutes to run it doesn't take very long that's because you don't actually have to kind of stagger your scraping but if you connect to an online database or an API for example usually you do need to stagger you are requesting of the information so collecting data from a database can take quite a bit longer from a website it can be reasonably reasonably quick you won't need to leave your laptop on and go do something else for 20 minutes a very interesting question any extra considerations for using this technique with private sector companies so is this private sector companies performing the web scraping or is this scraping information from a private company's website if it's scraping from a private company's website that probably brings you into a bit more ethical issues the terms of use probably explicitly ban scraping that website so you do need to be careful if it's a private sector company performing the web scraping then similar to academic use I mean as long as you're complying with the terms of use of the website you're complying with copyright law and you're complying with GDPR then yes you're perfectly entitled to perform web scraping excellent and we've had again similar questions about people who prefer or if you use or absolutely use that you can use Stata for example Stata has a a new web scraping functionality in version 16 I haven't used it myself but when I get that version I'll absolutely try it as well so broadly speaking this is a research method you can employ across multiple programming languages or even things like Stata and SAS for example and again is there an ethics code for web scraping I mean that's an excellent point I don't think there is a specific document I've seen that says here are best principles for web scraping one might exist if one doesn't exist I think I will put something together that's a really excellent point so how easy or difficult is it to debug your web scraping code it's reasonably easy if you think there's an issue with your code you can insert try accept clauses so you can try to do something and if it doesn't happen you can tell the programming script to print out an error and tell you what's what's going wrong and you can also very simply say at each state of the each stage of the scrape and print what's happening so you can have lots of print statements which which you know show the results on your screen can show you how successfully the script is performing so debugging is reasonably easy it's not a it's not a very difficult computational task in that sense this is a really good point as well what if the information is in a PDF and you want to actually write a scraper to extract the information from the PDF excuse me yes that's entirely possible in Python there's a PDF converter package I think it's called that will actually understand the structure of a PDF and it'll extract the contents excuse me so yes absolutely you can use web scraping to collect information from within a document as well so it's not I suppose web scraping but it's it's using scraping techniques to extract information from PDFs files yeah and a similar similar question I think that we had previously can you apply web scraping to social media websites not web scraping itself but you can write very similar code that connects to the online databases of these companies so Facebook Instagram Spotify Wikipedia The Guardian these all have online databases called APIs that you can download data from so it's not scraping it's requesting information from online databases and our webinar on the 30th of April goes into detail on how you would do that as well nearing the end what are the benefits of using coding against I think pre-existing software packages so yes so as I said Excel has some functionality for collecting data from a website there are lots of web scraping programs that you can download from the internet for me because it's a relatively simple task to learn I think it's worth writing the code yourself and then having the flexibility to adapt it to control when it is executed to update it when websites change I'm a big advocate of writing the code yourself if even if it's just for the intellectual dexterity that it produces in you you know coding is a very challenging but rewarding task and I think it's worth engaging in in general yeah and then final question I think so can you link web scraping with sentiment analysis absolutely web scraping is a data collection technique once you've collected the data and put it into a CSV file for example you can import that file into well for me Python and I would use the NLTK package to perform some text mining and some sentiment analysis and I think it's text minor in the or software package so absolutely if you can think of web scraping as a data collection social science method which can then be used to produce data for text mining sentiment analysis network analysis conventional quantitative statistical modeling approaches etc absolutely okay one more one more I'll be good to you so how flexible can the programs be if you're scraping from a number of web pages in different formats that's a good question usually websites are built so the web pages of a website will be built with the same file extension so they will all be HTML files for example but of course some pages might be different that a web page that has a piece of functionality in it might have a different file extension for example so yes absolutely there can be some challenging situations where if you need to scrape five web pages they all have a different file extension it may be it may be slightly trickier to request those web pages and then scrape them so yes yeah it can it's not a it's not a problem I faced but it's certainly something that might be possible yep and finally one more can we use web scraping for qualitative data absolutely in fact that's probably most useful for collecting qualitative data so for us I know we collected the charity number and whether it paid the levy or not but those were pieces of text information you know we didn't scrape anything that was a number for example everything that we scraped you scrape it as a string you know as a series of characters so absolutely you could scrape entire paragraphs entire web pages of text and save those for content analysis at a future at a future point yes absolutely it's probably particularly good for qualitative information and your follow-up question yeah is there a bibliography yes on the 23rd of April in the next web scraping seminar that I'm going to host I'm going to show you how to actually do it and you can follow along and I will suggest lots of additional reading and some excellent resources so yeah don't worry about that excellent so I will show you a couple of links that you may find useful so we've got a public repository with the UK data service where you can find the materials underpinning this in future webinars quite shortly we will be publishing this webinar on our YouTube channel so you could watch again if you fancy you can also reach out and help ask ask some help with us you can also just contact me directly I'm happy to answer further questions or you can get in contact in general with UK data service we're on Twitter we're on Facebook and I can show you that repository really quickly yeah hopefully you can see this so at the UK data service github page and we now have a new forms of data repository so for example for the web scraping series we have three webinars you can find details about each webinar and then you can find the resources underpinning them as well so the slides from today are currently up and for the future webinars I'll be posting bibliographies Python programming scripts that you can amend yourself and yeah reading lists etc so we'll be publishing lots of self-directed materials here over the next couple of weeks so please join us