a lot of stuff, so I'm just gonna get going. So, thank you, David. My name is Paul Bruce, like he said. I've got a lot of stuff to talk about. Mostly it's the fact that I work with a lot of different groups, and so I get to see a lot of different perspectives, and mostly my hope is that you'll take this not as I'm an expert in any particular thing, although I have my wheelhouse. You can get the slides here. There's a lot of links in them. I hold many degrees in awesome stuff from a long time ago, not really, but ultimately my wheelhouse is performance, right? So, high-scale systems. I'm a performance and scalability engineer. That's what I do. I also work a lot with CIN testing automation, right? Additionally, what I call learning frameworks isn't just artificial or machine learning frameworks. Come on, we're the original intelligence. Human beings need learning frameworks, and so I work with a lot of different groups to figure that out. My most recent thing is Creative Thinker's Toolkit, if anybody has ever heard of that. I also am starting to build this concept called Forward Dialogue, which actually, we did a number of things there. Some of the screenshots here. So I also work with this Selenium conference in two weeks, and I'm re-engineering the grid. I don't know if anybody's familiar with Selenium Grid, but it's pretty widely used, and there are vendors for that, but there's a lot of people who spend their own. And in terms of DevOps, we need visibility on how healthy those grids are. So I'm baking that into their code base because it's not there yet, and I think it should be. And I'm going out to that conference to volunteer as well. I also do a lot of stuff at home. I try to make sure that ideas that go, that work really well in work, if it works at home, that's great. Ideas at home also can get brought to see the light of day at work. So this is just organizational planning. So that's my dad life thing. I also do a lot of idea generation, particularly for the company I work with, full-time called Neotis. They're up in Lexington. They do load testing. They've got a booth out there. The only testing vendor here. And then also I do a lot of collaboration with people, particularly because I am not special. There's no unique qualities about me. I'm not a special snowflake. The idea is that the ideas come from collaboration. You can be the smartest person in the world and only have a limited amount of ideas. But when you figure out how to collaborate and communicate effectively with more than just yourself, if you're even lucky to do that, there's a wealth of ideas out there. So I work with a lot of different groups. This was, I guess, one month ago and two weeks ago. I did a thing about hiring and recruiting in DevOps, which turned out to be really useful for a lot of managers in the area. This is going to be fun. And hey, look, this was yesterday, right here. Who was in the open spaces yesterday? All right. There's a few hands. That's great. Thank you. I think you guys had a lot of really great ideas around the topics that you cared about. And so I have the luxury of just kind of peering into other people's conversations and then following up with them, which is really great. I also bring a lot of work home to read. I do a lot of reading, a lot of commuting, a lot of learning in that situation. And this is one of the best books I've ever read. Who's read Sightly Reliability Engineer? Come on. More hands. Sort of more hands than open spaces. For those who haven't, please, I encourage you, implore you desperately, at least just look at some of the first couple of chapters. There's some really important ideas in there.  And she turns me on to a lot of different ideas, like for instance, the fact that she wanted this book because she saw it and she can read. She's been reading for years. And I didn't realize that, but software engineering was a term coined by Miss Hamilton. So that's really cool. So why am I here? Is this a technical talk? Yes. Don't worry. We'll get to the end of it. I think you guys have probably had enough Kubernetes up the yin and yang. You know what I mean? Like, there's meta about our jobs we don't pay attention to that significantly impacts our psyche, right? What is it, James? Talked about burnout this morning? That's not just a technical problem. That's in every field. And it's just as important to talk about this kind of stuff in technology. So what I'd like to say is that there are things that are more important than your work. Who would tend to agree that there are more important things than your work? Hold on.  I'm not kidding. This matters to me. All right.  Excellent. So sorry about this, by the way. There's more important things than this as well. And empathy is a really squishy term to some people. It's not to me because I've had to go through young dad life. I've had to step into the shoes and see life from a very different angle. And I've started to do that for everybody. For every single conversation I have. For every single business venture that I have. For my customers. For the people that I support in my full-time job. I look into what is most important for them. And to be able to at least step into the feelings and thoughts and understand where people are coming from is super important to be able to communicate efficiently. So my other thing here, the concept that I pulled you in because of the talk title is performance. The performance imperative. What do I mean? There's a lot of really important things in software right now. Equifax. There's a lot of aspects of software that are being ignored for various reasons. This is one that fortunately I have the luxury of talking to a bunch of SREs. And realizing scalability, reliability, availability, these all bake up into performance. And so what I mean by performance imperative is that quite frankly, we don't need to see errors. We don't need to see failures. And when people see stuff like this, we turn around and use something else. Do I have to say anything else about this? Also, by the way, being able to predict, we can't predict very well. The future is untold, so we need to respond quickly. That's the whole point behind reliability engineering is to be able to bake this stuff in to make it easier to handle real-time stuff. I'm getting no love from AV. That's okay though. Everybody has this problem. This is Amazon on Prime Day. They knew about this how long ago. And it's okay, like, stuff happens. It's okay. But honestly, this is how we typically live when it comes to availability of things that we didn't necessarily plan for. It's fine. It's fine. Okay. So this is something I actually realized a couple weeks ago was that performance, it's imperative. And I realized that there's a framing here that I think you'll all pay attention to. Obviously, the user experience. If it's slow, it might as well, you might as well move on from it. We all get that user experience thing. But what is poorly performing code and software due for developers, right? Well, first off, it's really easy to push code these days. And so that stuff can be really poorly performing. And you might not know because this stuff is so complex and hard to solve. The other thing that happens is because it's hard to predict and our systems are so complicated, the feedback loop oftentimes ends up being really slow. And so we ignore it, right? These are the developer experience moments where we kind of just say performance, I'll deal with that later, or security, I'll deal with that later too, right? But on the operational side, and this is, I think, where we pay, maybe in this specific local group, it used to be a lot of sys admins, and now it's a lot of developers too. But there's this balance in DevOps between who are we talking to, just the devs or just the ops, because we oftentimes just talk to one or the other, and not the combination. I realize performance in an operational experience, it means, look, we're changing stuff all the time, right? Infrastructure changes, it doesn't have to be code. It doesn't have to be that. It could be a live migration that you didn't kick off that causes performance issues, and so on and so forth. So the rest of this is, and I'll move fast, is three examples of empathy, right? I've now combined two different concepts, and I want to overlap them together. And this is the sticky, just follow it later, but my son realized that saving state on a Nintendo is easier than actually learning how to be a good gamer, and that sucks because I grew up in the 80s. So just consider that when you put something out there, you have no idea how it's going to be misused, okay? So first thing, user adoption, right? Slow is, we're done, right? Slow is broken.  Is it our fault? Is it the network's fault? It's both. So stats. Boy, this is troublesome. There we go. Okay. The world used to be this simple, and now it looks like this. Or maybe worse. Maybe you got some data cubes underneath there, some R models that you have no idea about the predictability or the reliability on a large set. But the cool thing is, for the user experience, the work that I do helps me be able to play around with interesting uses, bizarre and twisted uses, but still effective because, hello, we're engineers, of getting feedback from production. This was an example where a large vendor tried to launch on Salesforce Commerce Cloud, and since Commerce Cloud doesn't really provide any operational visibility into their back-end systems for their customers, how do they answer the question if it will scale? They made a major IT decision to move on to a platform, and oh, by the way, that platform doesn't give them much view into the back-end. There's problems with that, I get that, but how is the customer supposed to know if something can be launched on it if they can't see the cause and the effect, the impact of the load? So we played around with things, and we got rucks at this old stuff from Dynatrace to be able to provide real-time visibility under synthetic load, it was pretty cool. So that's an example of that. The next one is going on to the developer experience, right? What do I mean by downstream rework? Come on, you know what rework looks like. If it's slow, it's broken, and this, I had to draw for somebody who wasn't in the software space, who is still thinking in terms of cycles, yeah, but can't we push performance off? This is just an example of saying, look, if you take a few sprints to do something, you push off load testing using monolithic tools that take forever to get back, or monolithic organizational structures to get feedback back, it's going to be so late that you're going to have to be tripping over your own shoelaces in the next sprint. You're losing story points to something that, not your fault, we didn't know. Maybe it's broken at scale, maybe it's not, but if the feedback loop is shorter, you wouldn't have to trip over the shoelaces. Here's another version of it for the slide deck, and by the way, yes, on Twitter, there's the rest of this, but this is a weekend venture of mine just to say, hey look, let's talk about it in terms of how you would do a commit and what that looks like from that perspective. I did a survey last year with another company that basically said teams that bake performance criteria into their planning process have half as many problems with performance, right? Duh! But why is it not a duh, right? Why are these the leaders? Why do we have to segment the world into the market leaders and the laggards, right? What, because the Dora research says so? There's a good reason to, right? Because there are some people who get it, inherently get it, and some people have learned from difficult experience. So my experience is things have to be the performance-specific stuff, sorry about the slides, but there are a few Cs, right? You need to think about concurrency, you need to think about conditions, and you need to think about capacity. Good Lord, I think my computer is messing this up more than anything else. Here's an example of how you might do that in a story, is actually be able to bake this stuff into a story. Instead of just being handed a requirement that says, oh, by the way, we need the thing to do this many transactions. In how much time? Under what conditions? What is the back-end doing? So consider that the component of time is really important. You could put it on the back of the card, so to speak. This is not just my thinking, it's a guy who works in the space that's really good, Todd. So one of the things that SREs really care about is how do we make prod a less volatile place, right? It's not you can't ship broken code, it's why does it have to be broken and why can't you know about that in real time or beforehand, right? And this, we don't have time to go through this, but this was essentially a story of basically people put hard-coded variables in their testing scripts, and guess what happened? Those testing scripts got promoted, and then the data got deleted in production. Whoops. So these are some of the awesome stories I get to collect in my spare time. Unfortunately, I can't provide you names because I don't want to sully the reputation. But the point is our feedback, our feedback mantra now is early, often, and easy. And we don't have time to dig too much into these things, especially because of the projector. But for the most part, early it should be obvious. APIs, if you're not doing API testing, I don't just mean API functional testing, I mean API performance and security testing. That's the best place to start early. Often, we can't get often feedback unless we're automating a significant portion of things. And peeling off of your main pipeline to get this feedback allows you to ignore the length of feedback loops unless you have really good visibility on that at the team level. So if you're just going, okay, we're going to, you know, dev release that to a QA or staging environment and then do performance testing afterwards. Okay, but that's not the be all end all strategy. You need a more advanced strategy than that to really get early and often feedback. And the easy thing is that, look, there's various appetites. People want to either have devs do their testing or they might want to have a separate team do various testing. Or embed the teams. Whatever it is, the fact is it benefits us all for this stuff to be easy and to be able to fit into our landscape. So again, don't have time for a lot of this, but again, slides are pinned on my Twitter account. So please go back to them. This is solely to say there is a time budget for testing and if we don't respect the time budget, if we expect that hour long performance test to fit in a cycle that typically took what, like five minutes to complete at most, we are going to eject that like a bad habit. So we need to be really conscious and careful about the time intervals that people need that feedback. Additionally, doesn't this look like waterfall? Who thinks this works, looks like waterfall? Okay, there's a note up there that says for every feature or patch or whatnot. I hate to argue with you, but the fact is there's a concept in the goal by Elia Gulrat, a fundamental primary book that predicated a number of elements that are acclaimed at DevOps Handbook, right? The Phoenix Projects. This was fundamental to that. The goal was fundamental to that. And one learning that he had along the way was there are, there is such a thing as dependent events. You cannot deploy something that you haven't planned. Is that not obvious? So the difference is, waterfall used to be this huge bucket of stuff. And now we're trying to micronize that. We're trying to turn that into small chunks, small batch sizes. So last thing, operational experience. If it's slow, what is it? Yeah, for ops people too. If it's slow, it's broken. The first thing we have to do is dispel the myth that there is 100% uptime. There might be an aggregate, but it's never 100%. And your business leaders have to understand that too. Who is a manager in this room? Okay.  No. It's hard. It's very hard to convince the business to say that it's okay to have a failure budget, but in fact it is. And in most cases, we'll start with this one. That's what most people think we do. We have a back room pet-a-server, and it's like, yay. Oh. The other thing is, I heard this recently on site. There was a thin wall behind me between what we were trying to do in one room and what some other loud business, clearly business group was trying to do. And I asked them, what is that? And they said, well, it's our change advisory board. One of many. And I was like, oh, cool. Tell me more. And eventually they got to the point where they said, reboots have to go through the cab. Okay. That's fine. You know, there's a lot of mainframes out there. No, no, no. All reboots on AWS accounts? Yes. There's a point at which if it's actually ephemeral, you might ship your stuff up to whatever it is, the cloud, but if it's not ephemeral, if you can't cycle servers in and out and manage that load, what have you really done? All you've really done is turn the problem of figuring out what your capacity needs to be and owning the hardware to what Adrian Cockroft would say. It's not my problem. That's what hard drives at Netflix was. They realized they could use AWS. It's not my problem to manage hardware anymore. Okay. Well, kind of it is in certain circumstances. So think about it. If you can't reboot a server, it's like, have we really gone ephemeral? Have we really gone dynamic? One last thought about this. I continue to kind of drill in on some of these questions, and they say we must have 100% confidence. And in reality, it turns out to be 100% confidence. It's because we have highly coupled and dependent systems. We have very coupled systems, and those are complex and highly coordinated rollouts that we need. Does that sound familiar? Complex rollouts? So therefore, it hasn't necessarily... All hands are on deck. Automation has been a second thought to getting stuff to work properly. And therefore, that leads to the inability to imbue steps reproducibly. It's just the time spend problem all over again. So I think that a lot of this comes from business leaders not understanding that things don't have to be up to 100% of the time, that there has to be the concept of an error budget. And again, it's problematic to push in an error budget to an organization that just wants your velocity to increase more and more and more, right? Does anyone know what I'm talking about here? I think it's worth digging into for a second. One or two people, you know? People say, ship the feature. Don't deal with tech debt. Tech debt is your problem, not mine. So that march of acceleration and innovation features, features, features. They're not even using the features we gave them last month. So that also has a negative impact on this. But ultimately, one of the things I do is when somebody says, oh, yes, we used the docker, I say, okay, how? How widely is it adopted in your organization? Oh, well, some devs are using it. Oh, what about the test organization? So this is where we actually start to go back and say, the book, a book, a really good book that talks about site reliability engineering, you don't have to be a unicorn to learn good things. And this is one of those clear quotes. And when I look at this and I look with my performance eyes, I'm like, damn, there's a lot of things in here, availability, latency, performance, efficiency. There's a lot of things in here that are under the wheelhouse of what used to be performance. And now it's kind of baked up. There's a lot of things that site reliability engineers have to deal with that aren't the traditional performance engineering crew. It's more than that. It's process improvement. It's efficiency across the entire pipeline. So too much text to read, but I think it's important to say, I wish I could give you specific numbers. I think the only specific number that I think on a regular basis makes sense is if your servers are at 80% capacity of anything, you got to stop and figure out how you're going to deal, how you're going to push in the next server. How are you going to rotate in more bandwidth? Other than that, it really depends. Your numbers are going to be different than his numbers, different than her numbers. It depends on context. What I would say is that there are some clear signals. They're called the golden signals in that book. And the goal here is not to just measure more service level indicators. We don't need more data most of the time. What we need is to align the organization and the objectives of our product managers and the rest of our team to the things that we're supposed to measure to begin with. Probably some of your measurements today are completely useless to decision-making process. Hopefully that's not the case. With the difference, we need to manage the difference between the service level objective that's supposed to be the team and the organization, their responsibility to define for you before it gets to production to begin with. And then cycle through and say, hey, how does production inform us afterwards? So the service level objectives, I think, is where a lot of the gap is in terms of thinking. And there's a couple of things that you can ask, to start seeding that thought process into your PMs, who usually take no responsibility for the performance of things because they're not devs and they're not ops. Whoops, what about planning? So when I mean dynamic provisioning, right, I have to deal with how long ago did you say five minutes? Oh, great. Oh, good. So there's a couple of things, right. What I've heard is that operational expense is a big deal in organizations. CAPEX is also a big deal. Balancing these two things is very important. There's a lot of reasons why. Go talk to somebody who has to do that, right. Get some context if you don't really care or know right now. Get some context because the push on operational is to reduce the operational expense, which ends up being that what we have to do is provide on-demand services. And that goes across the entire pipeline stack. We need to be able to provision all the testing, all the environments on and off when we need them. If only I had you. If only I had you about half an hour ago. This would be awesome. So Adam is an AV sort of expert. Anyway, so what that means, right, if we go all the way through is that, yes, of course you need to have dynamic targets. Yes, we can spin up a new stack on demand when we need to. Great, if you're lucky, right, if you get some main frame time, if you have that or not. Also, you need to have dynamic test data, which needs to be extracted maybe from production, cleansed, and then integrated into your testing harness. You also need to be able to provide dynamic warning and failure flags and update those as changes are made. They typically call them SLAs and tools, but that's a misnomer anyway, another thing in the book. Also, you need to be able to provide distributed load infrastructure, which is particularly my wheelhouse, what I do during the day. And also, you need to be able to provision this and put this into pipelines. So you need to be able to spin all this stuff up and then imbue that in a way, like, for instance, Jenkins files or Circle Orbs, Circle CI Orbs, in such a way where you can execute this stuff and then tear it down. It's really reliable, otherwise ejected right out of the pipeline. The real-time test results too.  Or worse, somebody put something in there, like new test data that borks the whole indexing in the database underneath. So just to wrap up, anytime you want to contact me and we'll share sources, we'll talk about your challenges, I'm down, right? That's why I created a liability company last year, just to have conversations, just to be able to help organizations figure out what's important to them. And that's logistics. But in the meantime, I post regularly on Twitter, for whatever reason. And I also write occasionally, and there's one that's really good that I did earlier this year called well, I wish I could have called it devops testing for dummies, but I did that on my personal blog. What I had to do for the dev exchange for Capital One was call it something else. So it's pretty much the same thing, but it goes through, what does it really take to think about testing if you want like an 11-minute read so that you can kind of go, oh, wow, that's what we lost. That's what we left behind by firing all of our testers and doing something different, but that's an introductory to the way that test engineering would think. So if you haven't done that, I definitely encourage you to. Or just refresh yourself on that. So, thank you for letting me spend your time. Thank you for spending your time with me. And there's plenty of ways to connect with me. But the easiest way is just go right up to the open spaces, and I'll be there for the next hour or so. So thank you.


Thanks, these are like the best MCs ever. Yeah, so, hi, my name is Grace and the title of my talk is Creating Map Visualizations with Open Data and Folium. So that was the title of the talk that I submitted to the PyCon organisers. Actually, I'm talking about Ruby, not just him. So this was the title, but as I was making the slides, the talk slowly morphed into a story about how I use Open Data to find myself an awesome home to live in in Singapore. Yeah, so before I launch into this long story, let me give you some background about myself. So I've been working at 99.co for like a year and a half now. And I look at these like amazing properties in there. Okay, so let me step back. 99.co is a property search portal. So for those of you who don't know what that means, basically Airbnb, but for much, much longer term rentals. Yeah, so I've been working at 99.co and I look at these properties every day, right? So one day I decided, hey, why don't I like move into one of these amazing properties myself? And 99.co's app is like pretty awesome. I can enter my search filters. I can enter like my budget and like the number of rooms I want and everything. But even after entering all these search parameters and filtering out the rental properties that don't meet my expectations, I'm still faced with literally like thousands of listings to choose from. So now I have a problem, right? How do I choose the best possible listing in Singapore?  I can use data to solve my problem because Singapore has this amazing collection of open data. So as a starting point, because I and everyone else can't afford a car in Singapore, I decided that I really want to live close to an MRT station. So the next thing I wanted to find out was which MRT station was the most affordable without being too far away from places like work and play. So to do this, I decided that I would create a map visualization, calculating the average price per square foot around each MRT station. Furthermore, I would color code the prices so that I could create a visualization that would make it very easy to digest all this information at a glance. Okay, so the first step was to map the MRT stations on Singapore itself. So to do this, I got the open data from MyTransport.sg. And MyTransport.sg has this amazing collection of transport-related geolocation data. So they have everything from the arrow markings to the road crossings. They even have the locations of those curved mirrors you see at blind spots. Yeah, and for our purposes, I needed the train station data. So when you download this data, it's in the form of an S3 shape file. So S3 is the Environmental Systems Research Institute. And they define the file format for this kind of geolocation data. So the geolocation data that you download has two files, the shape file, which ends in SHP, and the data file, which ends in dbf. So the shape file contains the actual location data, whereas the dbf file contains the attributes of this location data. So both of these files can be opened with the shape file library from PyShape, PYSHP. So what you do is you open both of the files, and then you feed it into the reader, and then you can extract the records. So as a first step, just as a sanity check, I wanted to see how many records there were in the shape file. So I just printed the length of the records, and we see there are 101 shape objects in this file, which roughly corresponds to the number of MRT stations you would expect in Singapore. So this is a good first step. Next, I wanted to see what kind of geolocation information we have in the shape file. So in S3 shape files, S3 shape files can contain a number of different types of geolocation data. So we could have a point in each record. We could have a polygon. We can also have 3D points, including the altitude. We can have 3D polygons. So yeah, I wanted to find out which one of these types was in our file. Yeah, so we printed the type of the first record, and we see it's type 1. So all we have is a point. So now we know we have 101 points in our file. Okay, and next I wanted to find out what kind of information we have about these points. So I printed the fields in the shape file object. And for S3 shape files, this always contains something called a deletion flag, and this is something to do with the way the records are deleted from the file. So yeah, we'll just ignore that. Instead, we'll look at the other item in the list. So the first value in this list is something called STN NAM. And the thing about S3 shape files is that the name of the attribute can only be a maximum of 10 characters long. So most of the time it's kind of abbreviated to something shorter. And looking at this, you can kind of guess STN NAM kind of sounds like station name, right? We're looking at train station data after all. Okay, the next value, C, just tells you what kind of field this is. So it could be a character, a number, or a date. Yeah, so in this case, C stands for character. The third value, 250, just states the maximum size of the field. So in this case, we know now that we have a character string that is at most 250 characters long. And finally, the fourth value has something to do with the number of decimal places that are allowed in the field. But this is not a number, so it's not applicable. So we just ignore that as well. Okay, so now that we know a little bit about the fields, right? We can iterate through the records to see what kind of information we have. So in this example, I'm just printing the data about the location data. And then the second thing is the location data itself. So iterate through the first three records. And we see that we get stuff like Crungy, One North, and Galang. So this kind of confirms that STNNAM stood for station name. And then when you look at the location data, right? For those of you who are familiar with the location of Singapore in latitude-longitude coordinates, this looks a little strange because the center of Singapore is actually 1.35 degrees north and 103.8 degrees east. So what are these like crazy numbers in the tens of thousands? So it turns out that the Earth is not exactly flat like one you mentioned yesterday. I mean, the Earth is not a sphere, it's not a perfect sphere. It actually looks like crazy bumps and dips on the surface and everything. And combine this with the fact that the tectonic plates are always in motion. This makes global positioning systems that were designed for the entire globe are actually pretty inaccurate for smaller regions. And you don't really get much smaller in Singapore, right? So it's actually quite bad for Singapore. So what Singapore did was they came up with their own positioning system called SVY-21. Yeah, and SVY-21, okay. So if you're dealing with latitude-longitude coordinates, right? If you travel all the way from the west to the east, you're only traversing like 0.5 of a degree, which is like quite sad. So if you move for a few hundred meters, you're dealing with like fractions of a degree. So in SVY-21, you're actually moving a whole 50,000 units. So when you move a little bit across the island from Tuas Achuang Bay, you're dealing with like more reasonable units, like hundreds and thousands rather than like 0.0001. Okay, so this presents a problem, right? Because a lot of the geo packages out there accept latitude-longitude coordinates only. So in order to work with these packages, we need to first convert the SVY-21 coordinates into lat-long values. Fortunately, someone has written a SVY-21 package that does exactly this. So we just import the SVY-21 package and we use the compute-lat-long method. And yeah, we've converted our SVY-21 format data into lat-long values. So this looks a lot more like the lat-long values that we're familiar with for Singapore. Okay, so now that we've converted our geolocation data to lat-long values, we can finally start to make some maps. Hey, so for my first attempt, I used this toolkit called Basemap. And for those of you who are familiar with Mapplotlib, Basemap is actually a toolkit that's contained within the Mapplotlib, like a whole collection of libraries. So I'm not going to go into too much detail here, except to say that Basemap being a Mapplotlib toolkit gives you very fine control over where you can plot. So you can control the lines that you plot, like the fill color and so on. So in this example here, all I'm doing is I'm plotting the outline of Singapore and I'm filling it with a gray background. So yeah, we've got Singapore. You can also add Mapplotlib-style markers. So here I'm adding a blue circular marker of size 12 at this coordinate. And it shows up kind of in the south of the island there. And you can also add text. So here I'm just labeling the points as Orchard MRT. So yeah, it's great. We have a map. And this is all well and good until you see what our JavaScript friends can do. So JavaScript has this library called Leaflet. And Leaflet is amazing. So this is what you can do in Leaflet. You can create these very rich interactive maps where you can drag, you can zoom, you can create heat maps like this. You can add pop-ups to markers. You can add HTML and CSS elements into these markers. It's really, really amazing. So suddenly our Basemap maps, which are static and non-interactive, are really dissatisfying. But thankfully, there's one rule of the internet. If it exists, there's a Python package for it. And that Python package is Folium. So Folium to the rescue, you import Folium and in one line, you can map Singapore immediately. So compare that to the 13 lines it took in Basemap to create that gray blob. In one line or in Folium, you can create this rich interactive map of Singapore. Yeah, so what I'm doing now is I'm going to iterate through the MRT records and add a marker for each record. So the first step was to compute the lat-long values from the SVY-21 values. And then I use the regular polygon marker method in Folium to add one marker for each MRT station. So I'm going to add a, I'm going to fill each marker with a dark gray color. I'm going to add four sides to each marker and I'm going to make it a radius five. So this is where you get a dark gray diamond at each MRT location in Singapore. Furthermore, I'm going to use the station name data that we extracted from the DBF file and add it as a pop-up to the map. So yeah, now when you click on markers, you get the station name. Yeah, so now that we've plotted the MRT stations in Singapore, the next step is to compute the average price per square foot of each MRT station. And furthermore, I'm going to color code these according to price so that red stations will be more expensive and blue stations will be less expensive. So it's kind of like a heat map going from cool colors to it's the warmer colors. Okay, so to compute the average price per square foot around each MRT station, I got transaction data from URA which is the Urban Redevelopment Authority and HDB for the HDB stuff. So URA takes care of the private transactions and HDB maintains a separate collection of their own transactions. Okay, so when we look at the transaction data, we see we run into a problem. We have the address information but URA and HDB don't actually provide the coordinates of each address. So how do we extract the coordinates from this address information? So thankfully, there's an API, OneMap, which provides an address search endpoint so you can submit the address and it will return you coordinates. So in this case, I submitted, you know, 71 IRRAC Crescent and this endpoint returns me the coordinates in SVI 21 format, but we've already seen how we can convert that to that one. Okay, so great. We've converted both our address information and our MRT information into that long coordinates. So now we can use this library called Geopie to compute the distances between each address and each MRT station. For this, I use a method called Great Circle which is a method to compute the distances between two points on the sphere. So some people have asked me like, why can't you use Euclidean distance to compute the distance since Singapore is really small and the earth is essentially flat, but in addition to not assuming that the earth is flat, Great Circle also is very convenient for converting these distances into meters or if you're not a fan of a metric, you can use yards. But yeah, but this is great because I wanted to focus on units that were close to the MRT station. So I basically discarded all the transactions that took place more than a kilometer away from the MRTs. Yeah, and that's just the code. Yeah, so what I'm gonna do now is I'm going to iterate through the MRTs again and I'm gonna skip ahead a few steps because I wanted to focus on the mapping part of this, right? So what I've done for each MRT station is I've pre-computed the average price per square foot for all the transactions in like the past three months, one kilometer away from the MRT. In addition, I've normalized these average PPSS and what this means is that I've basically converted the price per square foot to a zero to one scale. Yeah, and I'll elaborate on why I did this in a while. Okay, so some MRT stations are not going to have any information associated with them. For example, like Changi Airport MRT has no units like within a kilometer of the MRT station. So for those, I just set the value to be zero and then I color coded those markers as white. For the MRT stations with information, I used the weight values, the zero to one scale to calculate a hue. So if I submit this zero to one value to the HSV to RGB method in the Colossus Library, what this does is it converts the zero to one value into basically an RGB code. So yeah, it's basically like a collection of three numbers, each number corresponding to the red, the green and the blue of the color. And then I converted this into a hex color code, which is what the Folium Library accepts. Yeah, so basically you get a map like this. And the last step was to add the actual price per square foot to the pop-up message itself. So now when you click on the map, each marker will tell you the station name as well as the price per square foot associated with it. So yeah, this is not like, okay, so this is basically like a radial rainbow, right? That starts from the central business district in the south here. You see for like those stations in the south, they're really expensive. They're like $6 per square foot roughly. And then basically it's cheaper and cheaper as you radiate out towards the fringes of Singapore where it's more like $2 per square foot. And this is not exactly super surprising. What was surprising to me though was how much the prices dropped off from the CBD. So you know, if you move from like the downtown station, just one or two MRT stations out, the price already drops pretty drastically to like $4 per square foot. And what this means is if you care about living close to the CBD, right? The optimal thing to do is to move like one or two stations down, one or two stations away from the CBD because the moment you exit the CBD, the prices drop like super drastically and then they don't really drop very significantly until you reach like the extremes of the island, right? Yeah, so if you zoom in to this region, you can begin to see how you can make like the locally optimal choice. So for example, if Dyeong Baru has like an average of like $4 per square foot, and say you find a listing that is like a kilometer away from, within a kilometer of Dyeong Baru, and you see it has like a price of $3 per square foot, then you kind of like can get a sense of like, maybe you've stumbled across a deal. Yeah, so Martin was joking that, you know, there's like tons of machine learning talks at PyCon this year, so I couldn't help, but mention a bit of machine learning as well. At 99, we computed a model that basically like, finds the features that are most important in determining the value of a property. So besides the proximity to the CBD, the next most important location related feature is the proximity to the MRT station itself, the nearest MRT. So the most satisfying part of this site project for me was that I managed to find an MRT, a listing that was like one minute away from the MRT, which kind of implies that, you know, it should be more expensive than average, right? But it was actually cheaper than the average according to this map. So I'm happy to report that I was actually able to make a data-driven decision about where to live in Singapore. Yeah, thanks to OpenData and Folium. Yeah, so I actually want to show you a bit of the map and some of the code in a Jupyter notebook after this, but before I exit the PowerPoint, I very quickly wanted to thank Martin and Fiona because they basically encouraged me to give this talk despite some hesitance in the beginning. Yeah, and you know, Martin is like the most underappreciated hero advocate of like women coders in Singapore. So thank you, Martin. Yeah, I also wanted to thank the Pileadies community because they've been super supportive as well. Yeah, and this is my GitHub page. Yeah, my GitHub name is Miri66. So like the map and the Python notebook containing some of the code are on this page. Yeah, so now I'm gonna switch over to the browser. Sorry, I don't like switching between PowerPoint and browser because I find that pretty disruptive. But anyway, this is the map. Yeah, so like I said, Folium maps are pretty awesome. You can drag, you can zoom. If there's any MRT you want to check out, let me know now or not, the one. Okay, so actually, I found this quite interesting. There's actually some pockets of value here and there. For example, Caldercourt, right? You can see it's blue, whereas everything else around it is kind of a green or cyan. So I don't know why it's cheaper there, but it's like $2.53 versus everything around it's like $3. So that's pretty cool. You can start to find good deals around Singapore. Yeah, this is quite sad. Like the lawyer, Tanjong Bhagra, like $6. Yeah, so that's it for the map. I wanted to also show you a bit of the code. So this is running on our Jupyter notebook. So this is basically exactly what I presented just now, iterating through the records, blah, blah, blah. It only, it's quite exciting when you get to the Folium part because it's more visual, right? So this is the part where I first plotted the map of Singapore. And then you can change the coordinates to like re-center your map. So I just re-centered it to NUS. If we add stuff like 41 or whatever, it should go to like, that was just a random number. I have no idea where we are now. Maybe we can zoom out. Nope, that's a bad idea. Okay, we're actually somewhere in Russia or something. Yeah, okay, nevermind. Yeah, so you can also set like the zoom, the default zoom that you want to start at. And another cool thing about Folium is that it includes, it pre-includes a few tile sets. So there's one called Stamen Toner. If you don't like roads and buildings in your map for some reason, you can, oh nevermind. That doesn't remove the maps, but that doesn't remove the roads, but it changes it to black and white. So there's a number of these pre-built tile sets that you can use in Folium. Yeah, so this is the part where I added the MRT stations. Yeah, so you can do a lot of these markers. You can like change the color. So just want to make it like blue. It takes a while to render because I'm just iterating through 101 MRTs. So yeah, you can really see it's blue. So let's like blow it up. Changing the radius. Yeah, so now you can see that blue, right? Because it's like huge now. Yeah, and you can also change the number of sides.  I want like pentagon. Yeah, and you know, just now I plotted the rent values as circles, right? For those of you who are like observant enough, I actually cheated those one circles. They were actually 30-sided polygons. Yeah, so they look kind of circular. Can't tell, right? I wonder if you blow it up. Can you see if they're like circles? I mean, the polygons. No, they're still really huge circles. Yeah, okay, so that's pretty much it for the Jupyter Notebook. I also wanted to like plug the package that I wrote. You know the model that I mentioned? You can actually do a lot more complicated things with open data in Singapore. So what I did was I took the URA data and the HDB data and I put it through this algorithm that's becoming very popular around the data science community, which is a XGBoost. Some of you are nodding, right? It's like super popular now. It's winning like every Kaggle competition. Yeah, so what I did was I wrote this package that makes it slightly easier to use XGBoost. So it takes care of some of the pre-processing steps for you. Basically, all you need to do is construct your pandas data frame, do some feature engineering, and then you dump it into this package and it will generate all the results for you automatically. So it does all the things like one hot encoding, like dropping the features with no variation and so on automatically for you. Yeah, so basically what this model does is it predicts the values of each property and it also tells you the importance of each feature that you submit to it.  You can like sub-sample the transaction so you can like train your model faster so you can iterate faster. And you can also do like ensemble prediction so that you can split your data into like a bunch of sub-samples, train multiple models based on each of these samples, and then you know get like an average or like an average of the predictions and this has been shown to like yield more accurate results. Yeah, so that's basically it. I'll take questions with Danny. What can you look at a workability score? Oh, no, but that's a very, very interesting, yeah, that's a very interesting point. Let me see that asset away. We're actually calculating workability. Three data sets away. What three data sets? Working distance, road size, crime, I think. That's a three thing that you then put it in your model and then you can control very interesting results. Okay, yeah, yeah. In Singapore, I'm sure like crime is like just uniformly low everywhere. Maybe like Desker or something like that. Oh, that in Malaysia, right? Ooh, god. Let's not go there. But yeah, workability score is like, I think it's a very, very good idea. It's related to this whole MRT thing, right? Because basically no one can afford cars in Singapore so I think workability and transport are like super important and I do agree that there's so much data we can use to compute workability score because my transport also has like cycling paths and like walking paths and stuff. So yeah, definitely go into that. Question, no question? Yeah, no question. Thanks a lot, Supervisor. That way to say. Thanks, thanks a lot for me. Is there a way to split a player combined by features of the unit, for example, number of rooms, number of players? Oh, yeah, yeah, yeah. So, yeah, that's a really good point and I'm sure there's a lot of like breakdowns we can do, right? Besides the number of rooms which is a extremely important determinant of PPSF. Some people have also suggested that I do a breakdown of like HDB, condo because you're totally right. Like there's so many features that go into predicting the price. I really should do a breakdown of this at some point. Anyone else? Yeah, okay, thank you. Absolutely. It's a really wonderful talk to HDB. Thanks a lot. Covering for you. Playing with it for a while. It's really quite an interesting, exciting package to play with. There's obviously a fascinating project that you're working on. Is this something like hit up somebody with all the stuff you're doing that you're working on as well together? There are a lot of applications, actually. Yeah, so this part, this is on an iPython, yeah, this is on the GitHub, but the rest of it isn't. I could add it at some point. Yeah. I'm working on the print text side, so also you see there's a lot of APIs along with some of them. Obviously, a lot of other things that you can play with. Okay, yeah, cool, cool. Yeah, I will open it up. Thank you. Thanks. Okay, that's another question. I'll just give it a second. Yes, please make this available. Yeah, okay, we'll do it. Okay, thanks.


As promised in the previous video, today we will be comparing the reactivities of an allylic and a vinylic substrate towards an SN1 reaction and also the reactivities of benzylic versus phenylic substrate towards an SN1 reaction. Just to recall what allylic, vinylic, benzylic and phenylic actually mean, I have written these down here. A carbon attached to a doubly bonded carbon is called an allylic carbon. While the carbon which is actually the one that has the double bond is called a vinylic carbon. A carbon adjacent to the benzene ring is called a benzylic carbon while the carbon atom that belongs to the ring that is a part of the benzene ring is called a phenylic carbon. So let's just take one example and let's ask ourselves which of the following would react the fastest via SN1. Since we have already studied the mechanism in the previous videos, we can see that the leaving group is same in each case that is I- so we must focus on what carbocation would form in each case and compare the stability of various carbocations to realize which of the following would react the fastest via the SN1 mechanism. So let's form the carbocations. Why don't you try comparing the stability of each before we do it together? If I look carefully at these cations, I can see how the double bonded carbon has a positive charge in this case while the carbon next to the double bonded carbon has a positive charge in the other two cases. So the first one is a vinylic cation. Vinylic cation means a positive charge lying on a double bonded carbon and the other two are allylic cations. The positive charge resides on a carbon atom adjacent to the double bond. Digging a little deeper, I see there's a positive charge on a carbon atom that already forms a double bond. There's no possibility of resonance, right? This is what we have studied in the previous units while if the positive charge is alternate to a pi bond, there is a possibility of resonance. So let's draw the resonating structures. So this is the resonating structure for the second one and we can see how the positive charge has spread over all these atoms and has stabilized. Let's draw the resonating structure for the last one and here also I can see how the positive charge has spread out. So there's a possibility of resonance in both the second and the third one. What extra factor can you see? Well, there's an attached methyl group to the carbocation in the third case. Three alpha hydrogens, right? So there's a possibility of hyper conjugation as well. So I can see the last one is stabilized via resonance as well as hyper conjugation while the second one is just stabilized via resonance. What about the first one? It's highly unstable. Do you remember how a doubly bonded carbon was sp2 hybridized and it was a pretty good electronegative thing? So an electronegative atom carrying a positive charge, nah, not so stable, right? So it was highly unstable. So would it form? I don't think it would want to form. If you're not stable doing something, why would you do it? So the carbocation would not form in the first case while the carbocation would be the most stable in the third case. And therefore the rate of reaction is as follows. And I must mention that a vinylic substrate would not react via SN1 mechanism. This is highly, highly improbable for it to react via SN1 mechanism because it wouldn't form a carbocation. That would be highly unstable. So let's move on to comparing vinylic and benzylic cations. So what if I'm asked which of the following would not react via SN1 mechanism and which of the following would be the fastest to react via SN1 mechanism? In each case, Cl- would be the leaving group, right? So why don't we form the carbocations and compare their stabilities and figure out which one would actually not react via SN1? These are the carbocations. Hey, try comparing yourself first and then we'll do it together. If I look carefully at the first one, there's a benzene ring and there's a positive charge on the ring. There's a doubly bonded carbon that has a positive charge and it's highly unstable. Would it form? Nah, it wouldn't. So vinylic substrate wouldn't want to react via SN1 mechanism, right? Let's compare the other two. Shall we? In both the cases, I can see how a positive charge is alternate to a pi bond. There is a possibility of resonance in each case and through this resonance, the positive charge would spread out in the entire ring in each case, right? What else is there? What extra factor? Well, I can see the third one has three alpha hydrogens as well. So there's a possibility of hyperconjugation here. So the third one is stabilized via hyperconjugation and via resonance. While the second one is just stabilized via resonance and the first one is highly unstable. So what is the order of stability? The first one would not, not, not react via SN1 mechanism because this carbocation is highly, highly unstable. So the substrate that reacts the fastest via SN1 would be this one while if I talk about a phenyl substrate, it would not want to react via SN1. So what did we learn in this video? We learned how allylic and benzylic substrates are super happy to undergo substitution via SN1 mechanism while the vinylic and phenylic ones, not so much. The carbocations in those cases won't form at all as they would be highly, highly unstable. In the next video, we'll be talking about how and why rearrangements of carbocations take place in an SN1 mechanism.