at after that point, we should not allow abortions, except in very specific cases of things like the baby would be born with a chronic illness or the mother's life was in danger.  Well, then wait, then why could you kill a post-personhood fetus? It's because if we made the decision that the debilitative illness would be so much pain, life would be decided. And we're gonna make that choice for it? That's a different calculation.  Shouldn't you let it be born to make that decision for itself? It would be a similar situation. It's similar philosophical conversation to people that are in coma and have it articulated in their last participant what they decide. So we allow the loved ones to make the decisions in those points where they're unable to afford and why would it matter by the way? But it's an interesting, it's an interesting. Why would it matter, by the way, if the mother was going to be hurt or potentially killed by the birth? I mean, in one case, if you abort the child, that's a guaranteed murder, whereas with the mother, it's only potential. It seems like if you accept the personhood and reject the bodily autonomy argument, then you're morally obligated to permit, or I should say to deny the abortion no matter what.  That's not true. That's not true. We would never ask anyone to undergo a medical procedure that would likely lead to their death to save another person. Get back to the question real quick. I think the main difference that I would say if we're talking about mask mandates or vaccine mandates versus abortion and why they could or could not be compared, I would say that we can take a consequentialist approach to this, okay? A vaccine mandate would save lives. Making abortions freely available would save lives. It's been shown to work and lead to less abortions in developed countries. I am speaking from an American perspective. It would save lives in America. And vaccine mandates would save lives. That's not true. So I was- I hate to do this, but we've gone on this topic for quite a while. We want to get through these last couple of questions. This one, Nicky Erin says, should fetuses be covered under all protections as born children? If yes, it is child endangerment to impregnate a woman with a high chance of miscarriage. True. Oh, that's an interesting question. It's the exact same point we already went through. Could you repeat that please? I want to make sure. I may be at the same point. I just want to make sure I heard it correctly. They said, should fetuses be covered under all protections as born children? If yes, it is child endangerment to impregnate a woman with a high chance of miscarriage. It's the same point we had. It's not. It's different. I think it's interesting. OK, so motor, driving, you're more likely to get injured if you're riding on a motorcycle in an accident than an automobile. Does that mean your insurance rates are higher, but you're not endangering people more? So you can absolutely get in trouble with the state if reckless driving gets your kids injured, or even if it doesn't, you're just found out. That wasn't my point. What I mean is is that. Drive into a group of people if you wanted to. That's not the point that I was making.  Let's say you have a passenger who just willingly came along. They are more in danger riding the motorcycle than the automobile. Now, motorcyclists' insurance rates are higher because there's more risk, but you are not endangering people. Okay, we can take this example further if you want. If this is insistent that we wanna keep talking about this. So would Vaj and Stardust, would you both agree that we should charge people with child and turning instrument that are born into poverty situations where there's domestic abuse occurring in the house and they decide to have the child? Because they know that that child's likely to go through and in poversed existence where there's domestic abuse. So should we preemptively charge as soon as that child's born? I think you're missing my point. The mother and father for bringing a child into that situation. I think you're missing my point. My point is not about the single child, right? My point is what happens collectively to a community when you have a bunch of people who are forced to carry children to term that they didn't want and forced to carry children to term that would be born into situations where they are going to be exposed to violence and where their siblings are going to be exposed to violence. What is the net result of that? Let me explain. It's a separate argument to the argument with the question and the point that you were making here. That's a separate point. The answer to that point is, even if that's true, the question becomes, when do we consider that feed as a life or a person? Because we would never say it's okay to kill a person because killing that person would lead to a decrease in poverty, a decrease in domestic abuse, et cetera. So. But that's separate to the point this questioner's asking and the argument that you and Vosh are making that's, oh, if you know that you're likely to not be able to carry a- Please don't associate my view. I know that it's easy to do that, but- If you're agreeing with him, I don't think that it's unreasonable. He sees hazy after images of me every time he looks anywhere else, like when you stare at a light. Sure, if you're agreeing, if you two are agreeing- I don't even know what his position is. I don't know what the basis of his position is. I don't doubt that you don't. I honestly don't. I don't watch this content, dude. We do have to move to the next one. This one coming in from Daniel D. says, Rob, regarding your withdrawal from Afghanistan, do you believe that General Alden's plan could have reduced loss of life? Vence Vosh. It could have. I will be honest, when it comes to the actual specifics, it doesn't- I love the apologism for Biden. Again, I remind you that my biggest criticism becomes to the propagandist and all of the regimes and all of the parties, including Republicans and Democrats- Rob, do you have a response that isn't just repeating the same thing you've been saying over and over? Yeah, if you'd let me finish, maybe I'll get to it. So that's the biggest- Maybe, maybe, maybe. I was blaming you for this, Vosh. Those institutions- This is your fault, Vosh. Is this good? Is this good? Is this the answer you all want? No, Rob. We're fine with you- Would you like me to finish my answer? No, but you- So I remind you, that's the first organization that I'm going to play. But I would say this- Okay, so the answer is that the apologism for Biden says things like, well, we don't know what he could have done better. It is quite simple, and I don't know who exactly the generals would have been under Trump that had this plan, but it is quite simple that we shouldn't have let go of Bagram Air Base until we got our assets, including people and military assets and the prisoners that we didn't want to escape from gone. That's common sense, everyone understands. So the question was, what do you think General Alden's plan would have saved lives? I didn't hear a mention of the details of that plan and how it would have saved lives or not. So I asked when you were going to answer the question instead of ranting on the same thing you have been, and you said you would, and you didn't. So could you answer the question, please? Sure, why don't you explain to me exactly what General Alden's plan is? I don't know what his plan was. Right, which is exactly why my answer was- All right, we're gonna do it. Which is exactly what I said. I don't know what his specific details of his plan was, but it's quite simple that if we had a plan to leave Bagram Air Base, it would have been a disaster. I'm not making claims about his plan was. Okay. If you didn't know the details, that's fine. Okay, forgive me for interrupting. So sorry. I did promise to get you all out of here in about two hours. We're already 20 minutes past that, so I do want to apologize that. I do want to say all of our guests are linked to the description, folks. We do appreciate all of our guests. And so whether you're listening via YouTube or podcast, you can find all of our guests, Rob's, K's, Stardust, Sybil, Vosh, Sfinkter of Doom, SOD, and Emma's links in the description. We really do appreciate them and want to say thank you so much, everybody, for being with us. Thanks for your questions tonight, folks. We are excited about future panels like this, as this is our first one and a lively one to be sure. So thanks again to our guests. We are gonna let them go for the rest of the night, but one last, if you guys want to say thank you or goodbye, I want to say thank you so much for coming on, all of you. I love you. I love you, James. Thank you. Thank my first time with you, James. I had a lot of fun and it's been a hilarious drinking game to play for the few moments of good debate. Oh, you were drinking? The whole time. Wow. I'm really an anarchist that believes in vaccine mandates, just curious. Absolutely. Hey, listen, I believe in empowering communities.  I should point out that just because you're an anarchist doesn't mean you're against rules. You're just against rulers. That's what anarchy means. No, guys, no masters. You'll be subjected to the commune's vaccine. Yeah, so you can have communities can organize and require to be part of your community. It's all about her immunity.  I'm consistent. Great anarchy. Great. Hold them down. Yeah, if you have a vaccine, yeah, you should get a vaccine.  It's okay to be white. It's okay to be white. You don't need to merge your signal with anything. Okay? But I just wanted to... He calls himself the black anarchist that talked about these community marginalized communities. Look, I'm okay. All right. I just wanted to clarify, obviously I had a really great time. I have a lot of respect for everybody on the panel. And yeah, no hard feelings about anything, hopefully. So yeah.  Sorry. Yeah. This isn't my first time with James. My last time was with economics. Some of these subjects were kind of out of my wheelhouse, but I'm glad you guys had me. I hope I contributed to things productively and was not too boring. I'd be happy to be a part of the other one. Absolutely. I second that. I second that. Thanks Emma as well. Thanks for having me. I love you all very much. Take care. Thanks for having me. I had a lot of fun. Appreciate it. Thanks everybody. Have a great rest of your night and thanks everybody for watching and hanging out with us tonight. We will see you next time.


Hi everyone Hi Wayne Okay my name is Konstantin and I'm gonna talk to you about real-time rec But first I want to thank engineered They're sponsoring my trip here and Yeah, really grateful for that. Otherwise, I couldn't be here and talk to you guys Yeah a bit about me I'm on github and on Twitter and I have a blog like probably most of you have and Back in Germany where I'm originally from I'm a student at the Hasselblattner Institute of IT Systems Engineering Which is in Potsdam close to Berlin in case you ever want to visit I'd be happy to show you around In my free time, I'm also maintaining Sinatra. I know if you've heard of it. It's Yeah, quite popular I'm also maintaining a fork or a clone called almost Sinatra, which is probably going to be more popular than Sinatra really soon because it's only eight lines and I recently also know I'm also working on a book about Sinatra, which is supposed to land next month If I'll continue writing in time So if you want to learn more about Sinatra, please get that book. It's at O'Reilly Sinatra up and running and I'm also a member of the rec core team I'm going to talk a bit more about what REC is in case you're not fully aware of that and I'm currently doing an internship in Portland also sponsored by Engine Yard working full-time on Robinius Alternative Ruby implementation in case you haven't heard of that Okay, this talk is about breaking the laws of REC but Why do we want to break the laws of REC? So back in the day the web work like this you put in an URL You request a page you get a response back from the site and your fancy web browser displays it and But that's pretty boring. So along came Ajax doing all those fancy Requests you don't really notice getting data while you're on the page and so on but Ajax is really the client requesting data from the server but in some cases you want just to have the server push out data to the client and people came up with something something called and Comet which basically is Ajax, but this polling to the server to get Messages from the server, but this isn't what I mean when I talk about it real time real time is Streaming from the server to the client and pushing data at any time you feel like that the basic idea is to decide what to send while streaming not upfront and This is useful for Streaming APIs as Twitter does that. I know if you've written a Twitter client. You probably come across that Or server-send events. I go into detail about that in a bit Or you've probably all heard about web sockets, which is basically this is basically the foundations for Implementing web sockets over rec But I'll start with the demo You've all probably seen this prompt before can calculate stuff or write out code so this is basically a clone of IRB and the sweet thing is It's running in my web browser and It's also non-blocking so Yes, and which is really fancy is it's sending the data to the server The the string to evaluate and then the server just that doesn't eval and returns the results to the client Using server-send events, which is insecure, but well for a demo. It's fine, but the real fun thing is when I fire up a Second browser. This could be another on another machine, right? so Let me just do that and when I run some code here It's actually sent out to both Both clients or all clients a few if you'd connect to that port. I have a firewall on so don't even try and Yeah, this is basically what I'm talking about when I'm talking about real-time web And I'm gonna show you how to do this But first let's make sure you all we're all on the same page You all understand how rack is working internally rack is basically Ruby to HTTP to Ruby bridge. So it's mainly a specification how to translate HTTP requests into Ruby objects and then Translate that back into HTTP responses and it's also a Middleware API and it's the foundation for basically every Ruby web framework a library out there Including rails in our draw and so on And the basic structure is this you have the HTTP client, which is normally your browser could be anything else Which is talking to a thing we call the rack handler, which is basically your web server like Then a mongrel or passenger or whatever and Descender is the one doing the translation from and to HTTP and then using the rack protocol to talk to any middleware and then the so-called endpoint so what you usually do when you're writing your rails application or your Sinatra application is you define the endpoint and then Sinatra and rails set up some middleware in front of that for all the routing and filtering and whatnot and The handlers talking to that so a basic rack application if you're not using any framework looks like this It's it's it can be a proc that takes An ant hash, which is basically a hash of the past HTTP headers and a bit more of environment information And then it has to return an array with three elements The first one is the status code the second one is a hash of the of the headers And the last one is an object that loops through strings that will be sent to the client So but it doesn't have to be a proc it can actually be any object that Response to call and text and and hash or you know just this line I'm going to use a bit of Sinatra during my talk just to cut down the boilerplate But Sinatra really embraces rec and you could just return the rec response in here But so you can see what I'm talking about. I will cut out some of the rec code by that and What the handler is doing now is? Something like this is actually parsing the HTTP response into an ant hash and then it calls call on that and After that it sends out the headers and then calls each on the body object and after that closes the connection this is Basically the code you'll find in about any rec handler out there and Yeah Before I talk about streaming, let's take a look at middleware This is a stupid middleware that will take the body of Your rec endpoint your ads application or whatever and turn everything into uppercase. I mean who doesn't want that, right? So What it does it actually stores the endpoint or the next middleware in the middleware chain because you could use more than one middleware Obviously and then when it gets a call call it will send that call to to the application of course it could Modify the end-fash or keep a list of applications or whatever to do other nice things but that really doesn't matter for streaming and Then it calls each on the body object and Yeah, uppercase is everything and returns that I'm not using map in here because that would violate the rec specification since It's not guaranteed that the body object responds to map Okay, and this is how you would set that up.  It's usually there already or you could use some rec or some rails interfaces At a middleware. Yeah, and what the server is doing is instead of this line, which we had in the in the Rec handler code. I showed you It's doing this. It's calling new on the middleware passing the end point to it and then calls Call on that one Okay If you have any questions about that, please ask now I want to make sure we're all on the same page. So I didn't I gave this target Scottish RubyConf and people said I lost them like call for in Okay So let's talk about streaming as I said already the body object has to respond to each and Apart from that it can be basically anything so we can use that to implement some streaming This simple application would Send the Send the current time 20 times and always waits one second in between sending that out Which you could see if you would yeah, use the browser or just use curl. That's coming in bit for bit with one second delays and I'm using Sinatra again fancy, right Um, okay, but let's do something more fancy with that Let's do a messaging service where you have a list of clients like in my IRB example and can send data out to them So this is the interface we would like to have would like to keep a list of subscribers and then for everyone opening a connection to The slash there should be connection should be kept open and whenever you post To the application It should send a message to it so yeah adding the subscriber sending the messages down here and How would we implement that with just each? Well, we could do something like this The subscriber Obviously has to wait somehow for new messages to come in we do that with a sleep Which blocks the current threat and then or whenever we send out data We just wake up that threat which we stored when each was called and and send out the data This works fine on Some service, but there are some issues with that mainly it blocks the current threat with the sleep which Is an issue if you don't have a separate threat for every request which you do usually don't if you have a like server using a thread pool It will be able to handle as many requests simultaneously as the size of your thread pool is Which is typical for some service And it does not work well with some middle where imagine we take that application and change the uppercase middle We're in front of it that wouldn't work because the uppercase middle where it's calling each on that and And it waits until each finish until it tends to result on to the handle so we would never see any result and It does not work well or not at all with evented servers like thin or liar up or rainbows So yeah, let's see what we can do about that. My presentation is actually running in thin so it has to work somehow The solution is to do evented streaming where we don't have a threat per request But what is evented evented in this case means we have an event loop which is Usually run by event machine app is the only server not using event machine for this and this is basically a huge wild true loop which on on every loop through it checks for register callbacks or where callback is Triggered whenever whenever data comes in or a new connection comes in and it sends out data that is Registers for sending out and then it yeah You basically do all the stuff in the callbacks and register new callbacks and so on so That's web scale. Yeah And if you take this non evented application You probably every one of you sees that it will wait 10 seconds and only after that it will fire the request to Redis And if you would do that with event machine, it would look something like this It would start the event loop with event machine don't run and You can add a timer to trigger this callback After 10 seconds and you can add another callback to trigger this code whenever Redis responds and then the the event loop will take care of triggering that in the right moment so the Redis request is completely independent from the From the timer and this is what all the evented servers are basically using internally to handle a huge amount of connections and Yeah, so Putting that into our application We can use a nifty trick called the async callback where we basically Send out our rec response which is back here, you know the status code the headers and the body whenever the timer callback is fired and then we have to do some magic down there to tell the Rec handler that we are going to trigger the response later and it it should just go on with the event loop without caring about us for now and There are two ways to do that the first one is to use throw which was discussed shortly in the exception talk yesterday by and and this basically skips all the middle where it goes right through the to the rec handler which does a catch throw somewhere and catch async somewhere and then just Keeps on doing whatever it's doing or you could alternatively return a status code of minus one which is usually nicer because you don't use throw which will unroll the stack and Your middle where will handle that easier, but if you're running in development mode Rec will complain about this being an invalid status code So but what is the handler doing No, I'm sorry And there's also Nice library called async Sinatra Which adds for all the get post methods and so on an async version where you don't have to care about telling the direct handler to do this asynchronously and And yeah, of course instead of a timer. This could also be a request to Redis or Whatever Redis isn't that good an example because Redis is pretty fast But you could like query Google and there will take a couple of time and you want to go on to others stuff why why that's happening and The rec handler is something doing something like this Again, it's parsing the request then it sets up the asynchronous callback Where it basically does all this stuff it would usually be doing just after the request it sends out the the headers and then Uses each again to send out the data are you send data here because that's an event machine method and then it closes the connection and And then it stores the callback in the nFesh so the application can access it and and yeah After that it's just going on to call the the n but the the application and checks the status code if it's Minus one it will skip the callback for now Otherwise, it will just call the callback and send out the data and and the catch async is Here to allow the app leaving it with throw, which is basically behaving like an exception Okay, so far so good But that's postponing that's sending at some later point That's not really streaming. Of course We could do now do our each streaming where we're plugging blocking the threat again But then no other request would be dealt with and during that time since event machine is basically single threaded so There's an ifty trick to use event machine deferrable or Anything that behaves like event machine bit deferrable, which all the evented servers implement a deferrable is basically three states state machine Which has these states of not done yet Succeeded and failed and you can turn anything into a deferrable by just including event machine deferrable and then this adds a callback method that will be triggered whenever the Deferrable succeeds and an airbag method that will be triggered whenever the deferrable fails And then you have a succeed method and similarly you do have a fail method on that object and What it basically does is if it's a deferrable It will not close the connection until the deferrable fails or succeeds So this code you've seen previously setting up the callback It's changed to This code very check if it has a callback method. Actually the handler checks also if it has a airbag method, but I just wanted to save some code here and Then it closes the connection whenever one of those is triggered and if it's not a deferrable it just closes the connection right away and It's still using each up here So what we have to do is to get to send the data whenever we want like we would like to Okay, let's try to Make this example from I showed you previously make that event so it runs with him This is the changed API We are using a get now to have an asynchronous request and the the return value no longer matters Since we're going to send out the response later on Okay, but This is our new subscriber objects less code even so the trick here really is to when each is called Not to yield that block to that block but instead store that block for later on So whenever we call that the rack handler will actually call send data and of course we have to Include the deferrable to set up the callback methods And then all we have to do is whenever we want to send data is trigger that callback And if we want to close the connection later on Okay, yeah, I have to code again. Sorry We close the connection later on we just call succeed on that and Yeah, so my simple API is if you do a delete request close all the connections And That's basically the foundation for Evented streaming now you can use that to implement them service and events Which is a new standard for age from the W3c And It's kind of a one-way web socket So whenever you want to use web sockets, but you use it only to send data from the server to the client Just do yourself a favor and go for service and events instead. It's way easier on all the levels Yeah, super simple it they are the resumable if the connection is lost the web browser will automatically Open the connection again and knows where to continue in the stream and so on. It's no brainer and The client can be implemented just in JS for Browsers that don't support it with the full streaming support. This is possible including for Internet Explorer 7 I think and even if your client is using Internet Explorer 6 or Whatever else It's degrades to polling Which is really nice, which is not that possible if you want to run web sockets you have to include some flesh or something else to get that working and This is the client API you can use in JavaScript. It's available on Firefox Chrome I probably Safari. I don't know but you can use a JavaScript implementation as I said and You basically set up a callback that is triggered Whenever the server feels like sending data to you and The stream from the server looks like this first the HTTP headers So this is the real HTTP response you're getting from the server and then this just stays open and whenever the server feels like sending data, it's Sending the data or prefix with data to show that the Messages over it leaves a new line and you can use that to have multi-line messages Yes, and To ease resumability You can also add an ID for tracking and whenever the browser Opens a request again. It tells you which was the which the last idea was that it saw So you can actually track what happened there Do whatever you want with that and the ID doesn't have to be an integer. It can basically be any strings. It could be check some or anything Tweet ID whatever and This is the complete implementation. This is actually the code. I'm using for the example for the demo. I showed you Where it's basically the same as our subscriber with a more fancy Fancy send since you know, we have to split up the lines And and then send up an ID if we added an ID. I'm using the line number I'm checking line numbers, and I'm using that as an ID if the connection is lost to resume at the right point and Then alas desk web sockets, which is kind of an two-way event source This is the client interface You think probably oh, I've seen that. Yeah, that's correct. It's basically the same interface as the Event source interface with one exception. It also has a send method where you can send back methods messages Now the thing is this is way harder to To implement in rec. So I'm just showing you this little Jam there are a few others out there. This is I think the most popular It's called event machine web sockets where you start a separate server on the port and then you basically have exactly the same interface you have on the client also on the server and There are some issues with with web sockets So this is why I said if if you're just sending to the client go for go for service and events the client needs patching Not every browser supports that and you have to use flesh or something to to add that but also the server needs patching Rec does not support Sending data from the client later on while you're already streaming data back to the client So you basically have to Patrick there are projects out there doing this But it's not possible with vanilla rec and the proxies need patching h a proxy used to have huge issues Due to there being a protocol upgrade because basically it's not part of the HPT to be Specification that you can send data to the server after the service started sending data and as I said rec needs patching And the last thing I want to mention is there's this new speedy protocol Which wraps HTTP? S it's by Google and it also it opens one SSL connection for For a multiple request and that SSI connection can also be used to send back data to the client at any time Yeah, okay, that's it. Thanks My slides are up on guitar


Nice to see you, Frank. You're welcome. How are we going to get this done? There's the rest of us in the morning. Well, we'll have some fun. No. Here, drink. You're welcome. My theory is that everyone curses itself by morning, if it's a little bit more morning. No. I'm going to sit down. Sit back, okay? Yes. I'm going to sit around. I'm going to sit here. I'm going to sit here. I realized when I read the paper. There's no privacy. There's no privacy. There's no privacy. There's no privacy. There's no privacy. It's a... It's on. It's on. German. German. That's good. I'm always glad to see you like it. He's got shooting. Hang on. We've learned to say if you weren't with that camp boy. Yeah. You're right. We're having... My life was all... You cheated. You've seen us acting. I'm not happy about it. I've got halfway. The Prime Minister and I have just concluded a very useful and a wide-ranging discussion of some of the most crucial issues that are facing the people of Canada and the United States and indeed facing people everywhere. Prime Minister Trudeau briefed me on his recent discussions with leaders in Europe and Asia on his concerns for world peace and disarmament and improving the East-West dialogue. We fully share the concerns for peace which the Prime Minister has expressed. We appreciate his strong statements supporting the joint efforts of the Western Allies to negotiate meaningful arms reductions and to promote dialogue with other nations. And I thank you, Mr. Prime Minister, for coming here, sharing your ideas with us, and we wish you Godspeed and your efforts to help build a durable peace. Well, as you have just heard, the President supported what is being known as my peace initiative. But I think he did more than support it. I think he has been showing through his administration in the past months, at least, that as far as we are concerned on the NATO side, we want to change the trend line. We want to make it clear that not only the Alliance is strong, that it will defend itself, that it will not be intimidated, but that it is also pursuing peace. And if I were to tell you, for instance, that the President agrees that we shouldn't seek military superiority in NATO. We should seek a balance, that we do not think that a nuclear war can be won. That we think that the ideal would be to see an end to all nuclear arms that might come as news at least to some of the press in Canada, because we haven't been hearing that. But this is what the President said at the Diet in Japan. And this is what our foreign ministers have said just a few days ago in Brussels. That we respect each other's legitimate security interests. Whether this is news or not, I don't know. But this, at least in perception, is a complete change of a trend line which I saw when I embarked on my initiative several months ago as one which was going downwards rather than upwards, which was to use Carrington's phrase, which was characterized by a megaphone diplomacy. In Brussels last week, there was no megaphone diplomacy. There was a call for dialogue repeated two or three times in the message and in the communique. So I'm grateful not that I've said anything new this morning, but I'm grateful that I was able to hear from the President of the United States, the leader of the Alliance, that these are not just words, that these correspond to the intention of the Alliance, and that the other side can know, the Warsaw Pact can know, that we're not trying to be superior. We're not trying to not recognize their legitimate security interests. We just want them to realize we want to be at least equal in balance and that they should recognize ours. And I think this is a great step forward. I'll say nothing, though maybe I will say something, about the decision of NATO to send foreign ministers to Stockholm. This is really stating that the politicians are taking hold of the peace issue. It is no longer for the nuclear accountants, it is for the political leaders themselves. In French, yeah, in French in two words, I think you heard what I said, I actually mentioned the Brussels Declaration of Foreign Affairs, the words of the President as what they didn't want, that there was a nuclear war and they didn't believe that a nuclear war could be won. These are important statements, they are a confirmation of the intention of the politicians to seize the problems of peace and to pursue them. That's it. Thank you. Thank you. Mr Prime Minister, did you discuss your Pentagon Pipsqueak critics with the President? Yeah, third rate and third level Pipsqueaks. We say that I'm not allowed to talk about peace because somehow we haven't pulled our way in NATO. That is baloney. Equal time, Secretary Weinberger, equal time. They had Pipsqueak critics, sir, in the Pentagon. You know we've got rid of all of them.


NPTEL course on remote sensing and GIS for rural development. This is week 6 lecture 2. In week 6, we are looking at coordinate reference system, projections and other details of the map that are found on GIS data for download. However, some data may not be readily digitalized or readily input into the GIS system. For example, a map that we discussed in the last class. It is the only piece of evidence that we have to show how the land was 60 years ago, 70 years ago, 100 years ago, because at that time there were no satellites and photographs were not that efficient. So, there were extensive surveys done before the independence by the Britishers and after independence also by the survey departments. These have tremendous potential but also have a lot of errors because covering the entire country using surveys will have a lot of assumptions, limitations and challenges. Not all regions are accessible. So, that accessibility question is taken away by the satellites, drones and robotics. However, we also need to know what was the baseline scenario 70 years ago, 80 years ago to showcase the difference in land use, land cover for example, rural development, roads, transportation, electricity, all those things. So, there is very big information that we can collect from maps and that is what we will be looking at in today's lecture. How to tap into this tremendous information that is hidden in maps? When I say hidden, it is not that you have to find very carefully but you will easily miss it because there is so much information put in a map, a paper map. So, how are you going to extract it? How are you going to create new shapefiles, new data GIS data from your paper maps? So, let us look into that aspect in today's lecture. So, the first aspect we will look at is converting images to raster. So, we know that images already have a gridded fashion. For example, the image on the screen is a maps image. It is a digital scan. So, it was a paper and the paper image which is digitally processed or triangulated using survey data, it is scanned. It is a huge map. These are not small maps, it is a huge style and you have commercial scanners. So, where like for example, posters you see that big of a scanner. So, when this scanner scans the details, the output is an image which is it grins it and every grid very high resolution puts a color. So, now this color information is transported into your computer hard disk as an image but there is no geospatial location attached. This is as same as taking my own image like my photograph and putting it here like my passport photograph for example. So, both are same because my passport photograph does not have a geospatial location. It is just an image of my myself and it could be taken in Mumbai, Chennai, US, Malaysia, wherever it is. It is the same. It is same me and that is all is important. There is no need for a geospatial location. However, a map which depicts an area should have a geospatial location because that is what it is representing. My image is representing myself. My photograph represents me whereas an image of a map represents a geospatial location. So, I hope this is clear that there is a difference between scanning an image and georeferencing an image. Scanning is part of georeferencing but all scanned images are not georeferenced. Same thing all photographs are not georeferenced. If your camera for example if your phone has a provision of putting the lat longs that we discussed in the previous class latitude longitude along with the image taken then you have a georeference image. It is heavy because it has to attach a lot of files. Heavy as a not like a normal image it will get more information. So, we will have to be careful in triangulating the data for this. So, not all cameras are fitted with a geospatial location specific camera hardware but only some cameras have and it is expensive. These cameras are expensive than the normal cameras and you will find them easily in the market. The point is we would like to showcase more open source systems as much as possible and also showcase how you could collect information and data through GIS in a cost effective manner. So, in today's lecture we will be looking at an image of a map that has been created by a government agency and we will look at the specifics that one has to understand in detail before georeferencing or converting it into a raster. So, images can be made raster data with geospatial location. All rasters are images right. It has an image, it has pixels, grids but not all images become raster data. For the raster data you need to have geospatial location. So, I am stressing on the geospatial location otherwise I will show you an example in the QGS hands-on when we do this. I can open my own image in GIS it will open directly but it will not have a geospatial location. So, it will float but where can it float? It has to be anchored to a particular location and that is what we will look at in this lecture. So, moving on maps are still in the paper format as I said. The tremendous amount of data in maps if you want to do a long-term analysis. So, I have been to a lot of these survey officers when I was a student and a researcher, junior researcher after my PhD and it is tremendous amount of knowledge. It is not that easy to access it or you have to write a letter. As a student you can get a letter from your head of the department or a professor saying that why you need to access and then you can walk into these survey officers, give the letter and they will make it available. So, first is to examine the paper maps which means they will not have time to go through all the paper maps. Let us say you want Chennai-Mailapur map they will just say the number of the map but there are there could be multiple paper maps on Mailapur both temporally and depending on what is needed. Let us say road map there is a temple map because a lot of temples are there in Mailapur or a tank map so all these are different different maps. So, you will have to go through the library of the maps and then physically look into all the papers and stuff and sometimes they preserve it well but still it is still a paper. So, the old old maps sometimes do get degraded. So, you have to be very careful some people use gloves in foreign countries but because in those days they will not scan it. So, they made the maps but they could not scan it. Now you can scan it but the resolution the detail is not the same. So, you should be appreciative of these maps it has a lot of data. So, what people normally do is they take the information on a paper and then go back to the library it is like a library it is some books you have maps here and then you take the information and then you get back. So, they will give you an option to buy these maps from them in terms of copies and you cannot take some of the paper maps you cannot take out of the office. So, you will have to request for an image or a scan copies a rocks copy and they will make it available for a cost because there is a lot of infrastructure they have to maintain for it. So, there is always a cost. So, need to be digitized for extracting data. As I said, maps are just paper or an image is just an image with data but no geospatial data. A map is just a paper. So, it has slack lines and information of the geospatial but it cannot just sit into just if an image cannot sit into the GIS platform. So, there is a difference in using a GIS platform and a photo viewer in your desktop. So, you have multiple softwares for viewing image. So, what is the difference between looking at an image in a photo viewer and GIS? In GIS you give location whereas here there is no location needed. A photo of a bus is a photo of a bus it need not have a location. So, same thing analogy you use here. Tremendous amount of data can be extracted. As I said when they do these surveys every single they will have checklist. So, every single data is input into the system and we will have to be careful in looking at what data we need and extracting it. If you extract this all this image data into a digital image it will be too big. It will be too big to open it. Even opening this image in GIS is going to be hard. But what we are going to do is we are going to open it in GIS as a base layer and then from the base layer we are going to convert it into data that we require as a smaller amount. So, once you down scale the image. So, taking all of it for example this image has airports, post offices, hospitals, police stations, temples, houses, roads, railways etc. But we need only roads that are connecting rural to cities. So, for that part I do not need the other data that is present in the map. So, the whole objective of this week is going to be how to extract information from these maps. So, there are tremendous amount of data that can be extracted. As I said you will have to first make a checklist of what data you need because when they went to the field they have a large extensive checklist of collecting data. This is very similar to census data. Sensors you may only want the number of males and females in your district. But if you look at the census download and look there will be multiple columns. They will download a lot of data for every aspect and you will use what aspects you need. So, please understand that this data can be used but at a careful level. Usual digitization is expensive. As I said the scanners, the commercial scanners that you saw is very expensive to convert digitally and mark digitally on the maps and then putting it into a geospatial data. So, GIS has come in as an inexpensive tool. Still you need this image. So, the image has to be scanned. So, let us say this is a paper map and the paper map, this is a newer map.  But the maps that I am talking about 1980s, 1970s, 60s or even independence 40s, 50s, etc. You will see that these maps were still paper maps and these paper maps have to be scanned. So, the same printer that prints these maps can also scan. So, you scan them and that scanning is very expensive because every single inch centimeter of the map has to be scanned. So, it was very, very slow, very high in definition, it takes time. Not like your scanner like this. So, even the scanner if you see an A4 paper sheet you will place on the tray. So, on the tray you place an A4 sheet and then the scanning happens at every single centimeter throughout. So, that just continuous, data is continuous. It does not stop in the top and then go back in the front and then take data. A whole sheet is scanned. So, that is what we need to accomplish here using the digitization of these maps. So, let the scanner scan it. So, this is a kind of a printed map or you can also call as a scanned map from a digital device. However, it does not have geospatial location in the data. On the map there is, I will show you where you can find the geospatial location. On the map there is a lot of information. We are going to extract it. So, this is large scanners are needed as I explained and printers to make these maps but GIS provides a simple and open source solutions for all of this. So, instead of asking someone to do it, who you have to pay, we are going to make mappers through this lecture. So, I will be teaching you in this week how to digitize this map and then extract data which, honestly, I have seen a lot of consultants doing for a very, very high price for rural NGOs. For example, a map like this to extract and give data, I have seen NGOs charge 1 lakh, NGOs pay, I am sorry, for a consultant and that is literally half a day's work or even 2 hours' work, not more than that. So, why they give it is because the capacity is not built. The whole aim of this NPTM course is also to provide capacity to all students to use these softwares and also use it for a particular cost. Here the rural development is used. Maybe I will have a GIS and remote sensing for urban development. But for today, this lecture it is rural development. The principles are same but the applications are different and much more sub-objectives and principles will be discussed if the course goes as an urban development course. I think a lot of people are there for urban development but for rural it is very, very less. You do not see many courses for rural. So, I am very happy that a lot of students have registered for this course. It is a very important course and I have told in the previous lectures why rural development is important. So, as I said, GIS provides simple open source solutions and open source because we are using open source data plus open source software. If you use a proprietary software, then the price goes up and some people are very towards the proprietary software because it is easier to use. As again, if you have funds, you can use it. Otherwise, open source free QGIS is still good enough. It is as good as any other software. Let us take a look at an example closely. As I said, this is the image map that we are going to use for the next 16-week lecture. And throughout, you will see how we use this map in detail. But first, let us have a look at a closer look at this map. You could see that there are multiple, multiple information in this map. Are we able to digest everything? It is hard, difficult, but let us see what we can extract. So, you could see here that are published by the Director of Survey of India and then where it is Department of Science and Technology who funds it.  We will look at some of them and the legend. So, this is what is called the legend or what does each color or symbol represent? So, if you see here, you have this as boundaries and then here we have railway roads, train tracks, etc. on this part. And then you have airports, you have helipads and then you have post office. So, all these are marked on this map. How much of this is rural development? That depends on you because you are the one who is going to extract the data now. We will look into how we are going to extract data. The most importantly, there are sheet numbers. So, this is the sheet number for this image and you could see that in Bangalore, Karnataka. So, the major area is Karnataka and then within the Karnataka-Tumkur region and Tumkur-Bangalore urban. So, there is Bangalore rural and there is Bangalore urban. I am just going to show here Bangalore urban because it is on the border of rural and urban. So, you will see a couple of boundary effects here between rural and urban and you will see the tile number. So, this is the tile number D43R12 and then R11 is there, R15 is there. So, there is a schema they use to mark the numbers. Let us not get into that but every sheet, every tile has a number. So, D43 may be representing the entire belt and within that there are some pixels, subgrids which are labeled as R12, R1, R11, etc. It is also important to understand that these are gridded. So, the entire India is gridded into sheets and then within the sheets there are subgrids and here D43R12 is there and within that you have the lat long. So, you have the latitude which is here a 77 it starts and then goes on from 77 degrees 30 minutes, 32 seconds, 32 minutes and 30 seconds. So, 77. So, here is also 77 but it does not come as 77 they want to less write. So, you will see some writings are very half written why because they want to not over crowd the map with writing. So, that is why you will see a short form of writing here. For example, this is also 77 degrees 32 minutes and 30 seconds. This is 77 degrees 30 minutes 0 seconds but you do not see a 77 here again all throughout is 77 same here 13 degrees 15 minutes this is 13 degrees 12 minutes 30 seconds this is 10 minutes. So, how does it differ it differs by 2 minutes and 30 seconds. So, every 2 minutes 30 seconds there is a grid.  So, all this is there. So, it is minutes and seconds similar to your clock. So, 30 seconds is half a minute which is 60 seconds is 1 minute and that is how the grid is developed. So, that is one part and then let us look at source of the map the source of the map is given on the top and also on the bottom. You saw that the source of the map is survey of India which is a government agency established in 1787. So, as I said kind of British era also 1787 and then you come here at the bottom also you can see that published under the director of survey of India and then which state which location it was done. So, all this is there and the director Karnataka Geospatial Data Center survey of India Block Bangalore. So, all of this does relate back to your survey of India products survey of India. So, there was a price for this 77 rupees. So, some data is free open. So, 77 rupees is not big. As I said it is low cost the maps may be some of them are expensive also. So, you will look at some of the examples. So, then the first important thing is the source of the map that we have seen then we jump into the scale. As I said scale is how much is one unit representing the real life length. Let us say 1 centimeters how much is 1 centimeter on the map equivalent to on the ground. So, if you look at this closely here the scale is given as 1 is to 50,000. So, every unit that you measure on the map is equivalent to 50,000 times on the ground. So, if you look at this example let me zoom in you are zooming into this lake part right. So, okre is kind of a lake and if you say I put my centimeter scale is 1 centimeter 1 centimeter in length this part let me put my pointer so that you could see what I am showing. So, this part ok. So, this lake I am looking at and this distance. So, distance from the boundary to this boundary let us assume is 1 centimeter. So, what does that mean the length is 1 centimeter and that equals to 50,000 centimeters on the ground. So, now you know that 100 centimeters is a meter and then 1000 meters is a kilometer right. So, if you do the conversions you can convert it to kilometers or meters as per your need. So, that is how you scale up and then look at the distance how big this lake is it is a smaller lake compared to the other bigger lakes that we have down here. Yelanka lake is pretty big. Ok. So, moving on then you do the survey number as I explained earlier every sheet is marked by a survey number ok. So, there is an overall sheet number which is giving you the Karnataka region for example and within the Karnataka region you can choose a smaller region and then take an image. So, one of my PH the TA of this course and my PhD student Granath is working on this area. So, we had to buy these maps and that is why I am showing how to extract data from these maps. Again these were if you look at this region especially this region 50 years ago 60 years ago it was rural region now it has become urban right. So, there is a lot of shift from rural to urban we do not call that as a development we call it conversion. Development is rural development when we talk about rural development is how does development happen within the area for rural entities let us say farmers and ecosystem services, animals, plants, biodiversity etc converting rural to urban is not called development here it is conversion whereas if you increase the land under productivity that is rural development giving more access to people for water, energy, health in rural regions that is rural development not converting it totally into an urban thing which means the livelihood should still be rural. Here in an urban setting the livelihood is very different industries, offices etc. So, north arrow and direction so the map if you zoom out of this style you will see that it will be given a north arrow and here it is not this upwards which means I will draw it so not this like this. So, this is not normally you see the n simple and once you know the n simple the others can be taken up opposite to the north is south and the north you have to the right is your east and then you have your west. So, there is an understanding why West Indies was called West Indies. So, because from Europe they wanted to find India and when they say this was very very historic when when sailors were trying to find the root of India so that they wanted to do trade by gold, spices, silk etc. So, they found accidentally so for instead of Europe they had to come to east to find India that's why it's called east trading. So, from Europe you know Europe is here and India is here right so from Europe they had to go around Africa and then come to east, east is where India was there not west but accidentally that the sailors went to west side rather than east so they traveled this side and then they landed in a land which is very similar to India. So, the climate the people the food everything was very similar but they they thought something that oh I don't see the gold I don't see the silk I don't see the the the palaces that people said and then they labeled it West India or that's what West Indies is called okay. So, this is a small historic story people share we need to look at the the actual you know but but saying that the sailors found these regions from Europe and instead of going east and west you can see how a map this north arrow is very very important so if the sailor did not have this correct instead of east it was pointing west so the sailor went west side and then they found a different land right so here this can be avoided by just using a proper arrow marks and then the year when was this year done this is very important as I said you would need temporal analysis you can find the year on the map by zooming in as I said there's a lot of data in this map you will have to find where this map is having data and what data it has so somewhere here you have 2005 in the copyright you have 2011 2001 okay so that is what the year we will be using as the year of the map so there's also references on who to reference and then yeah we'll use the year when we open the full map we will find the year of this map also I've truncated the map so we won't find it here but oh yeah here first edition is 2011 so you have the 2011 as the map here then the projection and the coordinate system this we have discussed in the previous lecture and that has been given here right so the projections UTM it's a type of a projection there are multiple types so for this region the Indian subcontinent and Karnataka Central region UTM is pretty good so the projection as a light bulb experiment I showed is good for this part and then you have the datum where is the center for the coordinate system and that is given as WGS 84 WGS 84 again is a type of coordinate reference system CRS with a particular datum in the center so all this information is given and then you do have your lat long as I discussed earlier the lat to tube is the line that runs from top to down vertical and it is given as 13 degrees so here it is 13 degrees it starts and ends within the 13 degrees it doesn't go beyond and the same longitude is at 77 degrees so 77 degrees is where this tile is located and then there is some divisions legends and important places where here I have shown it when I zoomed in so these are the legends and important places where we need to collect data and stuff the cost is given as I said for a particular reason we had to buy this map and the price is 77 rupees again 77 rupees is still affordable by many for rural data extraction and the area covered here is pretty big okay so we're not looking at a small area we're looking at a large large area for analysis moving on I also would like to open this data portal where you can take the data there's multiple multiple things that you can download in terms of geospatial data let me quickly open it so if you click this link you will get into this webpage let me open it for you and we will see that you come to this webpage when you open this online survey map of India it's an online portal by the government of India department of science and technology minister of science and technology and it has a lot of products okay some are very very costly 6900 rupees for a particular geodata base high resolution etc whereas you can also get open source maps so you can say free maps and then it will format it's just searching for all the free free maps that are available so open street maps is there you can click to download the product it will open a page and then ask you to give the mobile number password etc for which you should register so always have this registered you can sign in and then do it but let's go back to the survey of India and you could see all the products here you have all products topographical maps which are the DEM and then digital elevation models you have the digital elevation models and administrative boundaries which is the administrative boundary of states districts sub districts villages talukas blocks etc DEM is a digital current model we'll be using this in this course georeference color raster and then digital geographical map village boundary database village boundaries the names the attributes a lot of people are asking questions in the forums i need this data this is very good why i'm promoting this is because the government of India and the boundaries are very very accurate because it is promoted by the government of India open series map it's a free PDF so this is where you can collect the open series map i'm just going to click it we need to take out the other digital products and other maps are also there so here when you go to open series map you will have to give the number of let me first go home and then look at the products so those who search for it on a different google search or other search database you'll come to the home portal and from here you go to products i have given the product link digital product link in the link itself also these are digital products open see yeah so open open series maps and free other maps you can click the free other maps and then you see that open series map osm is there which we'll be also using in this course okay but if you look at the boundary maps you can see that there are a lot of other boundary maps let me go through their maps so you come down there's digital vector database uh 68 000 okay so you read it right it's 68 000 um and the size is also there one is one million sheet and then you have the administrative databases these are free and i highly recommend you to use it so what is this this entire entire country taluka boundary taluka boundary with the names look at the boundaries this is the boundary that you have to use you cannot use different boundaries that other foreign nationals foreign agencies are showing it has to be accurate by law it has to be accurate so please use this map for any government of india reports and stuff and then you have the entire country district level entire country taluk level with headquarters some differences there between these two maps and then the district level map with headquarters so taluk district state and then the whole boundary there's a digital terrain elevation model the elevation boundaries we'll be using an open source but still people who want to use it can use this map and then georeference color raster thousand degrees uh digital geographical map you have different different maps that are produced by agencies through the through the database and then you have village boundaries as i said earlier and then open series open series map so this is where you click to buy it when you want to buy it you'll have to click download and give the sheet number or or enter the uh enter the first you have to log in and go and then you'll enter it stuff okay so uh this is where you could get um your data and information from um these maps okay so i'll also go through in the next lecture of georeferencing the map and extracting the data until then i'll conclude this lecture and see you in the next next show thank you