music and shake things up a little bit. So the title of my speech today is Increasing the Awesome. And as you saw by the first video, there's a lot of jobs out there where you're trying to decrease the suck.  You're just trying to keep up with all the issues at hand. And I'm in a lucky position, and several of us in this room are in a lucky position where we get to increase the awesome. And that's not just working at NASA. I think you should consider the work that you do with Wikis as a chance to increase the awesome. Think about the way that you have transformed your organization's opportunity to do different things with the same knowledge that was always there, right? And so the goal of my presentation is one, just to kind of give you another recap of what NASA has done with MediaWiki, our experience, but also two, to just kind of share some examples of how to celebrate the awesomeness that we've achieved and encourage you to take a look back at what you've done within your organization and use that as a motivation for your people. Don't just get stuck in a rut saying, hey, we need to use this Wiki, but go back and look at how cool it is and what it has afforded you that you didn't have before. And I'll just clarify that I'm speaking on behalf of the NASA JSC Wikis. Of course, we've also got Rich from Glenn and he's, we're now starting to figure out how to work across centers and for the whole agency, but I'm just gonna speak on behalf of JSC today. But before there was awesome, there was definitely a lot of trouble. There was a lot of negativity when we first started things. It didn't just happen magically. When I first started working at NASA, it was the same story as many of you have heard. It was very difficult to find the resources I needed to certify as an astronaut instructor. You'd go look in a folder and there'd be like three different versions of the same file and Rev Bravo had an older time stamp or a revision time stamp than Rev Alpha, but then there was also Rev Dave. And so you're like, well, which one do I trust? And many of you have heard the same story. And so I said, well, why don't we use something like Wikipedia at work? And so my immediate boss said, okay, sure, make it happen. Okay, well, but how do I do that? I have no idea. So it took quite a while before all the magic sauce came together. There were times where I was evaluating confluence. It was readily available, but it was for a different program. It wasn't for the ISS operations. And so I couldn't really figure out, well, can we really use this? Because it's not in our budget. We're not paying for it. And we don't officially have access to it. And so that kind of didn't work out. Eventually, James came along, unfortunately he's not here this morning, but he had some more experience than I did on setting up web servers with MySQL and PHP. He had actually built some applications at his previous job. And so he was familiar with the tools that were necessary to run MediaWiki. And just luckily, he also had set up a scheduling tool that we use for our classes at the Neutral Buoyancy Lab. And so he had access to a server. That was kind of critical. And so kind of on the side in the corner, he secretly installed MediaWiki and we started tinkering. But we kept it real quiet because we didn't want to get busted. We didn't want to get shut down. It kind of was a skunk works thing. And over the course of the year, we just started pumping information in there until finally people started catching wind of it and asking us, hey, I want access to that. And all of a sudden it became the cool thing. So eventually we got approval from management and so it took several years. But in 2011, that's where we started actually officially using a wiki in our workplace. And as I said, it took about a year to build up that critical mass. It was, it's really, as Mark was explaining why we're creating this kind of meta wiki or example wiki to showcase, it's really hard to explain to people the power of a wiki without showing them examples. And so we had to build those examples before people really got it. And so that took about a year. And we've covered this before. So this presentation is linked off of the schedule for today. So feel free to pull this up later or whatever and you can get access to all the videos. So I tried to link some of the prior presentations. So in Vienna a few years ago, we gave a background of the history of how we got started. We explained some of the examples of how we used semantic media wiki. So I won't go into too much detail on our beginnings today, but feel free to watch those later. One of the things that really helped kickstart our usage of a wiki was meeting minutes because that's what every organization does is they go have meetings and they talk about stuff, but we'll why not just feed that into a wiki and link those minutes to the topics about hardware and things that you're talking about and that way it's just all recorded there. And that definitely helped us organically grow more quickly. After about five years, we started having a lot of people come up to us and say, we want a wiki. And so Robo also, I mean, you've met a lot of the other NASA people here. Everybody wanted a wiki and we were just handing them out left and right. We were like, this is great. We're gonna have so many wikis, wikis are great. So by year five we had, we could actually say we had tens of wikis and this was a good thing. We had a lot of people using wikis. I decided to put together a kind of a retrospective of just kind of analyzing the usage and the growth of our wikis. And so linked off of here, I took a page that I had originally created on one of our internal wikis and I got approval to post it publicly. So now it's on a media wiki.org. So the page is five year wik, NASA wik anniversary.  So there's lots of little stats in here. We broke down our articles by category to kind of describe what kind of different things are we talking about here. Made lots of plots comparing how we started off with just a core set of admins making most of the contributions. And then over time we had other people besides the admins starting to fill in the contributions and then eventually we had people even outside of our organization helping to contribute to our collective knowledge. So it wasn't just the EVA group, it was people outside of the group contributing. So lots of different analytics here. I won't go into every one of them but I just had a lot of fun with R. There's a program called Exploratory which makes, I'm not sure if you guys are familiar with R but it's a programming language for statistics and it's got kind of a steep learning curve but there's a program called Exploratory which makes it really easy to just throw some data in there and then spit out some different plots and just kind of take her with it and find out what you really wanna get out of your data. And so a lot of these plots were created using Exploratory. This one here I kind of like because, so this is kind of like over time, the number of contributors. So you can see like of course when the Wiki started there was just James and me and then some other people started contributing and as time goes on you can see that it truly becomes a collaborative effort. Everybody's contributing their piece of the puzzle. I talk a little bit about watch analytics. We heard a little bit about that yesterday and I'll describe that a little bit more in a few minutes here. Let's see. There's also kind of something else to note is that we started off with just the EVA group and then Robo and Osso and some of these other disciplines joined and when you do it in that fashion, when you do it based on not just the Wiki for the whole company, but you start off with a single org and then you start adding other orgs kind of from the bottom up, you end up with kind of a step function every time a new group of people joins. So it might not just be a linear growth scale over time. You might end up seeing kind of step functions as they joined. For example, when Robo joined they had a SharePoint Wiki and James found a way to just wickify that. I think it used Pandoc or something like that. And that way they were bootstrapped with the same content they already had but now with the extra functionality that we have with the Media Wiki. We did a little comparison between our viewership and contributorship between comparing us to Wikipedia and of course the scales are way different but the plots were interestingly were very similar. The shape of the plots were very similar. So I'm not gonna dig into that but it might be interesting to take your analytics and compare to other Wikis and see how your growth is over time compared to others. So another thing we looked at, so there's kind of the 1% rule. So out of everybody you might consider that only a small percentage of your viewers are actually creating content. You might have a small portion of people editing content but then like maybe only 1% of your users are actually creating content, like creating new pages. So I don't know, does anybody actually go in and look at this kind of stuff and see across your organization are you actually measuring how many of your employees or how many of your workers are actually contributing versus just consuming the information? To me that's kind of one of the biggest differences between the public Wikipedia and a private enterprise Wiki is that Wikipedia is kind of voluntary.  I mean it would be nice but you're just not gonna get there. Whereas in a corporation you really do need every single person contributing as much as possible but you still end up with this sort of small percentage of people that actually contribute. So it is kind of a problem. This is, yes across all the wikis. And then we just did some kind of fun plots to just look at things differently. So here's a heat map showing the x-axis is the hour of the day and then these are the different groups, the different org codes. And so you can see that most everybody works between 7 a.m. and 4 p.m. because that's where the contributions are. I mean that's, I don't think that's very surprising. But then here's another one where we split it out by org. So each color is a different org. And then again we've got the hour of the day but now it's normalized to 100% of the contribution. So what we see here is a big red chunk, right? And so what is red? Well that's HSG, that's Greg's group. And they spend a lot of time over in Russia. So they're not in central time zone. And so they have a disproportionate amount of the contributions at 3 and 4 a.m. central time because they're over in Russia. I mean, I don't know, maybe this is silly but I think it's kind of cool to explore this and recognize, oh yeah, so that group actually is working over there. Their contributions might be masked because they're a smaller group but when you look at it in the early morning you better make sure you're not doing maintenance scripts that are gonna shut the wiki down. Because they're using it at that time. And then finally, I made an animation to demonstrate the growth of a wiki page. We had an EVA that was a very quick turnaround. We had to get out the door as quick as possible and go fix something. And so we just immediately threw a wiki page out there and just started adding pictures and doing research. And so this animation is a screenshot of every revision of the page and it just shows the growth of the page over time. So over here is kind of like the zoomed in and then on the left is the full page link because it had so much. It's just, you can't show it all on the screen. I just think this kind of shows the power of how simple MediaWiki makes it for everybody in a team to throw their information on one page and let's just solve this problem really quickly. And then I'm not gonna go through these but there are some testimonials at the bottom of the page. So maybe later on your own time, take a look at this page and I just asked all of our users for volunteers to please tell us your feedback whether it's positive or negative, whether you hate the wiki or love it. And we got a lot of really good feedback and it's just really interesting to see the different perspectives of each user because you might not anticipate how each person is using the wiki until you get that direct feedback from them. So I just thought it would be useful to share this. And when we shared it internally, of course initially after our five year anniversary, it was also good for other users to see what other people were saying about the wiki because you might get some people that are like, I don't really like using it but then once they saw these testimonials, you could see that it kind of had an impact on them and they thought, oh, okay, I guess I never thought about it that way. Let me go try what that person's doing. Just another plot showing that we're still continually growing, we're definitely still in the growth phase. I suppose eventually we might taper off but things are still going well even six and a half years later. And as I said, we kept adding more and more wikis but we didn't have a digital strategy and Mark's chuckling because this is kind of an inside joke but we didn't really plan this. This just started with the EVA wiki and then we were, as Oprah showed you earlier, we were just handing out wikis left and right but we didn't really think about the consequences of that. And all of a sudden we ended up with a whole bunch of wikis and at first we thought this is great but then we realized, okay, well, if a certain piece of hardware breaks and we need to fix it, well, the OSO group is gonna get that hardware ready by changing out the internal components and then the EVA group is gonna package that thing and take it outside and then the robotics team is gonna provide the robotic arm to get us to the work site and then once it's installed, you've got the Cronus team that is gonna command that box and check it out and make sure that it actually functions and so you've got multiple groups that have information about the same hardware but it's across different wikis and so now all of a sudden we've got the same problem. We've got multiple silos possibly with conflicting information. So, yeah, we kind of realized that we were kind of back in the same situation as before where we had a whole bunch of, where we created new silos and things were still disconnected and that's not what we wanted. And so now, something we're working on currently, we had actually hoped to have this finished by the conference and tell you about all of our success but it's actually a lot harder than you'd think. Eric has been doing the bulk of the work of trying to merge some of our wikis so we looked at, you know, which wikis can we merge based on access levels? We have one wiki that is open to basically any NASA employee and that's mostly just like printer information and how to use your time cards tool and that kind of stuff but then the robotics and EVA and OSO is like the internal maintenance so a lot of these wikis had the same permission scheme basically that you had to be a NASA employee and you had to be either a US person or a foreign person that has a need to know like if that's part of your job so like a Canadian flight controller or a Japanese crew member they need to know that information and we realized, well, if they've all got the same permission scheme let's go ahead and merge those together into just one ISS wiki based on the program. So we're working on that now. It's harder than it sounds. Basically you've got to look at a lot of the templates when Robo and OSO started their wikis they copied our templates for hardware pages and meetings and that kind of stuff but then they kind of made their own flair their own version of it and so when you merge back of course there's gonna be conflicts there and so you might think there's a scripted way to do it but really it comes down to manually comparing things and figuring out how to conform again and I don't know if we'll have time for a lightning talk for Eric to go into more detail on that but it's been a lot of work hopefully we'll get to report the success at some point. So I did want to share briefly like the fact that we're not we're trying to not just be consumers of this awesome technology. I mean definitely we're using media wiki and we're using a lot of extensions and we're enjoying the benefits of that but we're trying to also feedback into the community so one of the ways that we feedback the community is our participation. We're here we're trying to share our story we're trying to help other people out you know if you have something you're struggling with just chat with us and we'll see if we can help you figure it out. We've gone to at least one hackathon we went to a developer summit and kind of voiced our opinions there that was in January. We've also been growing our number of wiki savvy people so I mean it just started off with James and me and then Scott Ray he very early on got active in creating a ton of content for our wiki and then now as you can see we've got a lot of representation here from NASA and it's a little bit like the Fight Club Army you know you get this underground army of people that are savvy with the wiki but I mean everybody just continues to grow they level up in how they know how to use templates and properties and then eventually they start tinkering with extensions and so we've got several people now that are developing new extensions they're developing integrations we've got several people that never contributed to wikipedia before but now that they're comfortable using media wiki at work they say oh this is pretty easy I'm gonna go to wikipedia and so now they actually go contribute to articles in wikipedia we even have people that are now donors to the wikipedia foundation so I mean I think that's cool. Some of the software that we've created that everything we do we try to well everything except for our custom NASA specific stuff we try to share publicly so I'm not sure if hopefully James will get a chance to share more details about MEZA you've probably heard rumblings of this it's a set of scripts that we created that tries to simplify the installation and configuration of media wiki just so that you could keep that standardized and keep it as simple as possible we've created a lot of custom extensions for what we do at NASA but then there's several of them we've created that anybody could use we've modified existing extensions and we've also integrated things that were already out there at NASA I think it's important to not just try to cram everything into the wiki I think there's definitely cases where just leave some of these resources where they are because they do a good job of what they do but tie that into the wiki find a way to tie into their API and merge that information into a single search result within the wiki but let that other database keep that data as the source so I mentioned MEZA there is an animation here I won't show it here but it's kind of a fun way to show kind of the growth of the project it uses Gorse I think is the name of the tool it basically takes a GitHub repo and just shows the timeline progression of all of your commits which is kind of a fun way to go back and look at the growth of a project but just to very briefly describe what MEZA is it's a few years ago we had an opportunity where David MEZA one of the people at the chief knowledge officer office at JSC he gave us a couple servers for development and so in his honor we named the project MEZA and we later sort of forced the acronym back that we called it media wiki easy admin but the concept started off as a bunch of shell scripts to take a baseline Linux server sent to us and install Apache and MySQL now MariaDB and to install MediaWiki and a suite of extensions and to set it up as a farm if you want to have multiple wikis with just a couple commands on the command line since then James has converted it into a set of Ansible scripts so a little bit more on the technical side there but the kind of the big picture is that if you can get yourself, get your hands on a baseline sent to us virtual machine and you run just, I think it's like four or five commands and then wait ten minutes and then all of a sudden you've got a wiki farm so it's, I think it's pretty cool I think it's gonna help some people out to just get started and if you guys wanna contribute and participate in the project then by all means please as I said we've tried to integrate with some of the existing resources out there so we have imagery online there's a database that has thousands of pictures and so why try to upload those in the wiki why not just write a parser function that where you can list out the image IDs and then display those in line in the wiki page several other resources where we are able to pull information from another server into a wiki page and then we also started tinkering with, we call them bots I'm not sure if that's technically true but basically like scheduled scripts cron tasks that go off to a database and then compare that database data to the wiki and copy some of that information over and the reason we did that is that the wiki has a great built in notification system whereas a lot of these other resources don't have that so you kinda have to manually keep up and you might not get notified if something changes so instead since all of our workflow is in the wiki if we pull some of that information in nightly then you're gonna get notified automatically when that page changes and you can use the diff tool within media wiki that's already there to easily see what changed I'm not sure how many people here use Slack but you know it's just another chat program but we started using we started writing some different scripts that give us kind of analytics and real time reporting on our server performance in Slack and that way if we're just down the hall and we get a notification on our phone and it says oh hey by the way Eric just created 5,000 pages then you know he's probably working on merging the wikis and maybe you wanna go review that stuff but yeah it's just taking a look at how can you get notifications outside of the wiki whether it's using Slack or IRC or whatever just so real briefly something else I've been working on recently is we have a new class of astronauts coming in and so we have to train them on all the basic skills of EVA and so it's kind of hard to see this but basically every row is a skill that an astronaut has to learn how to perform when they're doing EVA and then every column is a class and so in our wiki we actually have a wiki page for every lesson plan for every class that we teach and then we also have a page for every skill that they need to learn how to perform and that way the new astronauts can go to those pages and watch videos and pick up tips on how to perform those skills properly and then this page is just a matrix that kind of shows us every time that skill has been practiced and so you can see these skills here they see it for the first time and then the second and the third time and then they've seen it more than four times then you think they probably haven't figured out and it's I don't know if this is real impressive or not but it's just you can use wiki pages in such weird ways where you just are able to report you know a whole lot of information just using a set of queries on a single page so kind of bluntly though but does anyone care? You know we do a lot of this stuff and we're pretty impressed with ourselves but kind of taking a step back and thinking about okay but does the rest of the community and the rest of the organization do they actually care? You know you heard about watch analytics we've done presentations on watch analytics before so there's a video there if you wanna watch it later I think watch analytics is a great extension it gives you a lot of insight into the activity of your users on whether they're actually keeping up with their watch list and reviewing changes it gives you a much simplified and more direct feedback on what pages have changed and then of course the diff view there's nothing new there but these links will take you right to the diff and so it's just a very easy set of directions for the users to go see what has changed and then it also suggests you know which pages need help being watched so kind of like your if you've got a bunch of pages that just nobody's paying attention to it'll say well I know that you're watching other hardware pages so maybe you should watch this hardware page and you know as Brian talked a little bit you know this tool gives you the opportunity to measure your success on actually reviewing the content as it keeps changing so he described this yesterday but just briefly to re-describe it so the outside here on the outer ring these orange dots represent people the blue dots represent wiki pages and then the lines indicate people watching wiki pages and so in here you know you would expect like this a lot of wiki pages are being watched by a lot of people but then out here you've got pages that are only being watched by a single person which is a problem right so like if I'm watching 100 pages and I go say a bunch of wrong things on those pages and nobody's watching them then they're not gonna correct it they're not gonna catch that incorrect information but then the plot actually also includes red lines to indicate a red indicates when someone has not yet gone and seen that latest revision and so this was our initial we had that same as Brian just said yesterday we had that same assumption we just said hey people are watching pages they're reviewing it and if it's wrong they'll go fix it and we just thought hey this is great and then we when James created this extension and we actually saw what was going on we realized oh wow they're not not only are people not reviewing pages but what the heck's going on here so once we started vocalizing this to our management for a brief moment they were really responsive and they said all right people go out there and do your reviews and as you can see our status got a lot better briefly this is current this is our I took this a few weeks ago so I think we're kind of back into a bad situation and so that's kind of why I put that slide of that page in there does anybody care initially people said oh this is great we need to take care of this but then it just quickly became forgotten and we try to raise this up with our management periodically but everybody's busy and yeah we'll get to that eventually but I mean how important is it to you and your organization to have your documentation current and correct I mean to me it just seems like that's so important but I don't know how to better convey this I don't know what we're doing wrong so approved revs your own described this yesterday we use this a little bit we try to only use it where absolutely necessary so like lesson plans and some of our console handbook pages that people think are super critical we put this extra layer of protection where it has to be approved before that revision will be viewed by everybody so we do make use of this but when you do that you add a layer of bureaucracy you add a bottleneck so I created this little script that every week right before our weekly group meeting it goes off and it checks all of the pages that use approved revs and it sees if there is a revision that has not yet been approved so basically that means that whoever owns that page needs to go review it and either approve it or fix it and then approve their revision and so it spits out the date of when that revision was made and then the link to the page and then if possible it lists who the owners are if it doesn't have the owners then it's just the management team and at first it was kind of nice because it spit out this list every week and I thought maybe we would address it but then I started noticing that at the bottom it was truncating in the middle of a page name and I thought oh man I got a bug in my script but then I just realized I was exceeding the limit of a string of whatever the data type was for passing all this data through in one value so our list was so long it was running out of memory for that data type and so I said to management like hey our list is so long it's breaking the script, come on I mean the oldest one is 2013 seems like we should address that but these things they just kind of linger and I don't know I'm not in a position of management I'm just in a position of providing data and trying to encourage the group to do better with their knowledge with these analytics but I don't know it's kind of weird being in a powerless position where you say hey we've got these problems we should fix them but then where does it go and so we've got a ton of other ways I mean there's so much data if you go into the tables of MediaWiki and you start analyzing all the different things back there along with watch analytics you get page scores Brian touched on this yesterday but we don't really use it in our group but it's there it's an opportunity to take another look at how well are we managing our information but we just don't use it you can even look at the timeliness of reviewers like how quickly do they go check their pending reviews but I don't think anybody really goes and reviews this I even started tinkering with some gamification stuff I made some plots to kind of show like compare that so every color is a different person and this is over time showing their contributions and so you don't necessarily want to say the most page revisions as the winner but you definitely want to see is everybody kind of continually contributing to the wiki but yeah I mean we have all this information out there but I don't know if it's not being used then maybe what's the point I don't know it's just kind of frustrating I guess is my so how can we motivate our group to make use of this information how can we motivate our group to make more use of the wiki instead of just letting it sit there and stagnate so we did one doccathon which is where our management set our rate no classes, no astronaut activities today we're just all going into a corner and we're going to work on the wiki we're going to split up into groups and each work on these pages and it was great I mean I think we got a lot done that day we worked on our lesson plans we got a lot of kind of the grunt work that nobody wants to finish we actually got that done that day but then that was just one day and that was two years ago I think so I don't know why aren't we doing more of these every week in our group meeting we do what we call a wiki-quickie it's just limited to five minutes does anybody have a question on how something works okay let me answer that real briefly and then we can talk more later or hey we have this new feature I just want everybody to know about this I think that kind of helps to get people's attention but and then I mentioned gamification several years ago in St. Louis I gave a presentation about how I was really excited about adding gamification to media wiki and it's easy to just throw some points in there like everybody gets a point for editing a page or everybody gets a point for thanking someone using the echo and thanks extension but then you can easily game the system and what are you really trying to reward you don't want to just reward page revisions you want to reward quality of content and that's a much harder thing to measure so I mean I tinkered with showing this is a plot showing over time different people and how many contributions they're making this is I tried looking at so this is for me this was like in 2013 these were the top 10 pages that I edited so like I guess my to-do list I edited 126 times I guess I used that quite a bit that year but then categories of pages that I edited the most and I don't know this was out there for everybody to be able to go look at their own information and see like well what did you spend most of your time in this year doing and I don't know maybe that's useful it's a little bit less on the point system it's a little bit more on kind of introspective kind of learning more about your workflow I tried putting together like a little bit of a story like you're a hero over the past 1605 days Darren has made 13,000 revisions to 3,967 pages let's estimate that if it just saved one person five minutes every time he did that and he has a time savings of 330 hours I don't know is that motivational to some people maybe but that's the trick is trying to figure out like some people are competitive and they want those points but that's probably only 10 or 15% of your group like what actually motivates everybody else some people are kind of social animals they want to pair up with other people and work on the same thing so I thought maybe if we look at the categories of pages that I edited and then find other people that edited a lot of the pages within that category maybe we can be wiki friends and edit the same pages I don't know I don't know that that would necessarily motivate me but it might motivate some other people and so I'm kind of just sharing all this just kind of I don't know if you have ideas later and you want to pitch some ideas and if you think your users would actually be motivated by any of these different aspects let me know because I've really hesitated to like actually release an extension and put it in use because I don't want to do it the wrong way I don't want to ruin the wiki and turn it into a system where you're just doing it to get points like I want it the whole point of the gamification is to get people to participate and make the wiki better so just a couple of final things here this is another plot from that I like looking at the time this is from the beginning of the wiki to I think this might be like maybe a year ago that I made this plot but it's you know here's just the number of contributions and every color is a person so you can see over time it generally goes up this is that same blood where it's like normalized to 100% so you know here we are at the beginning of the wiki and one person and then two people and three people were contributing and then here at the end you've got hundreds of people editing and I don't know I just think it's kind of cool to share these things with your users and show them show them the power of what they're working on so again here's the animation from before so if you want to take a look at that it's in the presentation and then I've added some links to some other videos we've got a lot of different presentations already out there on YouTube and I just tried to consolidate all of them and link them from this single presentation so that's it does anybody have any questions we've got microphones right turn it on when you're ready go ahead I was wondering what your sort of bottom line is because I'm seeing the activity measurement your graphs and the various things of what the activity is do you have any bottom level it appeared that your objective was an educational one from what I could tell in other words you're looking at skill development at one point and therefore using the wiki as a support tool for education as opposed to a factory call you know that you go to something to go and find what is the specific fact or specific information that you're looking for do you have any bottom line if you're going to upper level leadership and you're going to say what's the effect of all this on your education program is it improving the quality of the education or the efficiency of the training process do you have any bottom line feel or how you could get to that bottom line so I think I understand your question and so I'm going to I kind of skipped over this because I realized I was running long but this plot I showed you earlier the way it works is that here's an example of so an NBL run when we do a training run at the NBL we will perform several objectives and here is just one objective and that objective is to practice APFR APFR is a foot restraint it holds their feet steady while they do work and so you have to practice getting your feet into that tool and so you know for 30 minutes we might practice that skill and so there is a lesson plan wiki page that has all of these objectives and it has lots of semantic properties and then on another page there is a page that describes here are the steps to get into that foot restraint so this tells an astronaut this is what you need to do to succeed eventually we'll have videos and other resources but that's kind of like that tells you how to do it so there's definitely a lot of content in the wiki that is educational that is your standard kind of encyclopedic material but then the point of this was more the wiki can be more than just encyclopedic this is more of a reporting tool so what this did is it helped us figure out hey look at this skill we're only training it twice in their entire training flow is that enough? do we need to add it? like we might not have missed that had we not had this reporting tool whereas we can easily see so like this one oh man we train it pretty much every class we're good to go so it's sort of looking at you you're getting towards the human resource issue of people and skills and their training and the whole management of that but I'm thinking what's the bottom line of this it's obviously got to be something that's being done better than the alternate mechanisms I don't know if this helps answer but the way I saw this the bottom line was the previous groups of astronauts we've trained three or four years later we determined we're not what we expected in the skill level but we had no data to show that so the bottom line of creating this on the wiki was we actually now have data we can show our management and say we can analytically show how we're training and eventually increase that to to benefit it so we use the wiki in many cases but the bottom line for this specific skill I think that was kind of the bottom line of how do we get more proficient students and show that we are doing that with the data we have you're getting to holistic skill management well it identifies gaps in our training in our data set okay so you want to pass the mic back to her in your room I think you're next just briefly I think there's someone in a previous talk who said that you know wiki activity was integrated into performance reviews I don't remember I think it was Brian right oh yeah okay right right I don't know if that's an option for you guys it seems like I mean if I was a manager absolutely right but I'm not you know so I'm just this guy that you know trains astronauts and builds the wiki and I try to I try to provide this information to management and say here look what everybody is doing and I think to a certain extent they do look at it but I don't know I guess they're not they don't because there's a lot more to our job that's not necessarily tied to the wiki right like that's not everything we do so I don't know so my question builds on the first question so you're building or capability here and my question is can you tie what you've done on the wiki to a cost savings because if you if you are sending all these people out for training but now they can do it on the wiki you know there there must be a cost savings that management would pay attention to and if you could take that to them they could help you socialize market the wiki you know put it put other training on there to build OC definitely I I I think I agree with what you're saying I I don't know how you exactly formulate the the quantification of that though like well so for example this is what we're doing in our company I'm gonna put oil full essentials on our wiki okay so for each person that takes that class it cost let's say it cost about a thousand dollars okay but if I can have a class of twelve go through each quarter then I'm saving my company money and I think that's what management pays attention to is what you know what is that return and you do wonderful graphics and you know graphics tell the story you know that's the narrative so I think maybe I mean it's something to consider do you have a microphone for James because I think I know what he's gonna say so yeah I was just gonna comment one one area especially in training especially in this job where it can take three years to get somebody certified and they're basically they're non-value added in the beginning and then it's a slow ramp up as the percentage of work that they're doing they're just building building training time non-value added training time versus and then their percentage of contribution start to build and then they're certified let's say they're a hundred percent contributing one of the concepts that's been getting some traction is just in time training so it's especially with astronauts but it's also with flight controllers and other people is the idea that you get this class one year but you don't use that skill until you launch a year later and so you have to find out which pieces of training have it treated by then what they actually need to get refreshed on and then also especially from a flight controller training perspective so much of the old system was learning how to look things up learning which dusty bookshelf has the technical manual that you need and so when we we be examining and doing evals on these trainees sometimes we would say no you don't need to memorize that you just need to know how to find it but now as things are moving to the wiki how to find it is just typing in search box and so it's really interesting seeing how much of that entire training flow has been eliminated where okay you can look it up instantly sure the time savings I mean that yeah it's got you know how much of the hour and then versus looking things up the thing is the metrics were never tracking how long it was taking for people to find that dusty old bookshelf and even if you could sort of generalize you know and even yeah you could show them you know that value and I think that you could get more buy in James did you have a response to her or was it separate so we have kind of standard training flows different levels of certification is it's different for each group but for our group specifically it's about a year after you start you get you get your certification in in our instructor flow and would be really great if we could say by use of the wiki we have produced that down to six months because then we could go to our management and say you know number of new employees times six months times average pay this is what it's worth but unfortunately it's not that that straightforward and I have no data to back this up but my general feel is that that our training flows are not any shorter they may actually even be a little bit longer but I think that our certification level is actually a lot harder now I think because it's so much easier to find all of the information now when when instructors go in that are certifying people they have a wealth of information that they just didn't really have access to before and it's like now I expect you to know this too which you weren't really even required required to before just because our entire corpus of knowledge is so readily available we kind of expect new people to know more of it than we did before the other element of savings it's tangentially related to saving to training but when it came to updating documents what was interesting in my group is that using the old paper system in which you had the serial signature process so you have all these people review the latest revision to the handbook and they would all be collected into this one rev and then this technical expert had to sign off of it and this was the second signature and then and just goes up for signatures they had a system of tracking that so you could see it was taking on the order of a year or longer to get the next rev to the document published and so just by switching a wiki not only not having that linear signature process sped up publishing new information on a paragraph level but if somebody wasn't doing their reviews and wasn't doing their signatures the rest of the team was sort of moving over like water flowing around the bottleneck so my next question is do you have lean signal trained people in NASA that you could leverage because you know they're trained to help do these small measurements you know reduction of cycle times and that's what I'm hearing from you and actually they can help you with those metrics I mean we use them all the time in our company and you know you just a quick meeting you tell them what you want and then they come in another quick meeting and then they because they're trained and they do all the metrics and stuff and then you talk about them and then you normalize them but you get at least you get a baseline of where you are and if you you've had the wiki for six years six seven years so to me it's a success it's just fine-tuning it and getting you know sort of that monetary number that people are looking for yeah I would love to have anything monetary out of this


Now, the 6th of May will be Keir Starmer's first big electoral test as Labour leader. Commentators will be looking out for whether there are signs of the start of a recovery in Scotland, whether Labour can remain dominant in elections to the Welsh Assembly, and whether the party will go forwards or backwards across England's councils. However, perhaps because Starmer has spent so much energy relentlessly focusing on winning back MPs in the red wall, the loudest chatter within Labour circles has been about the by-election that will be held on that day in Hartlepool, where constituents will vote in a replacement for the former Labour MP, Mike Hill. Now, that chatter went into overdrive on Tuesday when Servaiton released the first constituency and only constituency-based polling in Hartlepool. We can get up the results here. So, Servaiton have projected that the Tories are winning 49% to Labour's 42%. They also projected the brand-new Northern Independence Party in Ferd, which is quite an achievement. They mainly exist on Twitter. They do have a great candidate, Falmer Walker, who's knocking on doors. Most people think of them at this point as kind of a Twitter phenomenon. So, to get 2% would be impressive if that is borne out in fact. They also asked about the favourability of the different leaders. As we've discussed, it's not looking great for Keir Starmer at the moment. So, Boris Johnson there, 49% favourable compared to 30% unfavourable. Keir Starmer, only 24% favourable compared to 38% unfavourable. And that's obviously the people in the constituency who are asked. They're not national favourability ratings. They're local favourability ratings. Now, the poll was commissioned by the Communication Workers Union and their General Secretary Dave Ward joins me now. Now, Dave, welcome to the show. Hi, Michael. I'm going to go through some of the policy questions which the poll asked, because I know this is close to the CWU's heart, the issues about broadband, etc. But first of all, I want to talk about the pushback you got for commissioning this poll from certain Labour MPs. This is what Brent Bradshaw, this is how he responded to the poll. So, he tweeted, hi, Dave Ward, how about helping on the Labour doorstep in Hartlepool and elsewhere instead of spending CWU members money on a dodgy looking unweighted poll with leading questions and using it to undermine Keir Starmer and Labour? That was followed up by this from Toby Perkins, Shadow Minister for Apprentices and Lifelong Learning. Toby Perkins wrote, instead of spending his members money running tiny polls to undermine Labour campaign, Dave Ward could more usefully get himself up to Hartlepool. A lot more than 42% of voters I've met are voting for Paul Williams. That's not really how the Labour doorstep works. You normally knock on people who are expecting to vote Labour. But Dave, I want to know, what do you say? How would you respond to Ben and Toby? Yeah, I mean, I think you saw our response in our tweets last night that sort of really sort of said they're irrelevant in this debate if that's their view. I think it's a very shallow response from those MPs. I guess it's the start of a coordinated response from some on the right in the party. And I think what it tells me is that they're still blind to the real reasons why they're not connecting with working class communities. And the fact that they would rather attack, not us in that sense. I mean, this was carried out by salvation, one of the most well respected polling companies in the country. I think it just demonstrates how off the pace they are. But to me, personally, I mean, they're on the relevance in the debate. I mean, the CWU, Michael, we commissioned the poll for a number of really important reasons where we thought it was right to make this intervention. Firstly, the CWU is a union where we've worked really hard to connect our industrial and political agendas. So it was important to us to test some policies that were around in the general election that Labour then supported. But these are policies that support our members' jobs. So as you said, things like free broadband, renationalization, roll amount. I have to say, I was quite taken aback by the scowl of the support for free broadband and the difference between how that was seen in this particular poll compared to what a lot of critics on the right of Labour were criticising Jeremy and John McDonnell for coming up with that policy, which we were part of developing. So I thought that was really interesting. And therefore, it helps us understand what we need to do to connect with our members' jobs and in supporting the industries that they work in. The second reason we do it is the CWU is a union that strongly believes that it's time for the trade union movement to harness our collective strength and to really change some of the imbalances of power and wealth that exist, that unions can make change happen on the ground. So it was a strong message from us that we wanted to reconnect with working class communities as a trade union movement. We wanted to understand what was happening. And then thirdly, the poll was clearly that we're unhappy with the direction that Labour are taking. We think that Labour are genuinely in danger of becoming an irrelevance. And my feeling for some time has been that there's no pathway to victory for the Labour Party without a radical platform that changes the inequalities, that challenges that balance of power and wealth in the world of work, in the economy and in wider society. And for me, the poll showed that without those policies, a Labour Party that spent the last year obsessing with telling everybody what Jeremy Corbyn, sorry, that they're not Jeremy Corbyn, rightly, it seems to us that people are saying, well, who are you then? And I think there's a message there for Labour that if it wants to win seats like Hartipal, it now needs to go back to setting out a bold agenda for the future. I want to get up some of those poll results. As you say, they were quite striking. So first of all, let's look at what people said in terms of their priority being investing more in public services or paying off the country's deficit. Now, 67% when asked that question chose investing in more public services. So they're a clear rejection of austerity, although to be fair, most parties are talking along those lines. Now they've been quite a turnaround there. There was also, as you mentioned, overwhelming support for Royal Mail nationalisation. And then the real headline result, the real surprising result for many people was on broadband as you say. So you can see here, 69% of people supported having a policy of providing free broadband internet to all UK homes and businesses by 2030.  It's nonsense. People don't actually want free broadband. The more I suppose subtle and sophisticated critique is to say that, yes, we all knew that people like nice things. The reason why Labour going into an election promising all of these things was a problem was because it sort of fed into a narrative that Labour just offer nice things and you can't really trust them to deliver them or they're offering so many nice things that will end up in loads of debt. So whilst the public might say, oh, I'd love free broadband if the Tories gave it to me, but if Labour give it to me, that would seem a bit suspicious there. What do you make of that critique? Well, first of all, it wasn't an unweighted poll. So it was done exactly in the way that salvation would do any poll. And we actually saw the unweighted results and then they're weighted at that point and they go through all the same processes that they do with any poll. So that did then come back towards a more narrow result, actually, in terms of whether they would be supporting Labour auditories. The critique, I think what's interesting about it is quite a relatively short period of time since that policy was ridiculed by a lot of people, including many people in the Labour Party. But actually, I think what's happened is that during the pandemic, I think people have realised the importance of broadband, the importance of that in tackling inequality, how it keeps people connected. And I think everybody recognises that unless you've got full fibre broadband connected across the whole of the country, that without that is a key part of your infrastructure to rebuild the economy. Because there's been lots of discussions about rebuilding the economy, I think people have just realised that these are essential things. More importantly, I think there's a willingness to recognise there that the state should intervene in funding and investing in free broadband. So for me, in some ways, it was a justification of the work that we done with John McDonald leading into the last general election. And I would say that John and his team were ahead of the game with this policy. Boris Johnson, I think, has recognised that a year on more than what Labour are. And so it's a wake-up call for Labour to get back to a radical policy agenda and to set that out as part of its election offering. I think that's a really important point, actually. If you ask most people, who do you think is more likely to now offer free broadband in the next general election? I think it probably would be Boris Johnson. I want to ask you, because we're talking about Hartley Paul, the other big takeaway from that poll was that in third place was the Northern Independence Party. I want to go to a statement they released in response to them being shown to be inferred. So they say, a political party that was set up by strangers over Twitter and WhatsApp five months ago is now polling higher than the Liberal Democrats and Brexit Party Reform UK. We've not even started our campaign. We've spent nothing on this election, yet we are already beating established Westminster parties. The Labour Party is a spent force in British politics. They are not a fit opposition to the Tories. Now, Dave, in a way, you seem to share some of their analysis. The Labour Party is becoming irrelevant. They're not providing proper opposition to the Tories. Are you in any way tempted by this new, you might call it a very online party, but they're polling inferred. What's your opinion on the Northern Independence Party? Well, I haven't really formed an opinion on them other than I did see the guy who's the leader. His name is Proudfoot. Was he on your show? I think I saw a clip. Yeah, Philip Proudfoot. He was on the show with Aaron. I thought he was pretty good, to be honest with you. I thought he was talking on a more sort of radical platform than what I hear from Labour. So again, you know, a reason for Labour to watch out. But look, you know, I mean, there's a general consensus, I think, building that people are moving towards more separatism and they seem to be supporting more political parties that are closely aligned to where they are, what their issues are. So I think there's lessons for Labour in that. For me, you know, the big takeaway in all of this is, and this is kind of where the CW is at the moment politically, I don't want us as a union to get engaged in a debate about disaffiliation. And because, you know, I don't really think that that takes us anywhere. But equally, I'm not prepared to, you know, see Labour use the union movement like a bank, where it can withdraw money at the time of elections and not represent the policies that we feel that we need in this country, that our members would support. So, you know, Labour should never get anything, something for nothing from the trade union movement. I'm a very strong supporter, Michael, of the trade union movement showing the way for Labour. And I think we're due to talk in a couple of weeks time about this new deal for working people and why I think it's important that the trade union movement, whether you're affiliated to the Labour Party or not, that we are willing to step up and take on some of these companies, take on whole sectors of the economy to tackle insecure employment, to tackle the imbalance of power in the world of work in the way that the economy shape. That's our job. We have to do that. And what this poll is about for me, it just, again, says to me that we are the people that can connect with working class communities. And that's what we're going to do. We're going to develop that. We're going to diversify the way that we spend our political money. I mean, the other interesting thing, which we haven't touched on, is my team have been looking at this during the course of the day. And it was really interesting to see how many CW members went back at people like Ben Bradshaw, almost saying to him, don't tell us how our union spends their political money, because they're putting it to good use. And I think that's a warning sign for Labour not to treat trade unions disrespectfully. And I have to say, it fills in the last year that they've got more distance from the trade union movement and from engaging with us. We're not fringe movement. We're a powerful movement. And I think we've got a really good chance of bridging, you know, a lot of the divides that exist in society and unite in working class people. So, you know, that's our strategy. That's what we're going to do. And whatever people like Ben Bradshaw say.


It is time for Q&A for the clarification. So for our protocol we use mostly well we try to do the load balancing based on the size of the transactions or the cost of processing the transactions. And it was suggested by address to extend it also to have a look at the state size to make sure that in every single shot we have also dissimilar size. We initially didn't think about it because the storage doesn't seem to be a bottleneck. On the other hand, the long run it may be because if you say that you will store something in the blockchain while you get to store it for potentially forever which is complicated. So we are currently also looking on on to that. Although it is not clear whether you want to calculate the storage as for instance for smart contracts is the size of the state of the smart contracts or the amount of the modification to the smart contract probably both. That's the question and an answer. I just wanted to take a note on the implementation of our protocol ring bft which is inside the resilient db I really suggest everyone to take a look at our resilient db platform and take a look and to see how we can implement actual blockchain means permission blockchain system or if the consensus charting protocol and other. I want to ask a question about ring bft. So basically, the theoretical advantage of linear communication amongst shards is kind of clear but is there a practical advantage and like the did you quantify this. Can you can you detail this a bit. When it comes to the consensus in multi short transaction when you scale up up to 15 regions, the previous work usually have all to all communication and in the practical system when you scale up to hundreds of multiple hundreds of nodes. The communication would always be a bottleneck, for example in previous work like a HL reference committee would be the bottleneck and. This pattern of all 12 communication would make the performance of the system to true put the latency, a little bit even worse, and usually when people talk about the communication and the latency it's all about round trip. But when it comes to the actual protocol when it comes to the practical system working the RTT or round trip time is not the only barrier for the ratings for the latency and the performance of the system. The amount of communication the amount of messages messages that are being processed for one instance of cross short consensus to happen is the is one of the main and most important overlooked teams that are missing in the previous work that we take a look into. And this whole ring architecture that we go over the ring the linear amount of communication, which makes performance increasing and scaling the performance by increasing the number of short company linear so. It's scaling up to more sharp with the linear but in the previous work, since there's a all 12 communication scaling would not be linear. So in BFT scale is much, much better than our other previous charting consensus for all. And to that, of course we have your two linear terms we have the linear terms due to the cluster sending, but that's the small thing because if your shots are not going to be too big that that is an optimization but we're really really talking about communication between chart and chart so one chart talks to one other chart, whereas indeed like I explained in a lot of protocols that really try to do it as fast as possible, if you have five shots in your transactions you're going to have five shots talking to all five other charts, and that will blow up if your transaction become anything more than just move something from one shot to another. So yeah, so I understand I understand the argument. But you had experiments which would actually be apples to apples meaning like you would use cluster sending support for previous work and for your work, just to make sure that for cluster sending there is no difference. But, so I would just like to understand how did you measure this. So, your question is that if the cluster sending the crossing that the cluster sending solution applies to the other to the other protocols. Yes, if you would do that what would be the difference yes. Actually, this part is not trivial as you are saying you can apply the cluster sending problem the cluster sending solution to all those protocols, because the proof of the proof of safety and liveness of these, these protocol are based on the pattern of communication the whole the whole recovery process of ring BFD and other protocol relies on the pattern of communication the fault you know how would be the novelty here the solid point here is that when there's a when there is a faulty notes in different two clusters and the cluster coming the cluster sending is not behaving as expected what would be the recovery process, how we can fast recover from this process and do your cross shot transaction in that order to see our recovery, our recovery solutions that are explained the paper deep and completely in the pay in detail will also explain how we are handled those those scenarios and situation. So cluster sending solution can be added to any paper but the design of the protocol itself will not change right and, for example, a hl itself is having a reference committee which is basically talking to all other committees or all the shots. And that itself, even if you have a linear communication that does not mean hl will stop talking to all of the shots or shopper will stop talking to all of the shots that itself means there's a huge amount of commission which is bound to have in ring bft we just ensuring that one shot talks to another shot in a ring pattern so I mean the combination of so this as yellow said, there are two levels of linear communication happening one is plus sending other is the protocol itself the protocol is basically asking things to go in the ring flow. So, even if you remove one combination the other will still remind there's a bottleneck. So the bottleneck is the bottleneck reference committee, another protocol this one, sending to everyone so the bottlenecks still exist so this that's half of the solution. And from my experience I looked at actually similar things in a model project, but in that project what you did see if you have a multi shot and protocol that works just like in three foot to a three phase with just massive amounts of communication that works fine as long as you have the resources for that. Then it's going to have very good latency, very good performance of course, but it also will cause issues if you want to do blocking to do like transaction management. It will cause a lot of performance issues if you're really trying to max out throughput on all charts and it's actually really much more optimized for single-share transaction than multi-share. If you want to reliably support multi-share transactions, get some sort of blocking to do concurrency control then doing it in the ring BFT way is basically the only thing that can give you not too much issues from what I've seen and some prior to another projects. So for the first talk, Mitosis, given that you guys are from EDC, what's the status of the code? So we implemented, we have a very basic proof-of-contact implementation in fabric where we, so okay, honestly, I was not taking care of the implementation but I tried to answer as again and maybe my colleague can take over and answer really in the chat. So I believe we, so we implemented the crash-fort tolerance protocol where we created, yeah, chains with 10 to 20 nodes, 14 nodes, I think up to 15 nodes and we split that and we measure how long it takes to effectively to boost new, so the title chains from that. Then we have cross-chain protocols for asset transfer and for knowledge transfer that allow us proving the statement about the blockchain to another so that clients can communicate with each other, even if they are touched to different chains and also get 10 transactions from one another. I mean, again, I would have to look at this exact numbers but for our implementation, we, for splitting, so if I recall, to split a shadow with 20 nodes, we need like to, I think 35 seconds or so and the time to split a bigger chain grows linearly with the number of nodes but this can be, could be optimized just that we didn't do that in our code. Yeah, it was like below five minutes for splitting. Thank you, thank you, Giorgio. I have no more questions. Yeah, I have a quick question. Will this affect miners in any way or should I say storage providers? You're specifically referring to Biocoin? Correct. So, of course, this is the independent work that we invited because it's interesting. I mean, if authors of the papers already thought about this, I'm inviting them to answer the question, yes. Okay, thank you.


I told myself that I won't say it, but I will. It is a long walk from there to here. Okay, this is the third day of a very busy agenda and there are 15 minutes between you, between me and lunch and we still have questions. So let me be very brief. I'm just going to respond. I'm going to try to respond to a question that was put, that was a view that was given by Monte Calualia, I'm sorry, he left. He said that economists have a lot of role in macro policy because presumably because there's a framework, economists may not agree on the framework, but the rest of the world thinks that there is a framework and there's data. So it's very difficult to argue with economists when it comes to macro policy, but he said sectoral policies on the other hand is a different matter. And I think it's different because we have not yet learned the art of using good evidence to inform policy when it comes to sectoral issues. And I'm going to try and illustrate how this is being attempted in the Punjab province of Pakistan. Punjab, Pakistan is about 200 million people and Punjab is about 60% of the population of Pakistan. So it's a large province and it has a very active chief minister who actually wants to get things done. So he's aware of all of the urbanization issues that were framed very nicely by Tony and then some of the empirical work illustrated later. But he put three questions. He said, how am I going to create jobs for all the people who are coming to the urban areas? Pakistan is one of the most rapidly urbanizing countries in South Asia, which itself is a rapidly urbanizing part of the world. 226 million people will move to cities in India, Bangladesh and Pakistan by the year 2030. So what kind of jobs will they do? You don't want these guys to end up in low productivity service jobs. You have to create manufacturing jobs for them. And the second question he put is, what do I need to do for that? What kind of interventions can I make in order to make these people employable? And the third is, where do I get the money to do this? So with these three questions and the access that one had to IGC researchers, thanks to DFID, I was able to actually embed the policy research work in the policy agenda of the Chief Minister of Punjab. And that helped enormously. And some of the questions that were attempted by the researchers were basically on the side of making workers employable. Policy research was focused on public transport. Because the Chief Minister did a big sort of project that he's very fond of, which is a rapid bus service in the city of Lahore, a city of about 7 million people and growing very rapidly. He has a very comprehensive plan for a rapid bus service. You can see that there's a green line, the blue line, the orange line, and then the red line. Each of these lines is going to cost about $500 billion. He had the money only to pay for the first green line. It's a 27 kilometer long line. And they announced where the other lines will be. So researchers came in and they said, okay, there are several important questions to be answered here, which is how cost-effective is investment in public transport. And they were able to gather the data based on the one line that's already been been established. And then say what impact it's likely to have in the areas where future lines will be opened up. So become a natural experiment given the investment that had already been done and investment that was going to take place. So the active research is going on in order to determine what the feeder public transport service route should be that bring people to the stations that are along the line that have already been set up and also do some experimentation on how to vary prices in order to increase the returns to the project. So researchers have come in. The timing is great and they're answering exactly the questions that are on the mind of the policymaker in the country. Similarly, health services, question is how do you make cost-effective interventions? Polio vaccinators is one area getting health workers to actually turn up in their offices. And two research projects were started. One was monitoring the monitor, which is a simple technological intervention. You give the supervisors Android phones. They go to the centers where they're supposed to turn up. And when they go there, they just make a call. The call gets registered. The time gets registered. The place gets registered.  And because the supervisor turned up, the rest of the staff started turning up. So this has now been scaled up, taken from the health service to a lot of services across the province. Another project in this area was optimal wage contracts. And again, its success has resulted in the project being scaled up. There are other studies going on in land use. There are about seven or eight large service providers in the city of Lohor. And each of them have their own data sets. And they have their own subdivisions of the city of Lohor. And they often don't talk to each other. So there's huge coordination problems. And what one set of researchers are doing is that they are using fairly sophisticated software programs in order to create the equivalent of zones in the cities. But zones which respond to the needs of different service providers. And again, this was done very successfully in the city of Lohor. And now it's being replicated in other cities. But I mean, I can go on this very exciting project on to address the question that the Chief Minister raised, which is how am I going to pay for this? And Jonathan has already referred to that as Tony. There are two very exciting pieces of research taking place. Again, embedded in the Department of Excise and Taxation, whereby they are experimenting with incentive payments to tax inspectors to deliver more of the revenue to the government services than to pocket themselves. And similarly, experiments are going on on how to use the existing rules of business of the Department of Excise and Taxation in terms of postings and transfers, which is a huge incentive for people to work. They want to get posted in areas where their families are and get promotions appropriately rewarded for the effort they put in. And what researchers are doing is to use existing rules of business to see how by rewarding people in terms of postings and transfers, based on how much revenue they deliver to the Treasury, you can improve revenue collection. Again, embedded in the Department of Excise and Taxation, the Government of Punjab is actually now budgeting the amounts of incentive money that needs to be given, and the project research is getting scaled up. I mean, these micro interventions which then have the potential of scaling up are important in themselves, but I think these have to be complemented by vision, by research that informs the vision, the growth vision for the country or in this case for the province, and therefore coming together and putting the urban vision as part of the growth strategy, overall provincial growth strategy is an equally important function of researchers. So when you come at it from different angles, you come at it from the vision angle and you provide inputs, researchers provide inputs to have a coherent vision, and then you come at it from very micro interventions using up some of the new research methods that IGC is using, you really begin to engage the policymaker in ways in the difficult sectoral areas, in ways that I at least have not seen for a long time, my long career at the World Bank. It's not easy, even though the actual budgeted cost of this research is low, but because it's embedded in the heart of policy making, it's very time-consuming. So researchers have to want to really engage with policymakers and when they do a reward, even a higher reward than publication and the best journal, is to see change being implemented as a result of your research, and researchers who are able to take that perspective, find it very, very rewarding to engage with policymakers. Also, you have to be lucky. There has to be a person in charge, the big policymaker, in this case, the Chief Minister of Punjab, who's a busy body. He wants to be active, he wants to engage, he wants to get things done. When you have these two together, then I think Montyx concerned that economists have no role in sectoral issues can be addressed. I will end here. Thank you very much.