Words appear. Our technology for equal access. Sensory impairments. I'm Eric and I have a visual impairment called Stargarts. And it's a macular degeneration which affects the center of my vision. So it makes it really hard to read fine print and see details. So I use the sides of my eyes to see more and the periphery. Zoom text enabled. The main piece of assistive technology that I use on the computer is Zoom text. Which is a screen magnifier that allows me to make everything on the computer screen as big or as small as I need it depending upon what I'm reading. There's also a built-in screen reader on Zoom text which I can also use to help read documents because my eyes get tired really easily. I use a TV monitor that I have on my desk that allows me to see the board and what is being projected on the board by the teacher. Hi, I'm Mike. My disability is that I am visually impaired. I can see things up close but for the way it gets blurry. I use a whole bunch of different technologies in my daily life. My smartphone works brilliantly for what I need it to. It has a whole bunch of different apps on it that will help me in my day-to-day life. I have Voice Stream Reader which is an input app so you can switch over different types of media and it will allow it to be read aloud to me. Lyceum, Lyceum to State July 19, 2000, 1650 p.m. And that's what the sign says. Scanning is very important for somebody with a visual impairment because there are a lot of printed materials in the world in general that are very not visually impaired friendly or not blind friendly. Hi, my name is Jessie. I am a fourth year student at the University of Washington. I'm doing an informatics with a minor in diversity. I get to find myself as a staff. So as a staff technology, I use all of my concrete implants with my own pose. I use an FM system which I give to my professor so that I can hear the professor more directly through my cochlear implant. It's like a microphone. I do not accommodate and use almost every day is called C-A-R-T which stands for communication access real-time. And so what it is is capturing and giving that real-time capturing that I can read the transcript on the screen while your professor is talking in real-time. My name is Takashi and I have a disease called retinoschesis that affects my retinas. Mostly I use my phone's camera to take pictures of the whiteboard or homework or whatnot and just blow it up as big as I can. I use screen readers, not so much, but I do like them and ZoomText on my laptop. Smartphones are such an amazing tool. I am oh so very glad that I was born in an era with smartphones because I just can't imagine doing school or just living my life without a smartphone. Grace, student. I'm profoundly deaf. I grew up with two hearing aids and then last year when I was 17 I got a cochlear implant on my right side because I wouldn't hear anything through my right ear. In school I use an interpreter, a sign language interpreter, and I also have a note-taker for each of my classes whenever I request one. And if the class has a video or a movie showing that we usually have captions or if there's no captions, my teacher will provide a transcript. There's a lot of group projects in my engineering classes and I just use an ASL interpreter pretty much. And I talk to the other students and make sure that they're aware that I'm deaf. So I just tell them to slow down a bit, or speak louder, and they're usually very flexible with me. My name is Vincent and I'm currently a PhD student at Georgia Tech in Human Center Computing. As it is with people with all types of disabilities, people with the same disability also access information differently. For example, I'm totally blind and I primarily utilize my computers, and I say that because I have five or six different computers with different operating systems. And I utilize them different ways. I access a lot using a variety of screen reading programs with hardware and software based synthesizers. I am wearing something no one's even noticed here. I'm wearing something that's off-the-shelf technology now that used to be military and then it was extremely expensive. This is a bone conduction headset designed for runners. I wear one almost continuously. This is off-the-shelf technology that anyone can use right now. And it's also my watch is connected to this. When things come through now, I hear them in my head, even with people around. And I just hear the notification or whatever text message it is, news updates. And now I don't even have to go to my phone anymore. Words appear. Final Thoughts My name is Cheryl Burgstahler and I direct accessible technology services at the University of Washington in Seattle. As you can see, it's really important that people with disabilities have access to the technology that they need, including assistive technology, so that they can be successful in education, in careers, and all of the activities that they wish to pursue. It's also important that IT developers, including those that create websites, documents, software, and other IT, make those products accessible to people who are using assistive technology and to everyone else. For more information about IT accessibility, consult uw.edu.accessibility. This video presentation was created with funding from National Science Foundation grant number CNS-1042260. Copyright 2019, University of Washington. Permission is granted to copy these materials for educational, non-commercial purposes, provided the source is acknowledged. Described by Audio Eyes.


Hey, thanks for the good turnout this morning when I originally looked at my time slot I was like, oh wow that is crappy, you know Sunday morning at 10 a.m Everybody at death puns gonna be at church Um, I'm glad to be pleasantly surprised that you're all here so With that being said we're gonna be talking to you about some research that has really been percolating for about A year within my mind and then I introduced Mars to it. I've been doing web application security since around 2004 worked with every conceivable application security technology from a black box perspective and the one thing that I have noticed is that Unilaterally the technology has the same problems That whisker had or that the most earliest spider technology had or that the most rudimentary form crawler and Fuzzer has and so there's this common set of Underlying weaknesses that have never really been fixed by the development community by any commercial vendor And so we've created a methodology and a system for exposing those problems And it can be used for a variety of ways and we're going to talk to you about that The big idea though the main overview is that is there anybody out here who's ever got a headache from a false positive? You've ran some commercial scanner and you've had so many false positives that it find reverse Benchmarking is to come up with a system that allows us to begin to expose the flaws and the faults in the mechanics of web application scanning technologies without violating the end user licensing agreements those little things that you click through have actually caused a significant amount of grief on the part of researchers because you can't just Debug the application. It's against the law of course to begin Opening it up and exposing its object code or looking at how it does its tests And it's certainly against the law to discuss these publicly So what's neat about this methodology is that it's totally legal You simply use the application in the right kind of Situation and those faults will percolate to the top now. I would mention just before we get off that We've started a community initiative and a nonprofit and we're going to it's at reversebenchmarking.org It's also the orb project and we're also going to be working with Dennis Cruz and with site generator to get our reverse benchmarking methodology built into that All right Mars you want to take it away? Mars luck I've been in security since like 99 I think Yeah, I've worked for a lot of places no place twice And now the people I work for don't let me actually say I work for them But continuing on with the rest of our speech We're basically that was the introduction by Tom And then we're going to talk about just some general concepts in the web application security span scanning space particularly black boxes We'll talk about the problem of false positives which most of the people that got up here are probably somewhat familiar with Then we'll introduce a reverse benchmarking and how that's somewhat different from benchmarking or baselining And then also just go a little more depth into that We'll talk about 10 of you know common false positive types and then further research and then also talk about some of the statistics that we came up with Here's an example of an end user licensing agreement If you'll notice it says specifically that you cannot disclose how it works And that's the part that we think is creating a chilling effect And the fact that there's actually been very little research in this and that you know the idea of reverse benchmarking while some people have certainly thought of it No one is actually executed on it I know OWASP with the site generator idea Dana sent out emails you know like 2005 You know starting to scratch on this surface and it's definitely a problem that's been known since the first of the web application security scanners I rolled off the assembly line I'm going to pass back to Tom for him to talk about his shuriken Well I'm going to skip the shuriken but I always read this Sun Tzu quote everywhere I go so I'm just going to read it here You know it's the obligatory Kung Fu Sun Tzu quote And you know it's whoever is first in the field and awaits the coming of the enemy will be fresh for the fight And whoever is second in the field will have to hasten the battle Now I think when it comes to reverse benchmarking there's going to be some organizations that are going to be first in the field And that's really what we want we really want people to cooperate and to coordinate because we want a community initiative that makes things better Ultimately though there might be some resistance and some vendors and organizations that give us flak or maybe even try and make what we're doing illegal And in that case it's great I'll just take off my pants and call it the Tom clause You know but at the end of the day we're going to be pushing on because really I'm interested in how we assess and improve the quality of security software overall We look at things like functionality the ergonomics how many clicks through the GUI these are things that people buy a software product for that does security Feature sets it's bling just do you like the way it looks but ultimately at the end of the day as users we have to deal with accuracy With false positives and that's what reverse benchmarking focuses on as a unit of study So ultimately there are a few concepts first off where black box technologies are kinds of scanners that you install from one point You scan and assess remotely the idea of black boxes analogy it's sort of just an opaque barrier because you can't see the inner workings of the piece of software People do this in a way today and the common technique is a bake off is anybody out there ever had a scanner bake off or your company or you've just ran comparisons Well a bake off is sort of the opposite of what reverse benchmarking is a bake off is a positive type of benchmarking Where you're after a comparison of which technology did better or which technology found more and had less noise Where reverse benchmarking is a way of making or creating soliciting false positives and causing massive false positives in any technology where there's a propensity for those things to get out of control So your goal is not to see who did better but in a way to see who does the worst Maris? Alright here's a little graph that we put together to talk about kind of the criteria that people have used to talk about vulnerabilities Like in the different type of product lines of vulnerabilities sometimes understood in different ways You have the IDS line of thinking the network security scanner line of thinking and the web application security line of thinking But in order to kind of frame this talk and the research these are just the types of four quadrants that you could possibly have when you're looking for vulnerability Positive detection would be that you have the scanner finds a real vulnerability that is there The false positive which we're focused on is where the vulnerability is not there but the scanner in fact finds a vulnerability that isn't there And so that's the focus and as things move in the space and web application security scanners become more developed Then we can start looking at the higher quality problems which would be the face where there is in fact a vulnerability but they don't detect it But that's the mystical part of the industry right now that we don't know what we're actually missing But the way our approach is that we'll first be able to look at what false positives are there and once we can get the false positive rates down in the tools Then we'll be able to move on to the higher quality problems of finding more legitimate vulnerabilities That's a great point and it's the massive sense meltfall sense of security you get from something that isn't digging into your application Yeah, no absolutely No, that's right. There is nobody and I mean I just we wanted to put it there because it is relevant Yeah, so Dennis just mentioned that there weren't any security technologies that would look at the application until you this vulnerability is not present But rather the technologies that exist are all focused on positive types of detection Whereas the technologies that could exist on the false negative and false positive side could even lean toward the certifying of an application Say against a particular vulnerability type I have looked at you with the following criteria and detail and you do not have this vulnerability That's a much stronger statement than what is currently being made but ultimately we want to phase this research into the study of false negatives But that will be down the line Or that would be the ultimate goal of any benchmarking tool is to do that We've spent some time on these slides I'm going not necessarily going to pick up the pace but I am going to say a little bit less This is just on positive benchmarking the idea is typically you throw out you figure out what percentage of accuracy did the tools have Scanner food is all found 8 out of 10 vulnerabilities that we know therefore it's accuracy was 80% Now that is what people typically do with positive benchmarking and then they whip out their numbers and decide what to do on the basis of those numbers The problem is that this methodology is examining or evaluating web application security technologies is limited There's a lot of factors like this selection bias do you really know what vulnerabilities exist within the sample application you're studying You might think you do it first until you get the vulnerability data At this point it becomes confusing to people and the process just sort of breaks down There's a you have to interpret the data of vendors can tune against a particular application so that you think you've you think you've found a particular technology that just a stellar against your set And then you realize oh wait they wrote specific rules for what I was just testing against Now this is just a point back that say like site generator was designed to sort of mitigate the tuning aspect of technologies And reverse benchmarking is designed just to go underneath that radar to make something you can't tune against because we can always create a new fuzz set that triggers your false positives The best example there would be like webgoat Yeah webgoat people tuning against it So what is reverse benchmarking my it's just designed to kick a scanner's ass and we're going to trademark that I think the point though is because I work for a technology vendor I'm not approaching this from my own vendor standpoint But that is why I wanted to create a community project an open project that way it's out of my hands because it can't be objective as long as Tom Strasner is doing it This when the community does it as a community then it becomes an objective methodology Now I'll still participate in this organization as a member of my own company and that's the way the Lord would like other participation but that's the idea Yeah are there any vendors here watch fire, spy dynamics, acunetics Oh hey what's up It depends on the false positives Yeah absolutely but that's a good point and there's also the idea of semantic differences right You can pick an obscure corner case that can cause false positives but how relevant is that to say a production environment It depends on how you're using the methodology but there's we need to work out issues like that We're definitely focusing on catching the lowest hanging fruit right stuff that should obviously have been weeded out in some type of QA process Like in the essence of full disclosure I used to work as SINZIC as well Overlapped a little bit with Tom so you know I know that there's definitely some QA gaps in regard to you know the industry in general Our focus is you know catching that low hanging fruit and so we're not you know focused on coming up with the obscure corner cases like Tom was talking about But just doing stuff that is completely wrong So once again just to make everyone on the same page what we basically what you do when you want to set up a reverse benchmarking environment Whether you're creating itself and we'll talk a little bit about how you could create one yourself But ultimately you have a web application scanning technology and then you have a reverse benchmarking target or web application with reverse benchmarking capability There are you might think of as trigger signatures and strings and strange architectures You can imagine say a javascript road test that's designed to confuse any type of crawling mechanism and make it believe that there are URIs or URLs that don't exist You can build within this trigger mechanisms that make it believe vulnerabilities would exist Ultimately you scan this application and then just get a report and if you design the application right the report can even tell you what types of false positives in a taxonomic sense have been discovered Now obviously that's going to take a lot of research on the part of the community to figure out well first what's a good taxonomy for all of the general and generic say false positive types But ultimately you're going to enumerate and then categorize the false positives It will reveal sort of broken or vacuous signatures if something has a detection signature that's just 200 then it might be triggered by it's going to be triggered until the year you know 2010 So it will have this false positive for the next three years Well something like reverse benchmarking can lay that bare really quickly It will reveal really semantic flaws and categorization and it will give you the idea if there are any like sort of systemic architectural problems with the technology When you go through the javascript road test type functionality and then into the scanning you will see where things begin to break down and you'll see limits in the technology Marce you want to talk about the trends of time wise we have where it tends to extend Well I don't really want to belabor this slide it's just basically the general idea is that applications web applications since you know 99-2000 have been progressively getting more complex And the scanners haven't really stayed pace with it they've you know taken some definitely increases But most of the changes in the scanners have been more GUI focused and how fast they can go through their signature base But changes in the signature base and the you know the technology there is you know just not really kept up with the GUI development And also the numbers they use to categorize their level One of my friends who does manual penetration testing was lamenting to me and he says it's getting harder to do my job every year I'm having to do more work not less and it's just because technology is growing faster than the technology we use to say perform assessments And that's one of the reasons why an environmental stressor like reverse benchmarking can help improve technology over the long run Because it's an equal and fair thing that will also allow the community to help to understand and educate developers And that way one day when developers are writing new security rules they'll realize that it's essential to put a no 404 check into a scanner Right that a 302 redirect this should not be considered evidence for vulnerability We'll show you some of these false positive types when we get right into them I'm going to skip ahead a little bit Just to give people kind of an idea like on my day to day it is a lot at doing black box testing of various web apps for large financial And it's not uncommon to get you know a 5000 page report that you know has a significant number of false positives that have to be manually validated So I mean that takes at least you know 40 hours of every week just going through that stuff So that's one of my motivations for doing this is you know the self serving one of hey maybe I could work less at some point in the future So I thought we'd warp ahead and look at the actual types of false positives And these are real examples of false positives that we're not showing who had it or who's was it But here's a good example of a partial what we call a partial match problem Now as we go forward as a community we can figure out say different terminology or extra false positive types But I'm calling this a partial match problem because the detection signature was literally a 200 a 200 So anytime a get request was issued for .pl back this 200 was matching the date And of course it will until we mention it will until the year 2010 And so this is this is something that would have been found in every assessment report ever run by the technology And the idea there is is it's good to know these things because you can weed them out as you're a worker I mean if you're using these technologies it can make your life easier to know what the false positive types exist You can flag them Parameter echoing is a big one and here I've put my my very elite PHP script This you don't get much more elite than this you didn't see this at Black Hat But there's something simple like this that's just echoing junk into a text area Will really show you a high degree of the semantic level problems with technologies And let me give you an idea there is there is a cross-site scripting vulnerability in this script That's echoing it in the text area and it's if you break out with a text area tag However you will get thousands or will sell hundreds of different types of cross-site scripting vulnerabilities that are bogus In other words semantically the name of the vulnerability may be a double quote Front tick backslash cross-site scripting attack works Well that's a vacuous result because everything works the the delimiters are irrelevant If you if you if you encounter a part of the application where the information is being echoed back in a non executable form Right and so areas of the application that are just spewing data back and cause false positives Because in many cases one little character that you would need to make it ultimately executable may be missing Where it may be injected into a JavaScript function causing a train wreck and the data is is just not relevant So parameter echoing this will give you just one example if you were creating your little sample application Sticking something like this in there and then a page behind it with some common strings would give you an example of just how susceptible some of these technologies are So here's a case of what we call mistaken identity now this is a good one because all there's so many open source bulletin boards And forums and blogs and everything has become so in in bread and cobbled together that you can literally get situations where you'll run a security assessment tool And go back and the test will be for search dot pl you know have found this pearl script and it tells you you know Well this is Alibaba search buffer overflow but the fact is search dot pearl matches hundreds of applications out there so you don't even know what the result is You'll also find this on if you're scanning same bulletin boards and there's a there's cross-site scripting in a Q comment area Well until you inspect the application you don't know if it's real because there have been hundreds of different applications with the same vulnerability The name that they're giving you may not even wildly match the technology you're using So this anytime you have issues where just the identity is confused there's more work for you to do on the back end and so this is a species of false positive The other other problems that we see are just simple ambiguity right the the conditions of the string that's used to detect a thing just doesn't go far enough And that just because they found to say a sequel error they're telling you give full-blown sequel injection vulnerabilities In other cases there will be some feature of the application that gets picked out and the vulnerability just resembles nothing like what is looked for And so these are issues that that make reporting confusing and are time consuming and so there are areas where studying and categorizing those will be helpful Mars you want to go ahead with response timing this is a really big section because there's a lot of tests for say sequel injection that use a wait for delay mechanism They will just wait for the they'll they'll inject a sequel command that will cause the database to count a hundred and then respond back Well you can get applications course that are just slow and so you can see these sequel mechanisms just misfiring like crazy and assigning vulnerabilities to you So with the reverse benchmarking target you can just create a portion of the application that is slow and make the timing different or gradable And see how many sequel errors just pop up this works with blind sequel injection as well Or just use portable web app and get that functionality for free Now with a custom 404 this should be ancient this goes back to the very first scanners of 1998 and 99 written in pearl and put up on root shell I mean the problem is if you do not check for a 404 message that's clean if you are simply keying off of the presence of a 200 ok As a status message as evidence that the vulnerability exists then if a web application is set up a custom 404 page where anytime you search for something in the URL You look for a resource it's not there you get redirected to a friendly page that says hi file not found but the underlying status quotes are usually a 302 followed by a 200 ok This causes scanners that don't have this mechanism to go nuts Now that may seem so old like no there's no technology in the world that would still do that today Well there is in fact commercial technologies on the market today that perform their file scanning without these mechanisms and there's a cause of rampant false positives We're going to look at some real data here shortly why don't we segue over into that Marston you can talk about the data segment Well just as an introduction we called the Tom called the application he originally wrote back at you Basically set up that application it had a number of tests some of which we've covered the slide deck will also be up on the reversebenchmarking.org Probably later this week after we get back to our respective home bases We took four popular black box web application scanners ran them against the target with their default policies and to get these results I took the results and put them into high level buckets some of the high level buckets and stuff like that are fairly arbitrary But you know we're not specifically saying which scanners did this or did that and we're not giving out any of that data so no one should be really concerned This is just the total false positives for the four different scanners as a percentage of a whole so you can see one of them did really badly And then the other ones are somewhat similar so there has been some changes in the community and some of the products are at a higher state of development as you can see Yeah just to point that back out this slide just shows of the total number of false positives we generated 92% of them were generated by one of the commercial technologies Now one of the reasons why such a desperate amount was that it did not have a 404 check so it found 5 to 6000 files that just didn't exist Just that every backup file in the world and every way you could possibly rename them is in your CGI bin Now if you take that data out they're all sort of doing equally badly right and so this doesn't meant to say that there are three technologies that are stellar and one that's just awful But really one just had a very very noisy problem that they should Well even controlling for that it would still have had considerably more Okay here's a scanner once, false positives Alright if you look at the different categories I just put in is path manipulation, it had some type of path traversal or path disclosure types of false positives There was command injection, windows, unix, probably a lot of people are familiar with that And then there was all of the scanners had cross-site scripting false positives Which that was just simply putting a script, a java script with cross-site scripting or XSS in just a blank page and they all triggered on that One thing I would add that is interesting and this shows that there is improvement going on Is that you might think that say cross-site scripting would be the most prevalent vulnerability if you had say an echo mechanism like our little elite PHP script But in fact in this application it was only 7% and the other characteristics we actually had SQL injection false positives we could generate upwards to was just 2% But with command injection and path manipulation that was combined 72% of the results And so that shows you areas of security testing where if you had to tighten the testing procedures, implement mechanisms that say Will during a crawl look for the presence of the strings that you are going to use on the basis of detection and then rule those out as sort of a pre-qualifying run Will technology improves, no one's doing that yet, this just shows you how reverse of inch marking could open doors to actually inspire people to code differently With SQL injection, those are basically strings from SQL servers that just were in text areas in a static HTML page So that's what generated those The file disclosure was any type of file, mostly the 404 types of problems Tom was talking about where it thought it found a vulnerable file that it didn't The known vulnerabilities that categorization was basically any of the semantic types of mistaken identity where it said Emily forums 1.0 had a vulnerability in PHP blah or whatever And then misconfigurations were mostly things like to do with the web servers or something like that With scanner 2, you can see that the majority of the false positives there with past manipulation and then also interestingly the file disclosure as well So as you can see there is a dispersion of the problems that are kind of across the various scanners that they each have kind of signatures that can be fooled fairly easy With very simple problems, low hanging fruit as we were talking about Scanner 3, the bulk of them were actually misconfigurations and things that it thought about a web server or a web version or a version of some type of scripting software or something like that that actually wasn't there And then file disclosure was also a problem with these Scanner 4 also had the same problem as Scanner 3 I mean these are just trying to give you an idea, the main takeaway here is just that each of them still have most of these Though some of them have less functionality and don't necessarily even test for some of these Was it a sequence or something? Scanner 4 had a big problem with SQL injection as well, it's SQL injection technology was just pattern matching And so the fact that these SQL error strings were in the HTML messed with it more than some of the other ones That's kind of just the We can let them download it from the web right off Yeah we're going to put these slides up and you can draw whatever conclusion you want This talk is mostly about getting this idea out and trying to get some community support and get people to start submitting Foss positive scenarios and foss positive types to where we can have a bigger set of tests and come up with better road testing applications We've also talked about having a tax on me of foss positives because until we truly understand like foss positives in a more robust scientific sense I think engineers are going to have a hard time understanding that and the web application of security space is still going to continue to be more of an art than a science And so putting some rig around that is also some of what we're looking to do For further research we want to improve the benchmarking target, add more tests like I said, improve the testing methodology Because right now there's so many different signatures that are kind of thrown out by the application scanners that coming up with the buckets is somewhat arbitrary And I did that late last night so it's far from perfect and then there's also I would also mention that there's a sort of a dual level here because dealing with the application spider And what causes the spider to enter erroneous states, get hung in infinite loops or even find things that don't exist is really key For instance if you just take an HTML page that looks exactly like a directory listing You can find out which of the spiders will begin following the links on that page thinking that they're entering new directories when they're just following HTML links that just double back onto itself You can create strange architectures that will expose the faults in spiders and so when the vulnerability scanning develops Sort of a side that is specific to kind of road testing and generating false positives in spiders is good And site generator has aspects of this because you can dynamically generate pages in javascript and flash And so I think that there'll be more room for fruitful interaction there I mean, what were we at? Okay, alright, that's strange, I thought I had ten more pages to go Well Tom puts that back up there, just to clarify the testing target that we used and the testing that we did didn't include any of the type of spider evaluations So that's one of the other areas to where focus is needed and like Tom said site generator So we also just want consumers of these products to be more educated and be able to make decisions based on how accurate and how robust the web application scanner is as opposed to the marketing GUI Okay, he's since we have a few minutes more going to pull up the sample app and show some of the tests Okay, and we'll also take questions, it is not with attribution, I mean we think it's legal because there's no specific attribution Dean screws everybody I was just asking is it legal to actually disclose on the eulah right, is actually disclose information of what they're not finding So because it would be interesting loophole right because if the eulah doesn't say you can't disclose what we don't find But they say we can't disclose the results right, so it will be covered by there Although from the OAS, what we will ask actually officially is that we'll ask each of the vendors to actually give us a license And we were talking about this a couple of days ago and I think what we'll do is we'll give the vendors one chance to comment on the results Just in case you know you get something really wrong So you do the thing, you send the vendors, hey we found these other results we have, is anything we missed pretty stupidly And then you run the thing again and you publish the results And we hope there's no vendor which is stupid enough to actually try to tune in the tool to actually match the patterns Because that would be a big problem Thanks Deanus, anyone else have a question or are you ready Tom? Yeah actually I want to just say a few things about kind of the junk that we put into the sample app First off just having an area which echoes back say things that look like a directory listing Things that look like the output of netstat minus AN Of course your default boot I&I where you'll have your obligatory bootloader Then some dates and directories Then you have things like Unix or Linux ID command where you just on the page following the form it tells you your UID 0 And so it's things like this that get put into the application that cause trouble for the mechanisms We begin to adapt and develop more sophisticated text testing procedures Then it's where these methodologies actually improve technology on going And so even if disclosure never happens or even if there isn't a showdown It's not our goal to make anyone look bad But it is our goal to expose false positives in a way that help the community think of understand the problem better Understand a taxonomy of false positive types, help developers when they're writing code go ooh Avoid these 10 things and ultimately make the technology better for everyone Now with Owasp and Dennis we may be doing some participating in some events and things like this But the ultimate goal of our methodology is research oriented and to help in understanding of false positives And to get people from all organizations to contribute so that we can understand the problem better No and the questions? Yeah No it's not available right now, it's a little too ghetto to give out That's my attitude of it but Yeah we're going to work with Site Generator to come up with some tests And we thought of coming up with maybe like a top 10 kind of false positive categories And then categorize different tests that test for those Which Tom you want to go back to the like top The 10 different false positive types that we kind of rushed through Oh did you take that slide out? No But anyways yeah here I'll walk out Okay Not going to work The court has length but Yeah there's definitely the custom 404 error pages that the different vendors have developed And with custom configuration when you know your applications you can certainly weed some of that out In fairness to all the vendors and to have a baseline We didn't change anything from the vendors defaults right Because if we were whittling around with all the various settings we could have biased You know for at least in our under in the way our thinking was just use everybody's default policies And default scan against the same target and that would be the closest to you know And it's a pure kind of test Let me answer in on this too also we weren't trying to do any official benchmark or showdown And we don't put any stock in our numbers that's why we didn't really even tell you who was who Our goal in getting the results that we did was just to show you that If you take existing technologies and do what we are talking about here to them You get useful or interesting results and then those are results that other people can do Showdowns or comparisons with right Right that's right and see that's where we would that's where working with you would be a big help because Reverse benchmarking.org doesn't plan to get into the business of comparing or publishing comparisons officially But we would work with you and OWAS carry that out Yeah, ultimately users tend to you know like in the marketing literature you're how many clicks to the user can set this off So I don't think many of those custom you know configurations are actually really used by you know the vast majority of users of the products Maybe I'm wrong Okay, well there is an alternative opinion So you're saying the majority of the users of the products are actually very skilled as opposed to the user base is not very skilled or Just by the check What happens then there's really a big wave of skills from the security systems Usually they know what they're doing. They usually are technically enough that even if they don't fully understand it You'll get it.  They might not be security experts, the lab will not be able to break anything Well that was the easiest way to do the testing right Yeah having some idea of sorry to cut you off Having some idea of where hours are spent by people and where you know how much time is spent and something like that is definitely I think useful And maybe some academic would like to explore that you know working with the vendor and having some type of you know pool of users And you can have a control set that were you know the clickies And no zero configuration kind of pool as opposed to you know expert power user type of pool And you know do the evaluation across those but I think that the precondition for that would be some type of reverse benchmarking or benchmarking app You know like site generator that can't really be you know ahead of time the scanners can't be configured to you know go one for one against its tests Having some dynamic generation having both positive benchmarking and the reverse benchmarking types of tests So yeah this talk is just to get kind of the ball rolling and getting people thinking about this kind of stuff because ultimately you know I think there's a lot of time wasted you know I know from personal experience a lot of time going through FOS positives Even with the various types of tuning using you know multiple different scanners Still there's a high rate of FOS positive it I mean it depends on a lot of things There's a lot of different variables that we can't necessarily control for you know right now and might still be very hard to control for in the future Especially with the pace of web application you know technology and development as web services come on AJAX and stuff like that Just the complexity is going up and you know so Also well you had a question in the front? Yeah Yeah by the way we're going to be in the We'll be going to the breakout room after this if you have more questions and you want to sit around and chat We'll be heading over there after the talk Yeah that's a good point Yeah, definitely there are you know Different classes of users and the audit kind of function or compliance kind of function Uses these tools as well Enterprise technologies will also capitulate this cycle Generally if depending on who did the configuration or how many servers are being scanned There's general you know not a lot of opportunity to insert yourself into a process and make changes So it just depends on the functionality that's present but you can easily find FOS positives being More of a problem in situations where you either have the technology being configured by a team And then ran over your whole network where they're not looking at each segment and or each specific server Looking over the reports Well, there's certainly even if there's no motivation and the people have the appropriate skill set There is still just the fact that this takes a lot of time to do and if you don't know the application When you hit the ground running and you're scanning something you don't know You're going to take multiple days just figuring out how the application works And there's still the case that you know manual testing Manual pin testing of a web application is more effective than any of the scanners There you know so I've no one's published though, you know specific real results in that regard I've seen either But that's kind of you know one of the big you know things in the industry that no one really talks about is that An educated you know skilled security consultant is still going is still more bang for your buck Then you know people running just commercial tools, but there's the time function of things that you only have two weeks to do something Or three weeks to do something like you know there's there's costs associated with it And if we can get the cost of a false positive down or the number of the false positives down Then people will be able to focus on other types of problems and ultimately be more effective And the third party third party roots, you know you can potentially get you know more effective results from them and more bang for the buck Because using commercial tools is a necessity You know at least in the world I live in there's no other way to do it just doesn't scale doing all manual testing And using the combination of the two is certainly you know the way I approach work is I generally use commercial tools to do A first pass kind of you know benchmarking and then I go through and go through the false positives and stuff like that And catch the low hanging fruit with the commercial scanners and then come back given time and look for more advanced things You know with just a proxy and you know raw tools and stuff like that Because we generally don't ever have time or are we given source code so we do everything black box And that's certainly more you know I think it's in some ways challenging more challenging some of the others But it doesn't require the level of knowledge that you know some of the other source code analysis and stuff like that So in general kind of the assessment industry is going to have to come up with ways to deal with this Because all the commercial tools black box gray box white box however you call them are still not as good as a human And which is good for us right we get paid to do it But having more effective tools where the humans can spend less time you know fighting the tools instead of fighting You know vulnerabilities or whatever it's what we're looking for so thanks everybody Yeah thanks everyone


How is the arm feeling right now? Great, ready to go. Was there anything that told you that the issue was behind you? It was no longer irritated and you could get back out on the mound? Yeah, I mean I took a few days off of actual throwing at the end of spring training and once I picked up a ball again I could tell that it was kind of refreshed and calmed down.  Was there any concern that it would be something more serious than it actually wound up being just a couple days without throwing and then back on the mound? I didn't think so. The images came back kind of how I thought they would. Fairly clear, slow irritation in there and got that taken care of, so ready to go now. Will you need to change your preparation or the way you train at all because of that issue? I mean there's a few extra things that do with the trainers, just to keep the shoulder in a good spot and some recovery, but nothing drastic, no. Do you feel like you'll be able to go back to back days or is that something you'll need to build up to? No, I feel ready for that. Pitch whenever my name's called, really. No, I feel good, no restrictions. Yesterday of course a little bit of a hiccup but overall what have you thought of the pitching staff to start the season? I thought it's been really well. Yeah, yesterday wasn't our best day but you know that's why we got another game today. But you know it's been really good from our starter standpoint to kind of the bullpen throughout, it's been impressive.  Very anxious, yeah, I don't like not being involved in it so I'm happy to be back on active and hopefully get out there today. Justin, thanks for the time. Thank you.


Okay, who gets excited with a bumper like me? Guys, kita, good morning, how are you? And why do you know me like this? Hashtag is why in the morning. Can I tell you a secret? And no, sayam bmw tu saa. Okay, twa kwa TV. And it's not much of a secret because there are like thousands of you watching, but sayam bmw tu. I think I'm shy. Like, generally, like... Kuli kwen metimbea, I was going to see somebody I care about. Alafu, nika kutana rendiflani. So just imagine we're two girls and then everyone else nini nichiwali. Alafu, I was expecting to be told hi. My name, I was expecting to introduce myself. Hi, my name is Val. Who are you? A-a, napraval! Bro, hi, hi, hi. Koza nichiwa nikiwa. I'm like, you are this excited to see me? I should be an authentic. You don't even know. You don't even know. What if I form Facebook? What if I form a channel on Twitter? Hashtag is why in the morning. Now, insto deal with me because Wednesdays I reserve for special people. I will remember you. Okay. I will be there on Wednesday. Okay. I will be there on Wednesday. I will be there on Saturday. I will be there on Wednesday. But hashtag is the why in the morning. Let me allow her to introduce herself to you. Hi. Hi, Hawaii. Thank you. Please introduce yourself to the people. My name is Grace Murema. K-a, Grisel. That's who call me Grisel. That's my social media. Even who knows me as Grisel. I am a special effects makeup artist. Yeah. Can you first take a moment to appreciate how good you look? I like your whole look. All of it. All of it from your hair to your top, to your necklace. I just like a bit of your hair. Thank you. Wei. For many people, good things happen when you watch it in the morning. Good things happen. Tell us a bit about yourself. Like I said, I am a special effects artist. I started in South Africa. What is your name? Kupakubuwa pole pole? Sipo, we are going to UASIU. We can't sleep because of English. Yeah, we are going to South Africa and you are saying it so nicely. Anyway, come, bless you. Well, the reason why I had to go all the ways is because there are no schools in Kenya that offer such a cause. So I had to do research. It was either states or I say, so I just ended up settling for South Africa. Studied for a year, then I came back to launch my career. Okay, just personally, tell me how is it like? How is it like? Just to say, job bag. Wei! Chok chiza jini. I want details. How is it in job bag? Like, how is life? Like, what's... Uli kuna kula ni nisembe? Skumas kuma manago? Terere. Like, what did you eat? What did you do? A lot of takeouts because I was living by myself. Exactly. I was living by myself. A lot of noodles. No. I can relate. So, cooking was very... It wasn't something I would do all the time. But yeah, it was just easy. There's nothing special about South Africa, to be honest. Yeah? We had a recent outbreak of streets in Ophobia, Juzi. Have you ever experienced it? A lot. Really? Yeah. I'm surprised. I have, but... Tell me the experience. This was back in 2015. It was just the whole thing. They felt like foreigners are still in their jobs. And it was just very dramatic. So, we were told not to go to town for some time because that's where they would target foreigners.  So, anyone who sells vegetables along the road, they have a problem. But then they wouldn't do such a job because according to them, kazi ofisidi o kazi, like you can't... They can't sit and start selling vegetables. But then they also don't want other people to do that. So, it was a bit confusing. But I experienced it a lot. Even in school, I experienced it, but I survived, I guess. I once met Tamu Indian. Tamu looked like a gozi out, same, but nearly tofouti. And he was darker than me, but he asked me why you so black. I looked at him, bro. I didn't even have an answer. I was like, are you serious? Okay, God be with you. God be with you, I'm sorry.   We live in a world where one person in America won. One person in America, Brianna Taylor, Sijuid George Floyd, Sijuid who is killed by a white person and the whole world is up, all up in arms and there's a whole revolution.  Nama liza na tui viki ovio, but we won't talk about it because, you know, black life is not matter then. I'm just asking for a friend. Hashtag is why in the morning. But that's too deep for in the mornings. No, tell us why you're here. I have to talk about my crafts. I recently just started, well not recently, maybe late last year, I started as a school to teach. Look at you. Special effects because not everyone is lucky enough, like me, to go study. It's expensive. It's just financial, it's draining. So that's why I decided to start the school. I have my own product line for special effects. Look at you. So I'm just trying to survive. It's a miracle, what do you mean? Okay, okay, okay, I have so many questions. So many, and I'm very random. So I'm sorry if just random questions are popping up. But why special effects? The makeup industry is a bit, even as a makeup artist, is through every single person. Not that I have anything against makeup. I just felt like it was just too crowded. And at that time when I was looking for work, they would ask me what was so special about it. Like why should we give you this job and what exactly what special thing are you doing? What are you bringing to the table? Exactly, and I'm just like foundation, lipstick. Like there's really nothing exciting. So I had to figure out a way to jujup my makeup. I'm going to use that. To just, you know, up my gameki dogo. So that's why I decided to just not get rid of makeup, but to top up to it, yeah. Okay, guys, okay, first of all, kuna shoflanin kuna penda kuwachi. That was kuna kumbuka, but I just started explaining it to her and she knew what I was talking about. So there's a show called Face Off. And Face Off is basically, they give you certain type of tasks and it's all about makeup. All for the waste you from sculptorings, do you nakawu vareen, kidok dogos, do you? They use all kinds of random things to just transform something. So if you like horror movies or if they, in fact, if they freak you out, you should watch behind the scenes because it's all makeup and effects. Na sa na sa na sound. You should try watching a horror movie without the sound. It's not as freaky, trust me. And I think the whole process just fascinates me. I love everything creative. Anything that I can see started from here and then suddenly is something. You're magic. You're just basically magic.  And if someone doesn't understand special effects or the gore or the goo or the things, has they ever told you, like, form? What's this? Um, yeah. People still don't understand. Like, my grandma, she just says I do makeup. Woo! Hwaza ancestors watasema nyu chawi. Why? Because special effects includes, like, blood and all the gore and all these things. So if you can create something like that, walaesu mero go. No, because my mum actually understands. She's not your normal, traditional, she's woke.  So for her even when I told her I wanted to do makeup, she was 100% like she's my biggest support system. So she's the one who was even watching for me like, okay, so what exactly is it? She's never tried to tell me, okay, this is not working out, try. It's like just keep trying, keep trying, you'll get what you want to get. So, I hope I over my everything. Hwaza, I was going to ask you, there's a time in life where kwa witu flani kama DJ or an entertainer, armam sani, or even just random, just basic makeup artists, ilik watu. Kani kuna, what are you doing?  But now look at us, tell me about your first experience with special effects. My first experience was straight, I think after school, this was in early 2016, I came back in 2015. There was a lady she's called, she was on, I'll never forget, because she gave me a very fast gig and she got me off Instagram. So she said, oh, we're shooting this short film, we don't have money to pay people, I don't know if you're interested to come on board. I was like, of course, I mean it's my first job, I can't be there saying how, I want this much, because at that time I didn't even know the rates, how much product I'm going to use. So, it was a short film for a project called 48, our film project. That was my very first experience and I actually really enjoyed it. What was your favorite part? Being on set, working with different departments, these hair, these arts, these wardrobe, the director, just blending in with everyone. Have you ever had a very difficult client? You haven't then said you missed the spot, have you ever had that? For special effects, to be honest, I haven't. Make up, of course, but special effects, no, I haven't encountered any difficult client, because they don't understand first of all what's happening. You're smart. You're smart. So it's fine. What was your most challenging project? My most challenging project actually had two. One was a movie that I did in Uganda. It was called 27 Guns. Wait, wait, wait. So, you get to travel also. Please do what you love. Please do what you love. Look how happy she is. She's even traveling. Anyway, continue. So in Uganda, it was a film based on what happened in the 1980s with the first family and I'm saving in them. So it was a lot of dead bodies involved, chopped off hands, legs, burnt bodies. So that was challenging, but yeah, I did it. And then the second one was a project that I did with Buen Fismangili sometime last year for aging people. I had to age like 60 people in a week. Wow. It was a bit challenging. What does that mean for you in hours? Did you sleep that week? 60 people in one week? I did. It sounds a lot. But of course you need to have someone to help you out. And it also depends now with the call time, how early people will get there so that we crack people up faster, faster we go. But yeah, I managed. So I'm grateful. I don't want to judge you, but I will just a little bit.  I keep time. You see? I know it. I know you have that vibe. Because I don't like people making me wait so why am I making other people wait? We have a couple of pictures we want you to see on her Instagram page. We have a lot of examples of the magic that she does that is a special effect. So remember to watch that on Facebook by 254Channel on Twitter. Hashtag is why in the morning. So maybe let's continue the conversation in a note projected on the screen and we can expound a little further. Where is the place you traveled that your job took you that you never thought you'd be? Well, so far it's 2G. Just 2G. We never left Nairobi. Can you say Uganda? But I'm hoping to get other jobs that are out of Kenya because working in Kenya sometimes can demoralize you. Why? Because of just the pay. That's also one of the problems I encounter in my career. People want a job well done but they don't want to pay for that job. Don't you pay with exposure? Not even expo. May that exposure thing I don't like I'll hang up. The commercial do not continue after I have exposure or experience or opportunity because you looked for me I didn't call you, you called me. So that means I'm exposed enough that you don't actually call me. How did you find out about me? So exposure doesn't work out for me anymore. But they still don't pay my full rates and that's also a problem. So I'm just hoping to spread my wings and look for work out of Kenya. When I was in school a lot of the work that I did was very practical. So if maybe we were learning how to design a website we actually had to design a website. It actually had to go live. We actually just had to be real and true and that's how we're going to be graded. Is that the kind of experience that you had when you were studying special effects? Abu, paint me a picture. Because in school we did a lot of practicals but then we actually did it on ourselves. You can do it on yourself? You have to do it on yourself or you do it on each other or you just prefer doing it on yourself because now you can take all the time because if you are practicing on someone else and then they are taking hours they are tired now you have to change then they work on you as well so it's just easier to do it on yourself. Spend hours and perfect it. But the funny thing is even in my class right now everyone else dropped special effects and no one will continue. Why? I have no idea. I think it's just me and maybe some other lady from Zambia who is still continuing special effects but everyone else.


The next item of business is a statement by Mary McCallan on the Edinburgh Tram Inquiry report. The cabinet secretary will take questions on the issues raised in her report at the end of her statement. Therefore, there should be no interventions or interruptions, and I call on the cabinet secretary for around 10 minutes, please. Thank you, Presiding Officer. I am grateful to have the opportunity to make a statement in response to the Edinburgh Tram Inquiry report. A comprehensive document totalling near the 1,000 pages that I received in the morning of publication on 19 September. The report was also laid in Parliament in line with the requirements of the Inquiries Act 2005. I believe that the report addresses the terms of reference that were set, namely to establish why the Edinburgh Tram project in curd delays cost more than originally budgeted for and through reductions in scope delivered significantly less than projected. I recognise the extensive work and efforts of the inquiry team in delivering the report. I thank all those who contributed to the inquiry, including the many witnesses who provided evidence. I am aware that the construction of the original tram line caused a great deal of disruption to the residents and businesses of Edinburgh. I believe that it is important that we recognise that frustration and ensure that lessons are learned and applied to future infrastructure projects, be those local authority or central government. As such, I can confirm that this Government has given very careful consideration to the full report, along with considering its recommendations, any actions required and lessons learned. I would like to be very clear that the primary objectives of this Government in establishing the inquiry and throughout the process have been to support delivery of valid findings and recommendations and to engage meaningfully and to co-operate fully and openly in the production of evidence at the inquiry's request. Significant resources were committed to diligently carrying out this endeavour, and all those who gave evidence on behalf of the Scottish Government did so in good faith, providing the inquiry with a comprehensive and accurate view of Scottish ministers' collective position throughout. Now, while I welcome the formal publication of the report, I understand and I empathise with the public's frustration at the length of time that it took to conclude, as well as the cost to the public purse. This is, of course, particularly disappointing, as it was this Government's concern for the prudent public spending that saw the commission of the inquiry in 2014. However, as an independent statutory inquiry, it would have been very much beyond the powers of government to seek to influence the proceedings, and any questions on the length of time and of the cost of the inquiry are rightly for Lord Hardy to answer. The report itself contains 24 recommendations. A minority of those are directed to the Government, and those mainly concern administrative processes and record management, including minute-taking and legislative and practical aspects of setting up inquiries. All recommendations, as I have said, are being considered in detail. The report also outlines 10 headline causes of failure, which contributed to the delays and cost overruns associated with the project. Nine of those relate directly to the actions of the City of Edinburgh Council and its arms-length delivery body, TAI, with the 10th and final cause only relating to the Scottish ministers. Indeed, the chair, Lord Hardy, is unambiguous, noting in a video statement that he produced alongside the report that, I quote, TAI's failures were the principal cause of the failure to deliver the project on time and within budget, and he adds that the City of Edinburgh Council must, I quote, also share principal responsibility with TAI for the delays in the design. That reflects the fact that responsibility for delivery of the project, including procurement and risk of any cost overruns, was solely and rightly for the City of Edinburgh Council. The only cause of failure attributed to the actions of ministers was the decision following the debate in Parliament on June 2007 to reposition Transport Scotland as a principal funder, as opposed to a project partner. Setting aside for a moment the fact that the Government was very clear at the time about the risks inherent with the project and that it was others who represented in this Parliament today who voted the project through, it is clear that the outcome of that vote transferred accountability to the City of Edinburgh Council and necessarily altered Transport Scotland's relationship with the project going forward. Indeed, any failure to clarify the role of Transport Scotland would have been an abdication of leadership, and it would have led to poor governance and confusion around roles. The decision to alter the governance arrangements was taken explicitly to avoid uncertainty about where leadership of the project lay, clarifying the Government's roles as principal funders and on that basis preventing further calls on the public purse. It was exactly because of that clarity and the clear setting of governance boundaries that Government funding for this project remained capped at the agreed £500 million and not a penny more. Following the parliamentary vote on the tram project, the decision to separate the roles of Transport Scotland as principal funder and City of Edinburgh Council as a project lead was in fact good governance, and it helped to avoid potential delay and increased risk. The report, as I have mentioned, identifies 24 recommendations, all of which I will address. There are 11 that are directed expressly at Scottish ministers. Of those, four refer to the establishment and delivery of public inquiries rather than to the Edinburgh tram project itself. Two are jointly for Scottish ministers in COSLA to consider a range of measures to ensure robust project delivery. One concerns record keeping and four relate to the provision of evidence and potential sanctions for providing any misleading evidence. We are working through all those recommendations, but I can report that additional guidance, similar to that that is suggested, is already in development with reference to the efficient establishment and delivery of public inquiries and has been shared with recent inquiries as they have been established. The Government is very aware of the impact of public inquiries and the importance of effectively supporting them, ensuring efficient and timely reporting. Robust and enhanced procedures regarding minute-taking and documentation management, as suggested, have already been embedded within the Government and the civil service for a long time. The permanent secretary recently appeared at the Finance and Public Administration Committee and reiterated that commitment to ensuring that robust systems and processes are in place to record and manage that critical information. Let me now turn to recommendations that involve collaboration with and working alongside COSLA and local authority partners. Effective collaboration sits at the heart of this Government, and the recently agreed Verity House agreement is testament to our commitment to embrace that collaborative approach to delivering our shared priorities for the people of Scotland. While responsibility for delivery of local authority projects must quite rightly remain with councils as project leads, I have absolutely no hesitation in championing close working with our local authority partners. The remaining 13 recommendations cover a range of areas relating to the governance and delivery of light rail projects. Whilst those recommendations are directed squarely at project leads and local authority officials, there is a link to much of the work being done by this Government and its agencies, including Transport Scotland. I am pleased to report that this Government and its public bodies, we already operate in line with those recommendations and the best practice suggested as evidenced by our excellent record of delivering major infrastructure projects, including the Borders Railway, the Edinburgh Glasgow Improvement programme, the Aberdeen Western Peripheral route and the Queensfury crossing over the fourth estuary, a complex engineering feat that has put our workmanship to the front and centre of global engineering. Furthermore, we follow detailed government guidance on procurement, on risk, on optimism bias, all as enshrined in the treasuries green book, the Scottish Government's client guide to construction projects and the Scottish public finance manual. Indeed, the identification and management of risk and adherence to best practice on business case production and the assessment rests at the heart of project and programme delivery within Transport Scotland and the wider government. Transport Scotland always follows published best practice guidance when setting up project government structures and has its own governance procedures for investment decision making, monitoring and review guidance. I emphasise that we will continue to very carefully consider each of the recommendations, noting where action has already been taken or has always been best practice, as well as crucially where we can go further. In conclusion, whilst acknowledging that we came into government with a manifesto commitment to abandon the project and to spend the £500 million of promised funding on other high-priority infrastructure programmes, once the will of Parliament was made clear, the Government endeavoured to ensure that our involvement with the project followed good governance practice at all times. It is for that reason that it was essential that we provide clarity around the roles following the vote in favour of the project, thereby providing a clear sponsorship structure that allowed us to assume the role of principal funder and ensure that public funds were monitored and grant conditions were applied in compliance with published guidance at all times. Whilst I reiterate the point that nine of the 10 lead criticisms in the report clearly lie within responsibility of City of Edinburgh Council and with TIE, I acknowledge that lessons must be learned from the report for all parties involved. I am clear that we are giving full consideration to recommendations and any actions that follow, ensuring that lessons are learned and best practice always followed for major project infrastructure. I thank all those who took the time to provide evidence to the inquiry and point again to the full co-operation of this Government. The provision of evidence that demonstrated a comprehensive, transparent and accurate view of events underpinned the approach taken to the inquiry by this Government, ministers and officials. I commend that approach to Parliament. The cabinet secretary will now take questions on the issues raised in her statement. I intend to allow around 20 minutes for that, after which we will need to move on to the next item of business. Members wish to ask a question that we have not already done so. I would encourage them to press the request-to-speak buttons. I call first Miles Briggs. I thank the cabinet secretary for advance sight of the statement. After almost 10 years and £13 million of taxpayers' money, there is very little in the statement to suggest that SNP ministers have accepted their role in the failures around the Edinburgh tram project. The former Deputy First Minister, John Swinney, is mentioned 156 times by name in the report, but not once today by the cabinet secretary in her statement. The cabinet secretary states that the only cause of failure attributed to the actions of Scottish ministers was the decision following the debate in Parliament in June 2007 to reposition Transport Scotland as a principal funder as opposed to project partner. That is the understatement of the century. Lord Hardy states, and I quote, The actions of Scottish ministers and the limitations imposed by them on the involvement of officials in 2007 was a serious error and resulted in the failure by the Scottish ministers to protect the public purse. Does she now accept that decisions to withdraw Transport Scotland was indeed a serious error and who in the Scottish Government is taking responsibility for that? Miles Briggs is absolutely right to point out that the cost of this inquiry and the time that it took to report are not satisfactory, although I have to say that these are matters over which Scottish ministers have no locus whatsoever and any interference in our part would have been rightly criticised. Questions of time, questions of cost, these are for Lord Hardy. I would caveat his comments about responsibility by being very clear that I and the Government have considered carefully all the recommendations and all the findings, but I point him again to the fact that one out of ten, the final out of ten, is attributed to Scottish ministers and a minority of recommendations are put to the Scottish Government as well. Many of those that apply to the Scottish Government were instituted years ago and Miles Briggs really ought to keep up. I want to focus on that point about that principal finding of failure to the Scottish ministers and about the separation of the roles. I narrated in my remarks that this Government came into Government 2007 not supporting the trams, but we were instructed to proceed by a vote in this Parliament led, I understand, by an Opposition amendment from Labour's Wendy Alexander. When we got that instruction to make the promised funding available, we did so, but that necessitated a separation of roles, where Transport Scotland had previously been part of project delivery into being the principal funder. I just want to quote Mr Heath of Partnerships UK, one of the only independent witnesses that commented on that point in the inquiry, and he said, I think that it was very sensible at the time. Complex projects require the simplest overall governance structure, and reporting to both City of Edinburgh Council and Transport Scotland, with inevitably different emphasis in their reporting requirements, would have been unnecessarily burdensome and introduced potential decision delay and risk. Therefore, I do not agree with the finding. I think that, first and foremost, it is important to acknowledge that, now that they have been delivered, the trams are running successfully and the feedback from the people of Edinburgh is largely positive. It is also clear that Edinburgh Council has recognised the errors that were made and have learned lessons from those errors in the successful delivery of phase 2 of the project. I would also have to say that Edinburgh Council has had the good grace to acknowledge their mistakes and apologise for something that this SNP Government seems incapable of doing. Even after the report and the statement today, I still pose the question, what lessons has the Scottish Government actually learned? The report was damning off the Scottish Government involvement or, indeed, non-involvement in the project throughout Transport Scotland's time. The question remains, is Transport Scotland fit for purpose, and does it have the capacity to provide oversight and support in the large-scale multi-partner infrastructure projects that Scotland so desperately needs moving forward? I think that Alex Rowley is right to reflect the proper running of the trams that people in Edinburgh now enjoy and the fact that the council did go on to produce successful additions to it. That probably underlines the fact that Transport Scotland's involvement was not required in those additional sections and probably undermined some of the findings from the report about our role at the time. Is Transport Scotland fit for purpose? Yes, it absolutely is. My colleague Fiona Hyslop and I have the pleasure of working with many experts every day who work exceptionally hard to deliver the running of our transport system and the development of major projects in Scotland. I was really proud, in my statement, to reel off a number of the successes that we have had in that regard. The Aberdeen western peripheral route, the Queensferry crossing, the Borders Railway, among many others. I have absolute faith in Transport Scotland. That ultimately is a historic piece of work, and while we have considered very carefully all the recommendations, so many of them are facets of the past, and the changes have been instituted years ago. The Scottish Government was clear in 2007 about the position on the Edinburgh trams and the Edinburgh airport rail link that some in this chamber also wanted to see happen at that time, which has taken even more millions away from the capital budget from other parts of the country. Can the cabinet secretary confirm that, after the Parliament voted to deliver the tram project, did the Scottish Government provide the assistance to those tasks to deliver the trams? As I have highlighted, the outcome of that vote in Parliament in 2007 was instructive. We respected it, we transferred accountability of the project in the City of Edinburgh Council, and that necessarily altered Transport Scotland's relationship with the council going forward. The decision to alter those governance arrangements was to avoid any uncertainty about where the leadership of the project lay and to ensure that the Scottish ministers' role was strictly as principal funders. I would go on to say that not only did it do that, not only was that reflected in the evidence that was the right governance decision, but it also arguably protected the public funds of the Scottish Government for being called upon in future years as the project languished. Graham Simpson, to be followed by John Mason. John Swinney is named throughout Lord Hardy's report, and most of it is criticism. Lord Hardy says this. Mr Swinney said that he would do nothing differently if doing the project again. The conclusion of what I have considered is that that would be an error. Is that a valid finding? I will open with one of the last remarks that I made in response to my last question, which was that the actions of John Swinney and the decisions that he took following the vote in Parliament, which I have to stress was not supported by the Government but by opposition in the Parliament today, that not only created the right governance structure with the separation of roles and everything that I pointed to in the comments from Mr Heath of Partnerships UK, but it also arguably ensured that when John Swinney said, £500 million and not a penny more that we were able to stick to that and the funds of the Scottish Government were protected. John Mason, to be followed by Daniel Johnson. This inquiry has taken a ridiculous amount of time and cost a ridiculous amount of money, and it does seem that some lawyers have no sense of urgency in keeping to time. I take the point that the cabinet secretary has made that they have to be independent, but surely going forward there is some way of putting a constraint on the length of time and cost of these inquiries. I do understand the sentiment behind Mr Mason's question. I agree, as I agreed with Miles Briggs, that the length of time this took, the costs that were mounted were not acceptable, frankly. However, I have to stress again that the time spent, the costs incurred, are not matters over which we have any say whatsoever it would be wrong for us to do so. Having said that, we are providing and we have already been developing guidance on public inquiries, not only for the operation of the inquiries themselves, but to support civil servants in supporting them. Of course, it is worth pointing out that the costs that can arise from public inquiries has to be one of the considerations that we make when deciding whether or not to convene one. The cabinet secretary's answer so far this afternoon has relied upon the fact that Transport Scotland had a reduced scope to that just funder. However, Lord Hardy's findings found that the Scottish Government provided inadequate oversight of that funding. Indeed, he describes the fact that the Scottish Government relied on my quote, covert influence with no record or minute keeping. Recommendation 12 outlines key improvements needed for transparency and accountability. Will the Government commit to accepting their recommendation of info and review its interactions both with Transport Scotland and delivery partners? Thank Daniel Johnson for the question. I have already pointed out that all of the recommendations have been considered and will continue to be considered by Scottish ministers, and the particular one that he mentions about record keeping and about miniting and document retention, were instituted years ago. That is capturing a period of time, which, frankly, is now very much in the past. The Scottish Government, Transport Scotland and the Scottish ministers committed ourselves to supporting the inquiry very fulsomly and in the best of faith, and that included written statements and oral statements from everybody involved. The Inveratram inquiry has to lead to good governance in urban infrastructure projects, and not to any future hesitation to invest in the public transport networks or cities in our towns need. What assurance can the cabinet secretary give that mass transit projects such as further Edinburgh tram expansion or even the proposed Glasgow metro will continue to be supported by the Government? It is apt to consider some of the learnings from the Inveratram project inquiry. As we embark on the Glasgow Clyde metro, I have mentioned a number of times that there are recommendations as to light rail projects and the engagement between Transport Scotland and local authorities. We will take all of that on board. One of the recommendations is about Transport Scotland's involvement in the delivery of projects, and I have to say that I have made quite clear this afternoon that my view that our decision to separate in 2007 was the right one, and I would like to see that continue to be instituted. On the Clyde metro specifically, since the publication of STPR2, we have a multi-partner group consisting of Transport Scotland, SPT, Glasgow City Council and Client Delivery Group to better define the scale of the work that the metro might represent and the associated governance, etc. Of course, we will take into account the findings of this inquiry as we take that work forward. Shopkeepers, business owners and residents in our nation's capital have all been denied answers for far too long by the length of time that it has taken for this inquiry to report. This inquiry is not happening in isolation either. There are other similar inquiries to which victims, particularly of Covid-19 and Professor Eljamill, will be looking to those inquiries for answers. What lessons can the Scottish Government learn from the time that this has taken and the mistakes that were made in the delivery of this inquiry? I understand the interest in the point about the time and about the costs, and I have noted that it is obviously something that the Scottish Government has to consider carefully when deciding whether to start an inquiry. However, once you have decided to institute it, once you have supported it, as in this case, to move to a statutory footing, you have to understand that, as ministers, we cannot be involved in dictating the time nor the cost. I note that one of Lord Hardy's recommendations was about only presenting the net costs of public inquiries less accommodation and less staff. I frankly do not agree with that recommendation. I think that full transparency on costs and the publication of that is in the public interest. There are serious questions to be asked about the value of the inquiry report's conclusions and whether they are sufficiently supported by the evidence. What process will the Scottish Government follow to ensure that it is able to identify the most valuable and evidence-driven recommendations in the report? As I have said, the report contains 24 recommendations and a minority of those are directed to the Government. Those that are related to the Government concern administrative processes, record management, minute-taking and some legislative aspects of setting up inquiries. We have considered all the recommendations, taking into account the length of time that has passed since the inquiry was originally set up, and the extent to which a number of those recommendations, as I have said, are already standard practice. That relates particularly to those about record management. The legislative and practical aspects of setting up inquiries and those relating to projects and programme governments. I have already said, but I will repeat for the member's sake, that we have already worked on guidance for public inquiries, and they are interacting with the Scottish Government's sponsor bodies but equally for several servants to help them to interact with inquiries. Mark Ruskell, to be followed by Sue Webber. In response to the inquiry, the charity Living Streets highlighted the accessibility challenges faced by those who are walking, wheeling and cycling and want to integrate their journeys with the tram. Are there lessons to be learned about how particular groups are engaged with and their needs are taken into account in the design and procurement processes of future transport projects? I think that there absolutely are. I would point out, just first of all for the sake of the record, that the design and the procurement were matters for the project leads, namely for City of Edinburgh Council and for Tire and not for Scottish Government, but I think that as a whole everybody has lessons to learn in the development of these matters. I would like to assure the member that suitable provision for all users, including pedestrian cyclists and wheelers, is a really important part of Scottish Government infrastructure projects. As is engagement with community and interest groups, we put that at the heart of the development of our projects. Impact assessments prepared at the early stages are evolved throughout the development of the policy and they always require engagement with those affected, and we will continue to practice that. Councils rely on the accuracy of reports to enable them to take informed decisions, but it is clear that councillors were misled by high-level officials at the council in Edinburgh. In the Edinburgh Tram Inquiry report, Lord Hardy blasts Mr Nick Smith of the City of Edinburgh Council's legal department for inaccurate reports to councillors. So can the minister lay out what action will be taken to hold those responsible to account and what reforms will be put in place to stop this happening again? I think that Mr Weber is absolutely right to highlight one of the parts of the report that stood out in particular for me. I am sure that this is something that the City of Edinburgh Council and local authorities throughout Scotland will be giving considerable thought to. In terms of what the Government can do, there were recommendations within the report made to us about the development of specific sanctions under the civil law of damages and equally a call for a criminal statutory offence. We are given really careful consideration to both of those recommendations. Our view just now is that there may already be provision for such developments under the civil law of intellectual liability and under the criminal common law of fraud, but we will continue to consider that. The time that it has taken for this inquiry to conclude, the previous SNP-led administration at the City of Edinburgh Council approved an extension of the tram lines to New Haven, which successfully opened to the public this summer on time and in budget. Edinburgh residents can take some satisfaction that lessons were clearly learnt from the first trams scheme. Does the Edinburgh trams inquiry report provide any further lessons for mass transit in Scotland that have not already been applied in the last nine years? Jackie Denbara is making a really good point. The extension of the tram line to New Haven was delivered on time and on budget, and I would add, thanks to the strong leadership and the hard work of SNP councillors. It also underlines that point about the principal finding of failure in the inquiry's view about the role of Transport Scotland and the appropriateness of separating those roles. I think that that was appropriate to separate the roles, and I think that the work to New Haven demonstrates that Transport Scotland's involvement was not required. However, as I have said to a number of members, there are a suite of recommendations, a minority of which apply to the Scottish Government. Many of them have been instituted years ago, but we will continue considering the remainder. This inquiry clearly shows that Scottish Government walked away from the major capital project. Recommendation 10, within the inquiry reports, advised Scottish ministers to consider a working group made up of Transport Scotland and COSLA to best take advantage of the necessary skills and expertise to deliver future projects on time within the budget. So can the Cabinet Secretary advise today if the Scottish Government will implement this recommendation in future large infrastructure projects? We will consider that recommendation on a case-by-case basis. I think that the recommendation itself applies only to light rail, and therefore that minimises the cases that it could apply to, but we will absolutely consider it. I must again reiterate that the inquiry heard evidence that the separation of the roles from Transport Scotland as going from a funder in principle to the principal funder was good governance. I have already quoted some independent witnesses to the inquiry. I would also point to the 2011 Audit Scotland report, which took note of that move and the separation of the roles, but made absolutely no adverse comment about it at the time. Thank you. That concludes questions on the statement point of order. Alex Rowley, you quite rudely stopped me there as I was trying to finish my question. Can you reflect on the time given for questions as opposed to the time that we spend debates in here? If we are going to hold the Government to account, we should be able to put questions and do so without being stopped. Mr Rowley, if you could resume your seat, that is not a point of order. I was not rude. I was exercising what was agreed in Buro in terms of the timings for the statement, which includes an opening question from the mean opposition party of 1 minute and 30 seconds. It allows for a minute for those from the Labour front bench. You have been in this Parliament sufficiently long, I would have said Mr Rowley, to know that. I think that you are coming pretty close to challenging the ruling of the chair. With that, there is going to be a brief pause before we move on to the next item of business to live from benches to chain.