Thank you, everyone, for attending our panel today. Usually, I find that I get these 8 AM slots, and it's really awful, so I'm very, very happy that it's a late slot. So thanks for everyone who has hung in there with us. So over the last couple years, several of us have been talking about vulnerabilities in depth, and we started down this path of saying, does anyone really care about vulnerabilities anymore? And it turned into this conversation, well, if it's a zero day, well, then it's really important, but otherwise, maybe no one would care. So we wanted to get some people here today. Got a pretty large panel of folks, so I'm going to try to keep them reined in as best as possible. And I wanted to get some people that had some good thoughts, but were really easy to manage that weren't that opinionated that wouldn't go off on rants. And if you know anyone up here, we'll see what happens, right? So yeah, fail already, thanks for that. So anyways, hopefully this will be a good panel for you. I know a lot of times panels can be a real pain, but we're going to try to make this interactive, get some good topics that maybe aren't discussed as they should be. So the first thing with a good panel is getting the right people up here. And I think that we have that. Hopefully you know all these people. If you don't, I'm not going to waste any time of the session introducing their bios. We've got Brian from OSDB, Steve from CVE, Karsten from Secunia, Art from CERT, Dan from HP Tipping Point, Katie from, I don't know who's whistling for Katie or Dan, Katie from Microsoft, and Alex, I don't know where you are anymore, so I'll just call you Alex. All right, so we're going to kick it off. We're going to try to do four questions, about 10 minutes each. And then at the end, there's going to be some free for all.  But otherwise, you can hold it to the end, your call. So the first one we're going to start off with is does anyone that doesn't work on one of these things really care about vulnerability databases, tracking, or trending anymore? And I'd like to start off with maybe Steve or Brian, you guys want to pop in? People care, but they don't know they care, near as I can tell. At least on the CVE side, we rarely get many complaints about what's going on with us. Sometimes we have like blatant errors in CVE descriptions that we never hear of from anybody. So in a sense, people don't necessarily care. It seems like a really easy job to just comb through thousands and thousands of vulnerability reports every day that most of which have one of the following four properties, which I call the four eyes. Vulnerability reports are either incomplete, they're inaccurate, they're inconsistent, especially, say, between a vendor report and a researcher report, not like we ever get those kind of inconsistencies, incomprehensible. Some of the stuff that we're dealing with comes in in broken English from people who live in the Midwest, poorly formatted advisories, where the most serious vulnerability, they don't even quite realize what it is. And it's buried in a single sentence in a three page screen about what they had for breakfast or something. That's the kind of raw information that we have to deal with on a daily basis. And unless you really deal with that kind of stuff on a daily basis, it seems like really easy. Oh, just scrape all these websites or take in all these emails and then just do a little bit of analysis in two minutes and then push this thing out. Unfortunately, there's a tendency that all of us in the vulnerability information industry have, which is we kind of care about correctness and quality. And in some ways, this is why I think we're having this panel now, because quality comes at a steep price. So you want to talk about CVE? And are we still working on CVE these days? Just get right out and ask me. Is CVE dead? No. However, we are going through a change. So we're kind of in a cocoon, and we'll kind of come out like beautiful butterflies or whatever. But there are a couple of realities. First of all, just the raw number of vulnerability reports coming out is increasing significantly. The complexity of the vulnerabilities that come out are really difficult to capture. Yeah, we could go along the lines of other people and just call things memory corruption, which is really code for some kind of buffer overflow that we don't really know how to describe exactly. But at least on the CVE side, what we've been doing is caring a lot about the kind of the academic strength aesthetics of what are the real root causes lying underneath these vulnerabilities. So terms such as memory corruption, we actually try and dig a little bit deeper. What that means, though, is that there's a lot bigger analytical overhead in our pursuit of correctness. And those of you who follow CVE on a regular basis, you may see increasing levels of precision, increasing levels of correctness, but it's come at a pretty high price, one of which is the entry to have people on our team gets a little bit high in terms of the technical skills that are required. And then the other price as well, though, is that we're at a constant level of funding and the numbers of vulnerabilities are getting reported are more vulnerabilities and more complex vulnerabilities. Something's got to give. And in this case, recently, what's been giving is the actual number of CVEs that we've been publishing. But this year, especially, we're working a lot on modifying our processes to change that ultimately. Brian? Yeah, real quick. OSVDB has two different ways to handle this. The first one we tried was crowdsourcing. So we would put memory corruption in the title and we figure, hey, there's a lot of smart people out there. There's the researcher, whoever else could clear it up. And of course, no one did. So after a while, we went with our backup plan, which is just let Secunia do it all. Any comment? I just wanted to comment on whether, does anybody actually care about vulnerability? It seems like everybody wants to talk about infrastructure, virtualization and cloud and mobile and puke. So at the end of the day, all these various infrastructures are all just delivering applications. Applications have vulnerabilities. It's the same thing getting delivered. It's still a web application. It's still a web browser.  And just because the infrastructure changes, doesn't mean the vulnerabilities really change that much. There's actually a blind spot that exists right now with respect to cloud-based services in general, which is that us as vulnerability information sources don't cover those. If there's a vulnerability in the Google search engine or something like that, maybe it can be used to hack millions of people or whatever, but that's an online service. That's not deployable software that goes into the enterprise. So while I think we've been doing a good job in general tracking trends, that is one area that's a really big blind spot. And it's going to get worse and worse as the adoption of services increases. I completely agree because if you look at the trends that have really occurred over the last five years or so, so many of the things that have actually moved are web based, whether it's browser or app. Tracking a trend like SQL injection actually becomes quite fascinating because it actually moves as opposed to some other stuff from Microsoft, for instance. With regard to the care, at least we hope that people care and that all the efforts that we make aren't in vain. At least there are some people attending today, so it's not completely CEO care. But sometimes we see ourselves as making an allergy like we're providing electricity. None of us are sitting in here now excited about their slide, but we would be complaining if there wasn't. And the same when we do the vulnerability databases, when we correlate all the information, when our advisories get out, when they're correct, when everything is as it should be, it seems like no one's really paying attention. It's not the type of thing where people say, oh, that's excellent, that's good. But if we send out an advisory with just a small typo, we within five minutes actually have a ha-ha mail sent to us. So at least with us, we spend a lot of time making sure we get it correct. Everything from the analysis of the core problem and the vulnerability down to spelling. So I think we kind of live in this, I mean, security people kind of live in this world where vulnerabilities, each vulnerability, it's a beautiful snowflake, right? Each vulnerability actually, yeah, they are, they're each one unique. But it matters a lot to us as security people and security people working inside of giant mega corporations that matters to us, each phone matters, but to the broader world and maybe to some people who call themselves security people but don't actually care, maybe they're talking APT in the cloud, whatever, puke as you said, right? Bingo. Maybe those people, they're thinking to themselves, well, phones don't pwn people, exploits pwn people. So they really only care about exploitable phones, they only care about exploits. So with all of these vulnerability tracking databases, I think for a lot of the population, what they really cull out of those databases is what's exploitable or what hasn't exploit out there as in, what do I need to get off my lazy butt and deal with right now? I love that because I think I saw two tweets on the zero Dan WordPress and all of you people keep retweeting that Metasploit four is out, so thanks, I know. Yeah. So at CERT, we had the same feelings as Steve and Karsten. We wanted to be correct and count every vulnerability in the world. Within maybe the last year or two, I've come to the realization that people do care. They do because they want to count the vulnerability, call it something, scan for it. They want to be in compliance, see if they can patch for it. They need to name it something. Sadly, they don't really care how accurate the advisory is to a large extent. If you're really gonna, you're doing something hot and at zero day and there's special mitigation advice and you gotta do something right, maybe it matters, but I think the big need is just having a label on the thing and being able to talk about that and it's the same label so that when we're all talking, you don't have to have eight different IDs. You got one ID that rules them all, which is probably should be CVE. The places where accuracy has come into play, at least what I've seen at CVE are, all this root cause analysis, I think is kind of cool and I've had kind of a mindset of, well this may help influence how people think about vulnerabilities as these darling precious snowflakes that each and every one of them is. But when we do get complaints on the CVE side of things, in generally it's two things. Either the affected versions of the software, we might be a little imprecise about and then characterizations of the severity of the issue, ultimately the CVSS score. And that kind of makes sense, right? Because that's ultimately what people seem to care about.  I don't care if this is some, you know, brand new, really, really cool attack that deserves a pony. Is it gonna hurt and you're not? This thing is a 9.8, I have to do something. Yeah. There's a lot of that mindset in the enterprise user sort of community so. Can anyone up here, does anyone know how many public vulnerabilities were disclosed in any given year? I think OSVDB might be the closest these days. Well, if you're talking about just the overall numbers, one of the reasons that we say that we have an accurate numbers, because we're the only ones that abstract to the level we do, where every single specific vulnerability in every script gets its own ID, and everyone else across the board, CVE, Secunia, everyone says no, we're gonna lump them together. And yeah, that kind of moves into some of the stats. Them's fighting words. Yeah. Yeah. There are lots of different ways of, there are lots of different ways of counting vulnerabilities. I'm not gonna dispute that you guys have a lot better coverage than any of the other sources up here, because you guys really try and track everything, and you guys put in really long hours into it. Also, your analytical overhead in general is very minimal. Your work goes into let's compose a title, and you have fairly simple ways of breaking things down that get a little bit more complex, at least for CVE. When we're dealing with, for example, shared code bases, and we're looking at two different bugs, we may intentionally combine them. In other cases, we kinda have to look a little bit deeper to figure out if we need to split them or not, because in some cases, if we have two CVEs that are out there that are duplicates, that's kinda okay, but if we have like one, I hate duplicates, by the way, don't get me wrong, but if we have one CVE out there that kinda combines multiple issues inadvertently, then the utility of the CVE goes down if people, vendors, are only fixing like one part of it, and not the other part. So that said, the way that you guys have structured things, I think is really good, because you can be the closest to counting the total number of vulnerabilities that are disclosed, using the way that you count things. Well, on that note, let's move to the next one, which talks about the trends and the metrics. This is just a natural flow, and plus, you know, it was average 8,000 a year. Yeah, and my point being, who cares if you can't even put a measurement on the number, or how bad are vulnerabilities? I mean, we're just making stuff up, which... I'm gonna cover that. All right. Is that the next question? Yeah, yeah. So what are some trends and vulnerabilities, disclosures, types, volume that we're seeing, and then are these security metrics even worth a damn? So what, if you count this many, what does that really mean? So... So the first question is, who out there thinks that vulnerability statistics are helpful and useful, and you actually do currently use them in any capacity? Anyone? That's a lot of people that are wrong. I'm sorry.  Does this slide help you? I can't see the slide. Yeah, we can't see the slide. Oh yeah, my panel doesn't know the slide. It's the vulnerability counts for most VDB. Okay. One of the things I wanted to bring up about vulnerability stats, just as a quick idea, there were 8,337 vulnerabilities in 2010. Does that sound like a useful statistic? Only compared year over year. It's over time that I think it does become useful. Okay, well, I mean, the media will call up and ask us, how many volumes were there last year? As if they knew what was the previous year. Right, well, they will get to that, and I will too. So I have to footnote, well, that's according to OSVDB. What about other VDBs? Well, there were 3,648 according to Secunia. Well, wait, why the discrepancy? Well, now you have to get into different kinds of databases. Secunia's database is geared for a very specific use. They have an entire customer base that actually uses their database for day-to-day patching, notification. You know, it's an entirely different system than OSVDB where we're looking for long-term stats, history, and we abstract the way we do. Yeah, like our customers, they would like flock me publicly if we started doing what OSVDB do and send out one advisory for every single issue. What they care about and how people, like the idea of how to use our database is how many actions do I have to take? So if there are 10 vulnerabilities being fixed by one patch, then they just want one advisory, listing the 10 vulnerabilities and the patch. They wanna know which product is affected. How many vulnerabilities? How critical are they? How do I fix it? And then there's that subset of people that actually care about the core details of each vulnerability, but they just wanna know how do we fix it? So they don't wanna have 10 advisories that tell them to install the same patch. Right, and that's why it goes back to that is that OSVDB would be horrible. It would be useless in that situation because it would saturate them. So after that, the next question is that figure more or less than 2009. And so I say there's 7,678 vulnerabilities in 2009, less than 2010's total according to OSVDB. Next question we obviously get is, is 2011 on par with the last? Well, there's 3,427 as of July 25th. This does not appear to be on par. So now all of a sudden we have one number that we start out with that as soon as you start putting context around it and you start even looking one year either direction, the stat starts to lose some of its meaning. There's also an assumption in those stats that your analytical capabilities are keeping on par with the publication sources that you're monitoring. So for example, a couple years ago, I think you guys started scraping almost the bottom of the barrel looking on various sites that no one else was looking at. And that affected your numbers for that year and then the year future. For us on the CVE, people do a lot of CVE-based analysis counting the number of vulnerabilities without recognizing that we don't have the complete coverage that we used to have due to some of the factors that I talked about earlier. Not to be a pedantic ass, but the stats don't lose the meaning. Stats are just numbers. What loses meaning, your problem with meaning is actually the model. You're inferring meaning in what you're doing there. Okay, so it's not the numbers, it's your quest for knowledge out of the numbers. Right, well it's interpretation of that and that's why I say it's all about context. Well, right. Right, exactly. And I think that's what Dan, I was just gonna say, Dan Guido's been pointing out lately, of course, which vulnerabilities do you actually need to worry about. I think where the context and the numbers actually mean something is when you then pair them with attack data. And then you can see are the trends similar, completely different. Yeah, ActiveX disclosure is going nuts and everybody's getting owned with ActiveX. Same thing with SQL injection, but not the same thing with pick whatever vulnerability type. So that's where I think it actually, you get more of a whole picture because people actually wanna know how am I getting owned? I don't care if there's a vulnerability out there that attackers don't use. Yeah, in a sense, in a sense it's too bad that the media can't ask you smart questions, but that doesn't make the numbers any less useless. Well the funny part is that they ask these questions or like, is there anything else you wanna contribute to this? And next morning they wake up and they have a 17 page mail from me explaining all this and they're like, okay, thanks for your time.  You gotta write smaller messages. This is the case of looking for the, it's the case of looking for the keys under the streetlight instead of where you think you kinda dropped them, right? This is the only data that's out there so people are looking for it. So real quick to jump back to my example, if 2011 isn't on par with 2010, the question is why? So all of us, we know some of the reasons. There's trends, look at phases like cross-site scripting, SQLI. There's certain years where people jumped on the bandwagon. There was the DLL injection on Windows platform where everyone was finding software that did that. A couple years ago with all the image ones. Image ones, yeah. And then the zip ones. So that's low hanging fruit that'll swing the totals. There's change in desires to disclose. A while back everyone was like, well shit, if I release an advisory that becomes free advertising for my company. And eventually these companies realize, wait, we're not getting business. And then researchers are like, oh, ZDI, they'll give me all kinds of cash and hookers and blow for this, you know? So all of a sudden they have a very different desire to disclose in the way they do it. There's the volumes we know about, the time we had to dig into it. Like Steve said, there was a few years where I was a consultant and I didn't work a whole lot, enough to live and I spent all my other time on OSVDB. And those years, our numbers jumped dramatically because I was scraping change logs. I would go through like the Apache bug tracker.  And yeah, I was the dumbass that actually went and said, okay, I'm gonna search for the word security and start reading every goddamn ticket Apache's ever written with the word security in it.  You still have to make a guess, right? Because half the time it's like six words. Right, some of them were. Fixed permission problem, what does that mean? Does that imply security issues or usability issues? Yeah, so not only did we have to write the entry, then it was a techno disclaimer. It says, due to the vague wording of this, we're not sure if it's a security issue. You know, you add this up and yeah, all of a sudden the numbers jump.  Who in here reads the security metrics mail list? Anyone? Alison.    What do you mean, we have to define vulnerability now?  And I say, well, I've got an idea what one is. And then he says, well, there's an infinite amount of vulnerabilities. These stats don't mean anything. I said, well, it's not infinite. And I gave him an example. I was like, I have a 10 line program. There is a finite amount of vulnerabilities in this. He says, no, there's an infinite amount. I was like, 10 lines, it's not infinite dude, trust me on that. And so he will sit there and argue and this is just one example. And one way or another, they will figure out a way to make all the stats useless. And you're wondering kind of, what's the purpose of this list again? Is it to get metrics or is it just to kind of like, have a civilized flame war? So long story short, we take all these factors in and we come to the conclusion of what I think Jack Daniel said, that that original stat I gave you is about as meaningful as my cat weighs 134 miles per hour. Without context, these stats mean nothing. Metrics aren't very helpful.  You don't. Come on, Alex. This is the perfect time for you. How many of you run all that software? No one. There's also a broad factor. Again, I don't understand, Brian, why are you getting spun up? Because the numbers are just the numbers.  So propose a model.  And that's what we call scientific method.  One of the problems is also that people need to be aware of what they can interpret out of a given number. Like, we have a lot of those cases like, oh, there's 10 vulnerabilities in product A, 20 vulnerabilities in product B, which one is the safest product? Yeah. And then they even take the stats perhaps even from our side. And that's one of the problems we have with VDB.  Like, oh, they might even add because it's on the Sikunya side. Sikunya says, this is more vulnerable than this night. No, we don't. We just tell you there's 10 vulnerabilities in that product, there's 20 in that product. If you wanna start evaluating more, what if I add, for instance, that in the product with 10 vulnerabilities, they're all unpatched. The product with 20 vulnerabilities, they were all patched within a week. So if we factor in time to patch, which one is then the most safest product? Some of you may have changed your mind now about which one it is. If I then go and add the one that has 10 vulnerabilities, they were all classic stack-based buffer overflows. The one that had 20 issues, they were more complex use after freeze. Which one is now the safest product? Which one would you prefer to use? Yeah, I think that's a really good point, especially from a big vendor perspective in terms of if we've actually put in the due diligence to, when we get a vulnerability report, we've actually put in the due diligence to look for variants of that, and we fix all of those too. For it to be bucketed as something like, oh, well, they just fixed, they had more. They had more volums. But that's not differentiating between vendors who actually do due diligence and find additional variants or additional vectors, whatever, with those who just do the lazy thing and patch one vector incompletely, whatever, and then it shows up in these counts as lazy vendor only had this one.  So there's no real way to differentiate the lazy from the diligent in this model. Real quick, in case anyone's curious, we're talking about Adobe. I was gonna say, we see it all the time through ZDI. The researchers are actually quite good about testing the vulnerabilities. And I can't tell you how many times they come back, oh yeah, it still works. They didn't patch it, or there's another vector, whatever the case might be, happens all the time. I do wanna hear Katie about silent patching. Hold on, hold on, hold on. I have one more, good mouthful of beer. I wanna claim that metrics are totally fine if you understand it, and it's your context, and you wrote the metric. Sturt has this awesome metric that if anyone knows, we still publish vulnerabilities once in a while. It goes from zero to 180, two decimal points of precision. So you can tell which ball is more important because there's a number and you can sort them. Which is totally worthless to everybody in the world, except the people at Sturt, and actually it only was worthwhile to us years ago when we used it to decide whether or not to publish document A or document B. It was very worthwhile for that purpose at that time, and that's it. So it is context. Right, my point is that you have that one line vulnerability and then you have a 87 line disclaimer rider saying this is what it really means. It's subjective. I mean, a lot of this stuff is. So that's kind of the trick.  I might care about Secunia's vols or Microsoft. I might not care about all the PHP includes you guys have. Maybe I do. Maybe I'm a PHP web app developer and I'm... You're sick. That's my point though. That's how people are getting owned. PHP. Then their metrics are bad because they're not telling them that PHP matters at the moment. Yeah, that's kind of it. Everybody always asks about Oracle vulnerabilities. Unless you're Lichfield, nobody ever cared. All right, unless you're a pen tester pointing out how broken every deployment of Oracle ever is, nobody cares. But people get owned with cross-site and, you know, everybody's PHP blog was getting owned left and right four or five years ago. And so, still are, right, exactly. And so that's my point. I like the way OSVDB does it. Because I actually know what is the attack surface available to attackers. I think that's important. That's entirely dependent on who the researchers are who are concentrating on things and what they're concentrating on. We, if those of us who've been in this industry for since about 2005 or something like that, remember a Latvian teenager, age 14 or 15, who basically decided to spend 10 minutes testing all the software that he could download and came up with 800 vulnerabilities within the course of like six weeks or something like that. And just a couple years ago, some guy for Debian basically used a super powerful vulnerability detection tool called G-R-E-P. I'm not sure what the acronym stands for. And he found like 500 vulnerabilities or something like that, right? And so we are still very much subject to pretty much the whims and the fads that researchers happen to go through.  What were your four eyes again? The four eyes? Incomplete, inaccurate, inconsistent, and incomprehensible. And you need a fifth, ignorant. All right, we're gonna move on. What are your thoughts of the value of vulnerabilities? Bug bounty programs, vulnerability, buying and selling, impact on disclosure? And we're gonna give it over to Katie and you can give her your little spiel and we'll see what Dan has to say. Then I have a question for Katie. Oh, good. Go back to silent patching. I believe the moderator has asked me a question. So the question about bug bounties and that type of thing. So I don't know, how many of you guys saw or heard about the talk that I gave a couple days ago at Black Hat? Okay, all right. Well, I'll just, I'll fill you in as I go. So I think that a lot of security researchers have varying motivations for what they do. It's not all money.  Raise your hand. Liars, come on, there's more of you. Anyway, if you wanted it, if money was it, right? There's a lot of unsavory ways that people with the dark arts know how to get money. Now, what folks like tipping point do is what we would consider the white market of vulnerability buying and it doesn't really, the numbers don't really end up equaling anything close to what the gray and the black market will pay for, right? So there's a lot of researchers out there who think that it's important to get recognition for either publicly or among their peers. So when we looked at the possibility of doing some sort of a bounty program, a nominal fee for vulnerabilities, we looked at the motivations that were out there and we looked at the motivations for the researchers who are actually finding loans in our products because not every vendor has the same profile of the researcher that looks at their code. We're a pretty popular target for research, right? Partner, right? We're, yeah. But we're a pretty popular target for research. Other vendors might have different behaviors and different main motivators for the researchers who look at their products. So we looked at what researchers do with our, why they do what they do with ours. And what we found was this past year, we've had about 80% of our vulnerabilities that were disclosed at all were actually privately reported to us. So 80% were privately reported, you know, gave us time to fix the issues and the other 20% were dropped to zero day. Now in that 80%, considering there are programs like, you know, Dan's over at ZDI that would offer a comparable price to a bug bounty like should we had decided to do so. In that 80%, 90% of those reports actually came directly to us. So even though they could have made a small amount of money, you know, they actually, the majority of the researchers who find molds in our products and wanna give them to us to get fixed actually prefer to come directly to us. So that's what we found when we took a look at that data.  And I think there's some great programs like ZDI that are out there that, you know, we love, we actually talk about quality of reports, actually the quality of reports that come from these guys is really, really good. So, yeah, no, no problem. But, you know, thank you, but, but hold on. I was gonna agree with that. Agree with me later, hold on. So, but, so that's what we found when we looked at our data, when we looked at our researchers, right? So instead of doing a bug bounty, because it seemed like, you know, there's lots of ways for researchers to make that money, we decided to do something different and that's what I talked about a couple days ago. So if you go to www.bluehatprize.com and take a look, we decided to offer over $250,000 in cash and prizes for mitigation research. So we're looking for the next generation of platform mitigations. Top prize gets $200,000. So we're gonna announce that the winner is next year at Black Hat and the contest is already kicked off.  But that's what we decided was, you know, sort of the best way for us to encourage the research community to do what it does but figure out, you know, ways to mitigate exploitation because like I said, you know, volums don't pwn people, exploits do and we wanted to encourage the research community to work with us like that. Actually, I wanna disagree because I think Microsoft is also setting a great precedent that they are rewarding not only badass exploits but the ones that are completely weaponized. So, your bug bounty does exist and it exists in the sense that I write an exploit, it becomes really good, it owns 200,000 machines and becomes part of a botnet. Now you guys offer a reward for information on the botnet, so my motivation is now not just to write a Microsoft exploit but to write a badass one. That's actually a different thing. Yeah, you're thinking about the other reward that we have that has nothing to do with it, yeah. But yeah, in essence, you were still offering money on what is fundamentally a very good working exploit against Windows systems. Nope, that's not it. So, I think you're thinking about the Rustock but botnet bounty, that's a completely different thing. So that's a quarter million dollar bounty for info that leads to the incarceration of the people who wrote Rustock. Totally different. So what I'm talking about is, this is a, I just announced this like two days ago, I'm sorry you weren't looking, but anyway, listen. So this is completely different. So we're taking this approach where, look, there are open problems in modern exploitation that breaks our platform mitigations, things that break ASLR in depth, right? So return oriented programming, jet spray, that kind of thing, there are open problems there that we're working on mitigating. So what we're actually rewarding are, take one of those open problems, right? And these are for memory corruption vulnerabilities. Yes, I know I said it and you don't like it. But anyway, take one of those open problems in the exploitation of memory corruption vulnerabilities and come up with a novel mitigation. So basically next generation ASLR, next generation depth, that kind of thing, you know, SEHOP, that type of research is what we're looking at. And just, just, well, so. He's asking, will the research be made public? So the question is, will the research be made public so that it can be used in other platforms? The answer is, it is up to the inventor. The inventor retains IP ownership of that research.  They wanna port it to Linux, go for it, my friend. Enjoy, you know what I mean? So yes, if the researcher who wins chooses to make it public, they can do so. They own the IP 100%. From a vulnerability point of view, we can see, though, that to an extent, bug bounties do matter and they do motivate people. I'd made a nice slide and he killed it. So now I'm just gonna describe it like this. I made a case, for instance, with the CA BrightStore. Had a fantastic track record, 2004, five, six onwards. There were like 80 vulnerabilities being reported in one of the BrightStore solutions, laptops and desktops, I think it was called, in 2007 and it actually triggered us. And that was the time where CDI was paying for CA BrightStore issues and a lot of them actually came via CDI. And in the beginning of 2008, as part of our yearly report, we actually went out and said CA BrightStore is a solution we consider to be inherently insecure. Not only because of the vulnerabilities, because we already talked about, we can't look at that alone, but we also found a lot of those vulnerabilities, my research team, and we could just see the code was terrible. So we went out and said we consider this product inherently insecure. A while later, CDI came up, backed it up, and also stated that they would no longer pay for vulnerabilities in BrightStore. After that, how many vulnerabilities have been reported? So either they magically suddenly just opt the quality of their product, or people just stop giving it to them and found other places. And I think Adobe Shockwave is an interesting one of us because that has certainly received a lot of attention lately also. And if I can understand correctly, you don't pay for Shockwave anymore either? Well, we had a presentation at Kansak West where we showed everybody how broken it was, and so after that, yeah. So, and I was also finding a lot of those Shockwave issues, and they have some problems in some of their components. So, and it's quite realistic to also expect that since CDI won't pay for Shockwave vulnerabilities anymore, that we will likely see a drop in it because then people will find another target where they can get money. So to a certain extent, it definitely does motivate people to in choosing which target they wanna go for. And this is one of the kind of metrics that's much more informative about the relative security of a software package than counting the raw number of vulnerabilities that have been disclosed. I can make this quite short. I even questioned CDI when it came out. If you go back to 2005, this room probably would have looked a lot different. The whole industry was different. The number of reverse engineers and researchers on the planet was far fewer, but it was a very naive position to think that that number was not gonna grow, that a black market was not going to spring up. And if any of you have ever read Freakonomics, that it pretty much proves that people, there's very good positive response when then there is monetary reward. And I think now as CDI is quite proven year over year, it's more and more popular. It's, you know, I think we do a good job, you know, being responsible and being popular with both vendors and researchers. But it's, you know, if you look at everybody that's got their own volume programs now, I think it's been proven that it's a model that works. Well, and for us, you know, the model that we chose for the Blue Hat Prize was something where we were looking at, as a platform provider, we were looking at ways to scale such that we were essentially blocking entire classes of vulnerabilities with some of the research that we hope to get out of this.  And quite frankly, we got ASLR in depth from the community.  So absolutely, I think the model that, you know, that we've chosen, and I think there's room for lots of models here. You know, every vendor is not the same. Not every vendor is a platform provider. You know what I mean? So for other vendors, other models might make sense, but for us, you know, it makes sense to try and make these changes that won't, that not only will impact our platform and our applications that run on it, but these are platform level mitigations that will also help third-party applications on our platforms and mitigate some of those issues. So for us, we're looking at this, you know, in terms of sweeping, you know, or making much more difficult to exploit entire classes of vulnerabilities. And I think this is a reflection of a growing trend in the area to move a bit more towards not only defense, like you're talking about with the Blue Hat Prize, but also prevention in the first place, right? There are entire classes of vulnerabilities. We know about these. In the common weakness enumeration, we document them, but we still have like 800 different CWI IDs, maybe 20 different ones for stuff that are related to buffer and memory corruption errors. I'm excited myself, sorry. I want to get on to the next question. Come on. This is a good one. I want you guys to talk about being the people that track and deal with researchers as well as vendors. Name names, tell us who they are. How do you really feel about working with certain researchers and vendors? And I know you guys are gonna be, you're not gonna be shy about this, so who wants to talk about the research quality and vendor response? Okay, so. Ryan, should you go first or last on this? Yeah, so I've had a few problems with researchers and I think I'm the only one out of any of us up here that will actually reply to bug track and full disclosure and call them out on it.  If Microsoft gets a report and I know that they've gotten probably hundreds, if not thousands of these where there's enough information and they're like, wow, this sounds like it may actually be an issue, but the technical information isn't there and then all of a sudden they're in this email back and forth and they spend two weeks all to figure out that, well, oh, wait, you have to have local admin privileges to do this, you know? So, you know, one of my, to name names, you know, one of the most recent ones for me was HT Bridge and I'm sure that one or two of you are in the audience. Hi, I'll respond to your mail from three weeks ago when I get home. You know, they started releasing advisories and it's obvious they're using them as a way to promote their company and there's all kinds of really crappy stuff that they're releasing because they're going after beta products, they're going after real low-hanging fruit, you know, they'll find, yeah, well, no, not only that, but they'll find like, oh, here's two cross-site scripting and two different advisories. Oh, and we forgot, or not forgot, we just kind of missed the remote code execution, you know, and the serious bugs in it. And I don't know how many cross-site scripting issues I've seen reported that our error messages clearly indicate RFI or... Or SQL injection, yeah. And they're missing these left and right and you're looking at it like, you know, if you guys would actually spend some time on this, you would find some really neat stuff and you're not. And then they also have this habit of, you know, as an example, it's like, oh, we're gonna contact the vendor and we're gonna give them two weeks and the fact that we type out on the email and the vendor never got it, doesn't matter, you know, we're gonna go and release in two weeks anyway. Bottom line is if you're discovering cross-site volums, nobody thinks you're cool. Yeah, yeah. Cross-site scripting is really old. It's really kind of lame and it's one of those that... Ask John Oberhide. Yeah, well, if you're gonna do cross-site scripting, just wait every 30 days and do one post with like all 750 of them. Okay, so if you can own a mobile phone at Pone to Own, then your cross-site is worth a cramp. Otherwise, disclose it to the vendor or the website or wherever the hell, ask for some swag and be done with it. Right, and I'm fine with posting it to the list. It's just don't think that it's anything other than, you know, a novelty for most of these. And the other big pet peeve is like SQL injection. It's like, well, here's cross-site scripting and they will actually include the script code to exploit it and you're like, okay, well, this is valid. And then when it comes to SQL injection, they're like, and the proof of concept is bracket SQLI bracket. Wait a minute, that's not proof of concept. That's saying, here's the script and here's the variable and wait a minute, why couldn't they actually put SQLI exploit code in there?  You know, either way, it's a cop-out and yeah, it gets really tiresome and I want to be clear that HT Bridge has kind of been my whipping boy for the past year but that's just the tip of the iceberg. You know, if I actually spent time to respond to all of these Liam advisories, it would be more than a full-time job. I gave up responding years ago just because of the amount of time that I took to do that. Right, so we spend time responding but it's to our researchers. We don't do it publicly. We do that, you know, we accept about 30% of what is submitted to ZDI. A lot of that is vulnerabilities that we're not necessarily interested in. A lot of that is crappy submissions and we want to work with the community and we've seen researchers come up through the years to make those submissions better.   HP. And that's absolutely correct. And so it's actually been phenomenal for HP because everyone decided, you know what, we're one of the culprits and we want to do this better. One of the other culprits was real networks. If you go back to last year and you see how many real network vulnerability advisories we disclosed, there were a lot and they took that policy change very seriously and look at how much better their software is. So yes, they were bad, but now they're good. So that's positive. We generally experience that like in the past 10 years I've been involved with VDB. I actually all think that researchers are getting better. They are getting better at providing the details we need. Don't get me wrong, we're still killing about 25% of what is posted on the list. But the level of quality seems to be improving. Now Katie has been baiting Steve and I for a while so let's go back to the memory corruption issue. That is one trend that so is going the wrong way. More and more people are using the term memory corruption.  There are a couple of valid cases where it's perfectly fine to case, call it memory corruption, but it's been like a thing covering everything from a stack-based buffer flow to a use after free. And we've even sometimes seen it, it's actually just a missing exception handing that just results in an application terminating. So it seems like being the standard thing. Oh, I ran a fossa, something crashed. I don't really know what it is. Memory corruption, done, sent. That's the kind of stuff we offer. And when you see the same from vendors also. And it's like, come on. I mean, the vendor should hopefully know what the core problem is. Please tell us, is it a stack-based buffer flow? Or is it an intergeal flow? Is the use after free? What is it? Don't tell us it's a memory corruption. So I'll also chime in, because obviously, I'm here representing Microsoft, a vendor, but Microsoft also, we actually do vulnerability research on third-party products. I founded Microsoft Vulnerability Research in 2008 to do this. And we started releasing advisories on third-party products for vulnerabilities we found and work with the vendors to get fixed. So we see it from both sides too. We are both the researcher and the vendor. And sometimes the coordinator will also, MSBR will step in and coordinate multi-vendor, super nasty apocalypse kind of issues. And we'll try and do our best to coordinate there. So we feel the pain from all three roles in disclosure a lot of the time. And yes, some of the researchers that we deal with are much more able to articulate their issue than others. But actually, we have seen that same trend where they do actually get better over time. And then- Art, do you see that on the cert side? We stopped paying careful attention. We stopped counting vulnerabilities. We get maybe 30 direct reports, about 30 a month, so maybe one a day. And we don't run with all of them. But probably half or more of those we go with. The only thing that really bugs us is that we get the researcher who is looking for some extra fame. And their company's not famous enough yet, but maybe if CERT has an advisory, that'll help. So they'll be on us to make sure we publish something that has their name in there. Hasn't happened a lot in the past couple of years, but that used to really annoy me. But do you think that the quality of the incoming reports to you has improved? No, it's all over the place. There are great ones. And there are horrible ones. And I can't measure enough to really say there's a trend either direction, but my gut feeling is it's about the same. And we actually see something really interesting too in that a lot of researchers are, they only come to us with one vulnerability ever. And they got lucky maybe, or they didn't like doing vulnerability research anymore. I mean, we don't really know what it is that made them come to us just one time and then disappear. I think a lot of times it's pin testing, same thing. Or accidental discovery.  I think a lot of researchers don't look for variance either. I mean, that was a major pain when PHP application vulnerabilities first started happening. You'd have one researcher go, oh, I looked at this PHP golf application with 10 downloads in its entire history. And I found this cross-site scripting in these 10 different vectors. And then 10 different parameters or something like that. And then like two days later, some other person with completely different reports, 22 different vectors for the same vulnerability type. And there's a little bit overlap, but not all that overlap. And it makes it very clear that the depth of the research is not necessarily there. Yeah, and one of the last things I wanna say about, Microsoft and the fact that we are in all three roles of disclosure or vulnerability research, both the finding, coordinating, and the fixing side. But as finders, when we go to different vendors, we've had to actually prove it just like any other researcher. We've had to prove it to them sometimes by popping calc. This has definitely happened in the course of Microsoft vulnerability research where a vendor just didn't believe us. So we had to show them. So, but part of that mission for us is actually education for them, right? It's just like any other researcher. It's education like no, really, this is exploitable, I promise, here you go. And they're like, why is this calculator showing up on my desktop? I don't understand. And then we use that as a way to start a conversation with them about secure development because we're saying to them, look, we've taken our lumps over the years, we've learned our lessons in the following areas and we'd like to help you because you run on our platform. We'd like to help you get better because that makes our platform more secure.  We not only say that, but we also definitely try to make it so that they don't keep making the same mistakes over and over again. All right, so we're starting to get the hand signals, but I want to ask you guys. One last thing real quick. Real quick? Just as a heads up, there are multiple vulnerability databases that do this. The data is not public. When OSVDB has a data set, we will make it public, but one of the things that it's been fun tracking is what we call researcher confidence and OSVDB is actually gonna eventually track vendor confidence as well. So researcher finds 50 vulnerabilities over the year and let's say 45 are accurate. Well, that starts to give us a percentage of success rate in finding a vulnerability and at least one of the VDBs represented here, and it's not OSVDB, tracks it even beyond that. And when you start to look at these statistics, Steve Christian and I, we're looking at the data and we're like, oh, yep, we know this guy, yep, that's accurate, that's accurate, and some of these, it's like, it's amazing that some of these researchers that are well known and liked all of a sudden have a 60 or 70% success rate. How many of you know that someone has a 30 or 40% failure rate on reporting a vulnerability that it's not accurate, can't be reproduced or something else about it is wrong? So down the road, look forward to that because I think it'll be very telling not only what we deal with, but a lot of the big names that you guys recognize, it becomes neat. All right, so we're gonna be going to the next room here, he's telling me no, but I want one comment from Alex and maybe Art on what do you think about CVSS that leads us into our room? Two things that are wonderful about CVSS. All right, so I'll back up. My problem with CVSS is this, it's an attempt at formalization of something that doesn't exist. I like the ratings, there's nothing wrong with weighting and scoring and trying to figure out how smart something is, but when you start multiplying ordinal values together, you break the fundamental life that the universe works. You just can't do that. And you end up with jet engine times peanut butter equals shiny. And you're telling me that the result is shiny. The second problem with it is decimals aren't magic. They're not unicorn poop. You can't just add them really nearly and suddenly it's a ratio scale. It doesn't work that way. So the problem is that it may be right where you have a 15.4 is actually more severe than a 13.2, but when it is wrong because you're doing the wrong things with math, it will be really wrong potentially. And that's dangerous. I like it, I wish they just wouldn't multiply things. Just give me a frickin' baseball scorecard thing and let me look at it, because I can look at that and digest it myself. So there are two answers. I have two answers to that. One of them is that there's- Fuckin' you. No, that was last night. I haven't had enough to drink. That's why I'm hoarse. CVSS version three. There are some rumblings within the Special Inches Group without thinking about that. So for those of you who are stuck with CVSS version two with its warts and all, if you have any concerns, you can bring it up to me or I'll name Katie as well or Art, because we're all one way or another, kind of least indirectly involved on this sake. The other thing is to address at least some of the limitations, some of which you've alluded to, Alex. There's this thing called the Common Weakness Scoring System, which isn't at the vulnerabilities. It's at the, when you find a weakness, indication of the potential for a vulnerability. It still has multiplying ordinal values by ordinal values, but it has built into it continuous values as well for those people who are sort of the expert users. I think we need to recognize that most people who are using CVSS, right? They need a score. One way or another, all they care about is the score. They don't necessarily care about a lot of the fancy math behind it. So my hope is that for CWSS, some of our lessons learned can feed into the future of CVSS. All right, with that, thanks for your time. Appreciate it. We'll be around. Find us for beverages and thanks again. Good job, everyone.


Helo, rydw i ddim yn ffotsio tofi TV, mae'n ffc-24. Caerio mod. Roedd yn Llyfrgell Cymru. Ithafodd yn roedd yn ysgol. Ac wedi'i meddwl i'r newid. Mae'n ffrwng hyffordd. Mae'n ffrwng hyffordd. Mae'n ffrwng hyffordd, mae'n ffrwng hwnnw. Ond mae'n ffrwng hwnnw, mae'n ffrwng hwnnw i chi'n bwysig. Mae'r ffrwng hwnnw i chi'n bwysig. Cwbwyl y Llyfrif ar bethau. Fy oes, mwy oes! Ymhyng! Fy fel rhan, rhan i ddim yn oed Llyfrif ar bethau! A hwn ar gyfer hynny, Ac yna'r bobl witnesses, a ddim yn oed i'r bobl! Fy fydd yn oed i'r boblwydd Constitution ac mae'n bwysig ar gyfer Deunyddio, mae'n bwysig o creu'r bydd fwy o 30.7 milion priodd, paref dechrau ar bethau. Yn oed i'r bwysig i'r bwysig arweithio. Efallai instig ar weithio. pan yna dwi ddim yn cael ei rhan o'n gyfle. Nawr, dwi ddim yn cael ei ddweud i gael. Ond ydych chi ni'n debyg ynglynthol eich oed yn ynach? Ond ydych chi, mae'n tychu o'r cyfrifio byddwn ni ei maes yn august ledd i gael. Ond ydych chi ni'n cytuno i gael eich oed yn ynran o'n gynhyrchu. Rwy'n wedi'n ffordd o'r b tissues o maes, ond am hyn sy'n ddweud. Ynglynthol y seis, ti'n fwyllt i'r ysgwrdd, yw'r hanqufyrdd ysgrifowych neu aerynau. Ac yn senaf mae'r ymweld. Mae chi'n mwynhau. O'n fy nginod, mae'n iaith heb bywyd yn ôl. Mae'n wybwy'r bywyd yn ôl, mae'n wnaeth i'r sefydl ymddangos i Byrni. Rwy'n gweithio gael weithio'n fynd. Rwy'n gweld, rwyf wedi chi fod yn gweithio'n gweithio. So let's see how we get on at Benny. But personally I want to keep James Garner on the bench. Garner's got to be there, we've got to look at golf free going on the bench. Yeah because he can play anywhere in the backfork. No, we've got to missy. Big should be attractive. Do we throw James Garner in for Drisagay or do we keep it at Drisagay for the legs? No, keep it at Drisagay. Yeah, have we gone with this? Yeah, that's what we've gone with. That's the team. Can we make it four in a row? Gomez, Beto. Go to, the last game against Chelsea. There we go. It's lean forward time, Barry. It's lean forward time. He's a bit dark, but he's a bit dark. But remember, just do what we did last time, just stay. Oh, I'm red this time. Yeah, I know, I was doing the same. I'll do that. I've got off side track. I'm going to be sulking me, he's by these colours.  I've got the runner, I've got the runner. I'm going, I'm going. Well done. Oh no, block. That's all right. It's all right, we'll have a covered. That was me with Mick. We'll have a covered. But that was a goal. That was terrible by me, there. It was dealt with it. Win it, go win it, go win it, go win it. Do you think we can change these colours? Don't think so. I think we can do anything. Oh, off we go. Go on, there. I thought I was pretty. I'm going, I'm going, I'm going. Hold it, there, hold it. I'm going, I'm arriving. Oh, that was going to go. That was all right. Well done. Well done. It's all right. I'm going there. That's Paola. I thought that he might make me. I thought somebody would be there. We've got to switch on the run there. Back in, back in, okay? The treats. That's what we're doing better. We started to treat, we're doing that a bit now, right? We should put our wingers foot in a while. We should put our wingers foot in a while. Stand in, stand in. All too in, well done. And Charlie of all time. Mae'r adneud wedi'u util fel oeddydd. Ychydig yn ymddangos fod y ball. Mae y dda wedi'u un bla Gref.eduau mili lost ddim ddwych! Y ddigonwys! Mae mai iddo i. Gwlad! Golb i ddoch! Jo! Fawr! Rai os y gwirionedd! Fawr! Ychydig! Mwch! Mae ddwych gyda heb! Fawr! Mae wedi'u gwirionedd! Mae'n gydydd yma! Mae roi byddoch! Mae gennych yn ôl! Mae gennych! Mae gennych! Mae gennych! Mae gennych! Mae gennych! Mae gennych! Mae genuch! Mae gennych! Mae gennych! Mae gennych! Mae gennych! Yn ymddangos, ymddangos, ymddangos ar gweithio, a mae'n gweithio'n ddigon. Mae'r rhaglau Newerry. Mae'r gwaith. Rhaid i gyd o'r ddechrau'n ddechrau. Mae'r ddechrau'n ddechrau. Mae'r ddechrau. Mae'r ddechrau. Mae'r ddechrau'n ddechrau. Mae'r ddechrau. Mae'r ddod i'n gwirio. Mae'r ddechrau. Mae'r ffordd yn ffordd, mae'n gwybod. Weldon, weldon. Get there, Beto. Beto, Beto. Mae'n iach dde, lad. Weldon, na. Mae ddi. Siad o ddron o'n mynd pais ddim, well. Siad o ddim. Siad o ddim. Mae'n gwybod, Beto. Gael update. Gideitrachel. Weldon. Siad o ddron. Gideitrachel. Get over on, Anna. Weldon. Time then. All right, that'll do. That doesn't bother me now. Ffwing. What the fff. I get over it. I don't know. It's not doing me a monitor. I have to check it out. I can't see where I play that. Bad Alberbillers. That's just bad. Practising. That was good. Weldon. Mae droid. We'll be worrying. I'm gone, I'm gone. Ell Genesis eru. Thought else was controlling. We do but there's another. Oh, dyma. Iawn mewn gweld rhaniaf... Mae'n gofyn. Mae yna ni'n ystod sy'r rhan o'w meddwl. Iwn gennym, nawr, yn gŵr o'r ni, yna? Iwn gennym. Y nifer o'r gweithio gyda ni. Mae o'n gweithio gyda ni. Mae o'n gweithio'n dd抹. Mae o'n gweithio i gyÔch yn cael posted mewn gwirio, ac wnaeth yna'r gweithio o'n gweithio? Mae o'n gweithio i gweithio i gweithio. Rydym yn eu gwasanaeth o'ch gael ei. Rwy'n g Compare yn dod. Mae'r 8 yn gallu yn ysgol. Mae'n adeg o'n gwasanaeth o'r gwasanaeth. Mae'r 1 yn gwasanaeth eich ddaf. Mae'n bwysig. Mae'n effeiniad. Mae'n adeg o'r gwasanaeth eich ddaf. Mae'n adeg o'ch gwasanaeth eich ddaf. Mae eich dal yn ddech chi'n gweld o'u gwasanaeth. Dwi'n cael ei gwasanaeth roedd eich ddweud. Mae'n adeg o'ch gwasanaeth roddiol. Oho, oho, oho. Tew'n e wedi amllun. Gwellaw ar y cwmnu. Oherwydd eich sefyllfa o'r grwyng. Cyngor, cyngor. Mae gwedd. Oho, oho. Gwellaw ar y cwmnu. Mae'n i gael. Mae'n i gael drwella. Mae'n i gael Llywodraeth. Mae'n i gael drwellau. Mae gennym. Felly mae'n castell yma i gael ei ddechrau. Llywodraeth. Mae gennym. Mae'n yn ysgrifio. Mae gennym. Rhaid i amlygu! Ieithde o wonegen i hefyd i chi eistedd i chi! Rhaid i! Mae'n ddeg Rywm. Rhywbeth yn cael ei ddechrau. Ma'r wneud o'r cysylltu gyda ni. Rhywbeth y cyrt. Rhaid i chi i ddisgruniaid gyda'r cyrt. Rhaid i chi. Rhaid i chi i chi i chi i chi! Yn amlwg, nid oedd wedi cael eu bach ar an. A'i wneud hynny. Yn amlwg arall, i'n ffordd. Rwy'n ddim yn gwneud i gael y bach. Rwy'n ddim yn gweithio'n gweld i'r yrod. Yn amlwg yw'n gwneud. Yn amlwg. Yn amlwg. Yn amlwg. I'n ddim yn ddim yn cydweithio'n gwneud i'r argynnu yma. Yn amlwg. Yn amlwg. A ddau'r amser. Rydw i ddod yn fwy o'r gwaith. Dyna'r cyfrif, yn ymddech chi. Mynd i'n ddau'r gwaith. Mae'r cyfrif yn ystafell. Mae'n ddau'r cyfrif. Mae'n ddau'r cyfrif. Mae'n ddau'r cyfrif, yn ystafell, yn ystafell. Mae'n ddau'r cyfrif, yn ystafell, yn ystafell. Mae'n ddau'r cyfrif. o'r ffordd, o'n losio. Ryf yn. Rydw i ddweud. Rwy'n gwaf. Ychydig y gallwn o'r adroddau. Felly mae'n fydde i'w ddiolwg. Pass that is, Danjim is running in Beto's own back, Danjim won't need him, I've pulled it back, Beto scared it, scared it for a move. Wel don't strike off, Steve. I could have been a leg break in that. Get there, get there, get there. Back in, back in, back in. Stay within brand three, get tight. Well done. Barely fans, still showing a support to company. Yw'r dda'n gofscu. Hold it, I'm going, I'm going, well played. D'chorau, on byr y ball, Larrish. It's done well, d'chorau. Hold it, I'm arriving, I'm arriving. Am here, cuts it. Ah, I cut it all the hard way. It's all falling in. Get over, get over. Ah, ref. Did he get that touch on the ball? It wasn't on the camera. The angle's shocking. Terrible camera angle, right. Let's have a little luck here. Baby, out give all you go. It's been bold, mate. Three strikers. We've been bold, but soon a lot. We've made the changes. Manjim has had a good game, but it's on it. Keep everyone fresh and give people the opportunity. But that means too much. Well done, and again, hold it, hold it, hold it. James Garnett turns out, which is the play here to watch too much. It's all his time in both. OK. I'm going, I'm going, I'm going. Well done, I'm going. And there's last one on. It's all right, give them a reducer. Now the bonus. It's all right, it's all right. Fine, it's all right. It's from Timothy.  Not there, Timothy. Not there. Timothy, that was my utter shite from me. Let's not let this slip.  No, we'll win this. That was crap from me, I'll make up for that. That was absolutely... Oh, no! You have to stop right there. It's spreading. Oh, no! Don't run apart, Barry. Don't run apart. No, don't. No, no, no. Relax. Relax, relax. Well, we can't go see two against these. OK, OK. Centre back your ball if you need it. Come on, Pat. Oh, overlap. That's a panic. That's a panic, Barry. Don't panic. I'm overthinking what I'm doing. Just get it back in. Back in. No, no, no! That's Tom and Tim. No, it'll be yellow. Oh, no. He jumped in from behind. That's a panic. That is no tactical foul. He touched the ball. Come on, a call, but Lou and Centre. Oh, no. There was no need for that, Ned. We've been at that stage. I was going, I've been booked as well. I've just been on the pitch. Go on. Don't fly in. You're telling yourself not to fly in? No, you've got to get tight there. Relax, relax, relax. Wait, wait, wait. I'll do. Tim, it's just got... Flick it on. Drop on me there, referee. Bow, ref. I'm going, I'm going to ball. I'm going to ball. I'm going, I'm going. Win that. Yes, well done. No, no, no, no, no! Relax, relax, relax, relax. Back to your ball again. What do you do? I want to carry it. I don't know what to do. He's gone, he's gone. That's always the ball. I saw it well too late. You believe it must have been blind? Oh, no. I'm getting there. No, no. Relax, relax, relax. It's your pickford. He's ended up keeping everything in again. Dobbins made the run. I'm getting in the corner. I don't know what that is. Neither one or the other. Right with one. Oh, my God. That was... I gave a goal away. You tried to then give a man away. 70% of the game was perfect. We've won. We've won it. We've won it. That is all that matters. We've got three massive points. We tried to have it and that up a bit there. And we shouldn't have done. But we've won the game. And that is all that matters. And we've not gone on the Premier League game for another week yet. We've got a quarter-final against Chelsea at home. And we've literally beat them last week at home. So I reckon we can get through this. Semi-final. Let's do it. Let's join us next time to see whether we can beat Chelsea in the Carabao quarter-final at Godderson Park. Beat it. Beat something else. See you later.


Hi there, I'm Sandy, I'm an artist and paper crafter and today I'm going to color an active volcano. I'm going to be using the Heppy Doodle brand new stamp set that is called Geology Rocks. So it has not only the volcano that I'm going to color but it has lots of gems and that sort of thing. Fun sentiments, they do tend to put some fun ones together. But I wanted to color the volcano because, well, long story but I stamped it and I wiped off the ink from the bottom line because I didn't want it to have a hard line at the bottom. I'm going to extend it and make it a whole scene on my card and if you saw the thumbnail and that's why you clicked then we are going to proceed and make this crazy volcano. I wanted to color one as soon as I saw it in their new release catalog because I live under a volcano. I live in the shadow of Mount Rainier and I don't mind living under the shadow of volcano because volcanoes are going to give you lots of notice. It's not like a storm that's going to come up and sneak up on you and not give you notice and time to get out. It's not like a tornado that's going to pop up out of nowhere. So I don't really feel too afraid of living under the shadow of the volcano because even though I live in the line of Lahar, I think that's what it's called when the paths of the rivers of lava that are going to come down the mountain. I'm not too stressed out about it because I can get out. You can see I'm extending my lava down so it's going to go kind of take over the trees and things that are down below it. I've put down a base color and then I'm just doing some short flicks to make it look like it's trees and the rest of the volcano, all that lava, I just threw colors on. I didn't get real stressed out about trying to make real shapes or anything. I'm going to add some more colors to it. But the way that a volcano pours out the lava is not going to be real regular. So I wanted kind of lumpy shapes and that sort of thing and yeah. So I'm going to throw in some darker colors now, some E19 to add some dark, rich colors to the lava and add some depth to it. And just merge that in with the reds and the oranges and the yellows and things. I'll put a little bit of darkness right underneath that top edge so that it sort of emphasizes that little bit at the very, very top. And as I was doing this, I was realizing I was going to have to put some glossy accents over this so I can have some lumpy stuff coming down. All of this beautiful, delicious lava, which I would like to imagine it being something sweet to eat. So let's picture it that way rather than picturing it to be something dangerous, it's going to take over the land underneath and destroy things. So there you go. I'll make up a story in my head to make it less of a bad thing to be coloring a volcano today. So add some some more reds to it. And I think when lava is really hot, it's more yellow. So maybe my volcano is cooling off just a little bit because it's got lots of reds and oranges in it as well. But one of the things I really wanted to color is the sky because I like really interesting skies. If you see my Instagram periodically, you'll notice I throw skies in there quite a bit. And one of my classes, the winter, winter wonderland class has a lot of not necessarily skies, but it has some of this really soft blending of light colors. And I've been really into that lately just to play around with what happens when I start layering light colors over top of each other. And I knew I wanted some of the clouds to look like they had some reflection from this orangey yellow going up into them. But it was going to be a murky sky. But I started with this B60 and it was too bright, but I figured I was going to go over it with some other colors. So throw a little bit of it in there just to get that color started. And then grabbed a YR to add a little bit more warmth to those kind of big clouds that are right where the words are. More lovely. And then add some more clouds here and there. And this is one of those when I do these skies, just so you know, a lot of people ask me how I do them when I just post them on Instagram. There's not really any way to tell you to do a sky other than just keep coloring until it works. This this is a hot mess until it's not because even I was thinking, I'm going to have to start this over. I'm going to have to have a better plan for this sky. By the time I was done, I was really happy with it. But it took a long time to get there. This was a moment when I grabbed a T4 instead of the C2 that was right next to it and didn't look. And I just started coloring. I'm like, OK, well, I guess I'm going to have nice dark clouds then. There you go. But look at those hard edges. How am I going to make those soft clouds? Well, I'm going to start lightening up the colors around them and just put lots and lots of color down. This is not for the Copic faint of heart because a lot of folks don't want to use up their marker or they don't want to re-ink things. Well, I'm at the point where I do what it takes to get the art done. I don't really stress out about the color other than I don't like to re-ink my markers. So since I have them all, I just keep changing markers. If I need to move back and forth between T's and N's and C's in order to get it done without having to get up off the sofa and re-ink, then I do. This was, of course, not shot at the sofa. But normally I try to avoid re-inking. And I just collect a whole box of markers until I have to do one big re-inking day and do the penance of that whole effort all at once. But as I was doing this, look, I'm just still struggling with trying to get the color to start working. It's just not softening. But then I started really working in this T1 a little bit more. And the further I went with it, I started seeing a few areas that were starting to work. They were starting to finally break down those edges. And I still got this light color underneath. I got this cascade of yellow and yellow orange. You can still see a little bit of it under there. You can see a little bit of that bluish-purple underneath. I can see it. But I have this really gray cast over top, which is what I picture a stormy kind of day would be when you'd have this big volcano exploding and casting a pall over the world and everything. My final touch was I wanted to add some roughness to it after I finally got it to start smoothing out. So I took some of the colorless blender on a tissue and just started tapping a little bit of it on to add just a little bit of texture to a few areas. And if you end up with spots that didn't blend well, just do this on top of them and you could often soften that out. But look how dynamic that looks. I was so excited by it. I used a dye set from MFT, which I have been using a lot. It's just a stitched rectangle and I'm using it on everything. But look at that delicious glossy accents on there. Wish you could touch that thing in real life because I made just a couple of streaks so it feels like it's a really lumpy embossed kind of thing. Anyway, thanks for joining me. Go check out the blog hop over on my blog and I will see you guys later. Have a really awesome day. Bye bye.


Why the narcissist cannot tolerate you. They cannot deal with you effectively They see you as something that is difficult to overcome you disturb and distract them You interfere with their normal arrangements You prevent them from concentrating on something Which irritates them It makes them become nervous because of worry and fear that is difficult for them to control Whenever there is a problem they require quick and immediate action and attention from you They require you to comfort them and make them feel better about themselves Because they have an inability to do that on their own They have difficulty managing their own lives But they will never tell you that when you express your individuality And you show them you're not the same They don't know how to respond to that So instead they make you out to be this person who constitutes a problem This person who is causing difficulty and is hard to deal with They will try to put you into their systematic plan or arrangement, but it really has nothing to do with you This is about them and how they choose to respond to things They don't know how to move beyond their immediate personal context To allow themselves to look at things from a different perspective They have a fixed mentality They have a hidden agenda It's their way of the highway They have to preside over you Which leaves no space for difference of expression meaning or response Because there's just no way to get through to them. They will criticize you They will blame you because everything has to be done their way They can't tolerate any other alternatives Because they've already made a decision and they expect you to go along with it They cannot share your ideas and opinions They cannot work with you as a team They cannot blend with you Narcissists lack self-awareness They're more focused on protecting their false image Which is why they can't accept the accountability They can't accept that there's things they need to change to improve They can't accept that they don't know everything Which leaves them to be unreliable You can't depend on them for anything Because if there is a difference way problem Something has to be wrong with you You must be the one who is at fault Because they hold such strong beliefs of who they are They have to be informed They have to be aware of everything They have to have access to all of the information about something Which other people don't have Because they can't accept that they really don't know anything They lack conscious knowledge of their own character and feelings And that is why they cannot tolerate you They cannot collaborate with you They cannot accept that you have your own ideas and opinions Because they lack empathy They cannot check your feelings They cannot put themselves in your shoes So they expect you to comply with their usual standards and expectations Your individuality confuses them It distracts them and impairs their confidence and concentration But they refuse to explore who you are Or your previous life and experiences They lack the desire to know or learn anything about you Unless it's something they can use to support the narrative of you They're intimidated by your power and right to act, speak and think as you want By your power to make your own choices and decisions So they will try to make you ineffective By applying an opposite force and effect Because it overwhelms them It intimidates them It makes them feel small in comparison to you So they will try to bring it down They will try to preside over you They will try to conquer you They're not going to be grateful or thankful for your distinctive qualities They're not going to recognize the worth or importance of it Because they lack self-awareness They lack conscious knowledge of their own character and feelings And there's nothing you can say dissuade them There's nothing you can do to make them see sense And it's not your job to do that You're only responsible for yourself But you do know how to allow the existence, occurrence and practice for things that you dislike or disagree with You do know how to deal effectively with difficult things Because you know and understand yourself very well And you don't have to give up on yourself Just to please someone else You can celebrate everything that makes you special and unique And if they don't like it That's their loss That's their foolish mistake That's something they are responsible for It's not something that you should worry about or get involved in Focus on peace and love Being welcome in and embracing And creating an environment where people can feel comfortable and confident to be themselves Because that type of attitude and perspective Produces good results But the narcissist will remain stuck in their ways They will remain and willing to change their way of doing it and thinking about things Which will not have positive results But you are not responsible for that That's just how they choose to be Thank you for watching I hope this video raised it with you Please like, comment, share and subscribe If you would like to donate My PayPal link is in the video description Coaching inquiries You can email me at coachingatnaqsurvivor.uk Thank you for watching and I'll talk to you soon