Thank you for everybody coming today. We have a special guest calling in from Switzerland, so I appreciate you staying up late Manuel is a postdoc at ETH Prior to that he was a PhD student at Johannes Kepler University In Austria his background is not databases.  Okay. So the reason why he got involved in testing databases is because For his own research, he was putting some data in a database and then it hit a bug and he's lost all his data And so he's like well, this is bad Let me go run some tests on databases and see if I can find bugs. So that's that's what he's done So sequel Lancer is a new project that he's been working on For a while and this is sort of the hottest thing now in like in database testing in the open source world And I I mean you'll talk about this like a lot of people are already adopting it So it's for the project only being around in a short amount of time. You've had an amazing You know an amazing response and reception about about it. So that's that's amazing at the power of databases, right? Okay, before we turn over and go to Manuel again The way we want to do this is that if you want to ask a question by all means just interrupt him But be sure to say who you are and where you're coming from so that everyone knows who everyone is Okay All right, man. You'll go boy. Thank you. Thank you for being here Okay, thank you, and also very thank you very much for this general introduction I'm really glad that you gave the opportunity to that I can present the work today So as Andy said, I've been working on finding logic bugs in database management systems together with my advisor general soon here from EDH Siri So in our work we have been focusing on finding logic bugs Which immediately poses the first questions first question. What are logic bugs? So let's assume that we have some query that we sent to the database management system here can see and that In this example and also in future examples, I've used this file a symbol to denote a boolean predicate And let's assume that we have a database here with two records for two The predicate evaluates to true Well, then we would expect that both these rows are fetched But in the case of a logic bug only one of them might be fetched Another example for logic might be that the row is mistakenly fetched or that the contents of the result set is incorrect We argued that finding logic bug logic bugs is a difficult challenge because they might go unnoticed by users And also by developers of database management systems This is unlike crash bugs for example, where a user gets immediate feedback because the process ends for example with non-zero exit code Isn't the problem already solved? So if we look at what testing processes database management systems or at least a mature database management systems like SQLite for example use it might seem so So specifically SQLite has about several hundred times as much test code as source code, which is already quite an impressive number SQLite's test cases achieve even 100% branch test coverage Even better they achieve 100% MCDC coverage, which is a coverage metric This is typically used for software development in the aviation industry. So this is also very impressive SQLite is also extensively fast for example by Google's open-source fast project and also by individual researchers And just as another example SQLite I mean does several kinds of testing like unit testing, but they even do anomaly testing where they verify That SQLite works even correct in the presence of out-of-memory error situations import output error situations or power failures Nevertheless in our work, we found over 400 bucks in popular and widely used database management systems So quite a number in SQLite, but also in other database management systems like mysql, MariaDB, Postgres, CockroachDB, TIDP and DarkDB So what are the ingredients for testing database management systems or for testing software in general? First of all, we need an effective test case In our context, this means that we want to generate a database that can stress the database management system in order to expose a bug and similarly also a meaningful query Also, we need a test oracle. A test oracle validates the queries result and basically says if it's correct or not Now for the purposes of this talk We assume that an automatically generated database and query already given We assume this because this is actually the easier part of the problem And this has also already been well researched and there are for example many random database or query generators We focus on the second part of the talk Namely we propose test oracles for finding logic bugs So what is a test oracle as I said basically given some input it determines whether the system works as expected and One test oracle would be to have a programmer who basically writes the specification Like writing down what what the expected result of unit test is but here we are interested in automatic test oracles They can automatically do this So what what could we consider do achieve this? I mean the first thing that you're probably all familiar with is fussing So we could take a random query generator like sql smith, which is perhaps the most widely used one And this query generator would send random queries to the database management system and hopefully expose a bug like a segmentation fault Where basically the test oracle is that the process exits with this error Now while sql smith and other random query generators have been very effective in finding crash bugs Unfortunately, we cannot use them to take logic bugs because they simply cannot detect these cases Now one approach to detect logic bugs, which works at first sight is to use differential testing Where basically we have a query generator that generates a sql query that is then sent to multiple Database management systems. For example, here both press my sql and sql and then we would get back three different result sets And we could compare this in order to determine Whether they are same or not if they are same then we didn't detect a bug But if they differ then at least one of the systems is affected by a bug So this approach has first been proposed by Microsoft research in about in around 2000 And the authors mentioned that differential testing has been very helpful for them. They could detect many bugs But only for the small set of common sql This is also what we and actually also other people realized. So for example Co-crouch labs, they published a blog post where they stated specifically that they cannot use both presses and oracle So both presses the database management system whose sql dialect is The closest to co-crouch to be namely the reason being that it has slightly different semantics It would be tricky to try to acquire is that execute identically on both the systems Besides fussing and differential testing also other approaches have been proposed So in terms of logic testing Perhaps the most impactful one is the one by color can others So they had the idea that they Can use solvers to tackle this problem like smt solvers and and they could find a number of Injected bugs. They could also reproduce already known bugs But they found only one new bug and here we believe that smt solvers might not have a high enough throughput to detect many bugs Um as another example that have been approached as proposed for performance testing Um, I guess a number of you heard the presentation Last or two weeks ago on a polo And this approach is effective for testing for testing for a performance bug, but it's unclear how logic bugs could be detected And for example another approach it was also to test for the cost accuracy of the query planner But overall we have to conclude That the problem of testing database management systems to find logic bugs has not yet been well addressed And this is where our work comes in. So we have been working on three new practical approaches for finding logic bugs namely pivoted query synthesis pqs non-optimizing reference engine construction no rack and ternary logic partitioning tlb So pivoted query synthesis was our first approach we found about one of her bugs using it And the core idea of this approach is that we want to generate a query for which it is ensured that it fetches a randomly selected row That uh to which we refer as the pivot row if the pivot row is the not contained the results that we have detected a bug The second approach that we proposed was non-optimizing reference engine construction The idea of this approach was to generate a query that is likely optimized by the database management system And then transform it in a way that the database management system cannot effectively optimize it thus allowing to detect Bugs specifically in the query optimizer of a database management system Is that is that thrown sequel or is that something lower? Um That's actually in sequel. Okay, so it's a completely a black box approach and actually the advantage is that While it's it's unobvious how to come up with it. It's actually quite straightforward to implement right And the last approach um is a term or the latest approach is ternary logic partitioning This has allowed us to find over 50 bugs in widely used database management systems And the idea of this approach is that we want to partition the query Into several so-called um partitioning queries each of which computes a partition Then um to combine this partition so that The same results that as the original query can be generated And this approach is actually applicable to quite test quite a number of different sequel queries And since you just ask Andy so all three approaches They work as black box approaches. They only work on the sequel level which makes them general And widely applicable to test any kind of system that supports sequel So maybe Maybe this was sequel answers. It's sequel answers the combination of all three Exactly. Yes. Beautiful. All right. Keep going. Sorry No, thanks. Please go ahead whenever you have any questions Yeah, so As as just mentioned a sequel answer implements all three approaches Maybe not all of them are as maintained But we're actually currently working on on improving the code quality and also the testing infrastructure of sequel answer and Since last tuesday or so it's actually available as open source software So feel free to check it out and report any issues that you might find also if any questions We have we also have a slack channel and so slack workspace And with that I want to Continue by introducing one of the techniques That we have been working on namely ternary logic partitioning So the idea of ternary logic partitioning is first of all based on a more general idea that we came up with And to which we refer as query partitioning The idea looks as follows. So First of all, we have a query generator that generates a random query q And to this query we refer as the original query You can also see here that in general we use this circle to denote a result set of the respective query And then we want to partition The result set of this original query by deriving multiple queries from it So each of this derive queries then computes a part of the result that can then be composed to yield a new result set and basically These results that should yield the same result If they do then everything seems to work correctly And if not then we have detected a logic bug in the system So um this queries that you derive from the original query, we will refer to them in the further as a partitioning queries And um these parts of the result data compute that we refer to them as the partitions And this a diamond symbol here Um, basically we call it the composition operator So I guess the general um approach probably sounds plausible to you, but of course, it's not obvious how to realize it Specifically the key challenge is that we want to find a valid partitioning strategy That first of all works for which we can define a composition operator and so on But also that stresses the database management system in different ways so that we can find bugs in it At the main site inside of this approach is the following So let's consider that we have some given radical which is probably randomly generated At any given row and exactly one of the following must hold Either this predicate evaluates to true either not and this predicate evaluates to true or This predicate is null evaluates to true basically meaning since a sequel is a ternary logic either five evaluates to true to false or to null And we refer to these um predicates like based on this given phi here um as the ternary predicate variants And the idea is the following now. Let's assume that we have some some given result set Then we can use this idea by filtering out rows where the predicate holds where it doesn't hold and where it average to null to partition it into three parts And that is the core inside of of our approach Now this was perhaps maybe a little bit abstract. So I want to continue by Demonstrating a bug that we found in my sequel So you can see here that first we generated two tables Both of them contain one row So here we have a zero value contained and here a minus zero And um, you can see here that in this query you want to fetch all records where these two um values evaluate to the same value And while this um predicate was expected to evaluate to true Unexpectedly it didn't and my sequel failed to fetch this row What was the data type for those columns? Um, so that's a good question I think for t1 it was float because we have minus zero Yeah, okay. Yeah, and I think for for t0 it it was int, but I'm not sure it might also have been a float. Okay, but I'm actually so um Yeah, the underlying bug here was I think uh a bug in the hash join of the of my sequel Interesting. Okay, cool So now the question is how did we derive this um that this is a bug based on tlb So we did in the following way First we generated a random query which looks like this. So basically just fetching the cross product of all values in t0 and t1 And this query is quite simple, right? So we wouldn't expect any Uh, that my sequel process is incorrectly and indeed my sequel fetched the correct result for it And based on this query we derived now the partitioning queries So you can see here that we have the free ternary-rebratic adherence here. Maybe once where this um evaluates to true Once where it evaluates to false and once where it evaluates to uh null And for this query my sequel unexpectedly didn't fetch any rows Which is unexpected because uh, we assumed that these two result sets would be the same And this allowed us to detect this bug and basically reduce it to the version that I showed on the previous slide to conveniently report it then So you automatically do that coalescing? Exactly, yes Interesting call So basically we just um randomly channeled this query and then there is a mechanic um process that can derive this query from From the original query if you have time I actually Um can show you at the end in a demo. How this is implemented because um, it only is implemented a couple of lines of code So um tlb and also some of the other approaches like norek that we proposed. They're quite easy to implement Especially since they only operate on on the sequel level Demo is really awesome So I have a question. Sorry for interruptions. I'm wondering what kind of means here you are. Yeah. I'm liang I'm a pg student here Working with andy on databases, right? So I'm wondering what kind of bugs will this cover? So clearly this would cover the bugs when you evaluate the predicates, but for example, if the query is a Aggregation query, right? You may have a bug in computing the average value or some mathematical operation on a set of rows. Will this kind of bug also solve those? Will this approach also be able to discover those kind of bugs? Yeah, that's a very good question and um Um, I will actually present still some examples for testing aggregate functions. So I would say we um, maybe like um I will come back to this later if Okay, but a very good question fix Um, yeah, so how does this approach? Uh, sorry kai kai ask a question here Uh, so my name is chi. I'm a phd student in georgia tech So my so so so so my question is it sounds sounds like this technique was basically trying to Generating because the left query and the right query are semantically equivalent, right? So they're expected to get the same result but Yeah, but but but so I so I so I have had so I have been working on these like a testing database For a while like a couple months. So so my wondering was like there are already many systems serve as the database optimizer, right? So basically a database optimizer is trying to Trying to like uh, uh transform a qq a query that's can executing fast faster, but there's a different query. So basically Have you ever tried like use a data? Uh, like a database optimizer trying to rewrite the query So basically you get like two different query and see if they can get like the different results to expose about Yes, I think that's also a very good idea. It would be possible to do um similar Like optimizations like database management systems do but we believe here that or I believe here that this would involve More work because we would need to define these individual rules for all the operators To define basically the metamorphic relations Yeah, but but to the the thing here I was thinking about was because all these Transformation rules actually have been implemented by the other systems, right? So the from the testing perspective we kind of get all these transformation rules for free But these I mean these these transformation rules are serving the purpose trying to optimize the execution For the uh for for the sequel so it might not be suitable for the testing purposes But I'm just wondering have you tried or like uh Some some something like that. Yeah, thank you. I think that might be a good idea. But like one, um One goal that we wanted to achieve is also that we wanted to have a system Um or an approach that works in general for all database management systems And we which we wouldn't need like to basically implement on uh For each query optimizer specifically to basically who can do the database management system so this works only on a sequel level and I think that's that's a Uh core advantage of this approach Because you do not need any knowledge about the underlying query optimizer You can just randomly generate queries derive the partitioning queries and then check for the result But it would be an interesting idea to to explore this further. I think Yeah, thank you. Thank you for it No, it's not the two my own horn, but we we actually generate the correct result for this query So I'm happy Okay, keep going Our database doesn't have this bug. We crash on something else. Okay So you can see here in the illustration. Um, how The previous example relates to this approach. So first of all You can see here that for this query queue We had this the simple query which simply fetches the cross product of all values Then we have this, um partitioning queries here They mean one with the non negated version of the predicate one with the negated version of the predicate and find it The is now version of it Well, and for the composition operator, I didn't mention it just now in the example But here we use the union all operator Which basically combines the records without filtering out duplicate values duplicate records Now, um I've demonstrated that the the technique is applicable to test work clauses But I received also the question before if we can test aggregate functions And the answer is yes So the approach is general in the sense that it can be applied to quite a number of different features And now I will do um demonstrate how we can find bugs in group by clauses in having clauses In distinct queries and also in aggregate functions so Are the other types of queries like say common table expressions Uh nested queries is that just because you haven't gotten that far to actually test those queries Or there's something more fundamental about your approach that it wouldn't work Um, so I guess it's just that we haven't really thought about testing them yet. So We're concentrated on the on the core patterns but I think um The idea of query prediction and also of ternary query prediction. It might be like there are several combinations also possible and also Um, I think we could test more different features Okay, cool Yeah, so, um, I already explained how we test Work losses or the correct implementation of them And for this and for the subsequent Test oracles, I want to use this kind of table format here, which is sufficient to fully describe The test oracles so you can see here that we have this original query for the work loss oracle Then we have here the partitioning queries in the second column Where we have the ternary predicate variants in the work loss And finally here we have the third column which describes the implementation of the composition operator Now here you might maybe not all be familiar And this symbol here basically notes the mild-desert addition Which combines the records but without filtering out Complicates and in sequel we can implement this using this union oracles Now, how can we test having clauses? Um, that's I would say quite a straightforward extension So this time we have the ternary predicate variants in the having clause And besides that we can also Randomly generate work clauses and group by clauses, which are simply copied copied over to the partitioning queries And the composition operators implemented in same way as for testing work clauses as well For testing distinct clauses the original query here contains a distinct clause And for the partitioning queries we can either decide to Also include it or to omit it because here the inside is that the composition operator is implemented differently You can see that we are using the set union to implement here union For group by clauses, it also looks similar. So here it's important to note that We need to fetch all columns that are denoted here in the group by clause Because also here we use the set union which filters out duplicate values Now we come to the more difficult part or also more interesting one. How do we test aggregate functions? There our test oracles are specific to the respective aggregate function that we want to test So probably one of the simplest aggregate functions to test is Are the max and min functions for these First of all the original and partitioning queries look perhaps as you would expect But here for the composition operator, I want to point out that before like in for the previous test oracles the Partition corresponds to a subset of the result And here it is an intermediate result So for the maximum value, it's basically here the maximum value that we computed for the rows And in order to derive the overall maximum value, we need to apply another Another max function Now since it gets a little bit more complex I also want to show you a concrete example the bug that we found in cockroach tp You can see that this required actually an experiment to vectorize a flag and also interleave tables And here we sent the original query to cockroach tp and it returned a single row with the value null And then we derived this partitioning queries from it So you can see the ternary-breticat variants again here Each of them computes the maximum value And then we need this another Outer application of the max function to derive the overall maximum value And this allowed us to detect this bug here in cockroach tp so like For that particular example out of curiosity like I mean you're using can you go back to last slide? Sorry Like you're using their vectorize engine, but then you're also using Like physical denormalization because you're interleaving into the parent table like That so like who like who told you to turn those things on right? Is that something you think automatically generates or is that Like that's that's like that for the custom code at the right for cockroach to say when you're going to create a table Here's different variations you can do. Yeah, that's that's a very good question. So In general the database a generator and also the query generator They are specific to each database management system that you want to test typically So basically we have at the beginning before we apply our test or because we randomly generate a database So basically some random first create table statements and then some inserts and also all the other Potential options that that are applicable in this context And only then we basically apply this this test or And all of these are specific to the database management system that we want to test And so in all the examples you showed you've shown so far There's been one column in each table like does this also work for multiple columns? Does that actually matter for what you're trying to do? Okay, that's also a good question. So It supports like we support the arbitrary number of columns But for to reproduce most test cases we We only need one It's also for actually for the majority of test cases. We also only need one table that we found So I think in in our first work. We actually analyzed this closer, but But I don't remember the exact numbers from from them Okay, and And basically also before we brought it back. We try to minimize them or reduce them as far as possible So there are some automatic tools to do this. For example, c reduce which Which has c c++ specific Passes, but it also works quite reasonably well for for c code Hi, can I ask a question here? So how many queries you need to generating for testing say croc range db to find these bugs? Have you like count like how many test cases you're generating? Yeah, that's that's also a good question. So for our evaluations, we always Develop the database management system at the same time also applied it And we did this typically over a longer period of let's say three to five months Like for for all the database management system that we tested So we cannot give you any Like statistics on this But from a personal experience is that actually after we implemented Let's say a new operator and we let sequel answer run for Let's say one minute then typically most of the bugs would already be found like within this one minute So depending on the database management system A sequel answer generates let's say about 10,000 queries a second And it seems that many of the bugs can already be detected very quickly Thank you Thank you. So just a quick follow-up on the on the question on supporting different systems So I wonder so you just said that For a different system the system developer would provide the database generator and the query generator, right? Then I'm wondering for the rules Do you have to re-implement all your rules for each system? Or can we can somehow reuse the same set of rules? But like who those rules those implementations Two different systems through like some layer in between That's that's a good question. I mean um, I think that's uh more implementation detail, but um, currently I think sequel answer is not Implemented that's a clean but actually since I have here the source code. I might as well. I'm just um I'll shortly explain this or demonstrate it. Sure. So here's for example So here's for example the Implementation of the world class oracle. You can see here that we have a common Partitioning base where here we generate for example a random select statement So here you can see that we generate a random like joins from clauses and so on And Here we're in the class again that implements the query partitioning where oracle Where we basically first set the work clause with the non-negative version the negated version of the is now version and then we get the results that is it's actually Quite reasonable to implement. It doesn't require too much effort and maybe also very quickly Um, the core component for generating random queries is random expression generation And also there we have quite a naive approach implemented. So we just select some of the applicable options for example some Unary prefix operation and so on And there we just select any of the applicable options. For example, not operate a plus a minus and so on and then we At the end have a visitor that it derives a string. So it's actually I would say reasonable from the terms of implementation effort Right, right, right, but but also I I saw this this this said duck db some prefix operators That means that for another db like cockroach db this thing would be really really implemented, right? Right, right exactly. So I guess it would make sense to try to factor this out, but um Yeah, it's it's we we haven't done this yet and it's every database management system Provides like unique operators unique options and so on which is why they are basically mostly separate Right, right, right. Definitely but it also means it like How do I say this um Like the this man because it's true in anything the scope. It's not like the fuzzer or the fuzzer just Permutes some random stuff and then Eventually you'll find something your case like if I don't write a code to cover You know part of the system then like your thing never never never explores it As you sort of want both and you want the fuzzer plus this thing Or take the schema that the fuzzer generates and somehow extract out the You know these different constructs of the operators Or just as you said like have a common denominator, you know Lowest common denominator of the sico standard across multiple systems and then build the customer stuff on top of it Yeah, so can I have like one very very very very very quick question here. Sorry. So like how how how did you randomly generating the data? Like the data Yeah, so basically the insert statements. They are also randomly generated So when you say like So when you say like each second you're generating about like 10 000 queries Are these like curious for one instance of database or for each query? You kind of like have their own instance or like what's the rate ratio here like Like number of queries compared to number of instance you generated Right. That's a good question. So um What I was referring to is when we already have a database generated So generating a database takes a little bit more time than generating the queries So typically they provide heuristics where the user can select how many Queers should be generated for each database management system So typically I think we would generate around 10 000 queries for each database that is generated or a little bit more maybe Okay, so so sorry for for for this but like so because I saw all these examples It seems like you only need like a one or two or three Or zero to both in your in any instance Have you ever find a bug that requires like a large number of tubos to trigger that bug? Yeah, that's that's a good question. So the examples that I've shown you they have been automatically reduced And uh, it's basically the original test case. We found them with more complicated like Data and also more complicated queries So they are basically reduced to the minimum to reproduce their respective back Thank you. So so if time allows actually We will see later if I can show you a demo or so to to show how how typical queries and data looks in SQL answer All right, keep going. This is good Okay, thanks so the um first, um Aggregate function that I showed you how to test namely max was a so-called self decomposer by aggregate function so I don't want to Explain now the formal underpinnings of this But I want to tell you that there is also another category of decomposable aggregate functions And this is best explained on an example so You can basically the intuition here is that if you want to test for example the average aggregate function Then it's insufficient that we in the partition compute a single value Rather we need here to keep track of the sum and the count to then derive the overall average And that's basically also allows a testing decomposable aggregate functions like by including tuples When computing the partitions There's also another category of aggregate functions namely non decomposable ones Group concat is one example Which concatenates strings and they're actually the order is important We didn't consider testing them because it would be more complicated But also because um, we would expect that few optimizations are applicable to them due to the same reasons as We have difficulties testing them. I think that's only my sego too. That's on the sego standard Ah, that that's possibly right. Yeah Yeah, so overall we evaluated this approach in I think In three months of implementing and running our tool and reporting and reducing bugs In this time, we have found our close to 200 bucks or 23 of which have been fixed um We evaluated the approach and on six database management systems actually only five here are shown because um For this latest approach, we actually didn't find any bug in in postgres Um, and you can also see here in c colite, which we comprehensively tested before we only found for new bugs What I also want to stress here is that the total number of bugs that we found reflect our testing efforts and not how like how Well, the database management system is tested or what what the quality of the database management system is So at this point, I also want to give a shout out to the bug db and the c colite developers Who have which have been especially responsive and typically fixed our bugs even before one day of reporting them So what test or across did we used to find these bugs? Well, first of all and there were oracle found most of them It's not only the simplest, but it seems also the most effective test oracle And what I also shortly want to mention here is that um in our relation We found that this oracle has some overlap with um one of the oracles that we had previously proposed For the other functions or for the other oracles we found fewer bugs, but still a number of interesting ones Um And here I also want to note that we implemented these oracles only for database management systems For which our bug finding efforts saturated using this ver test oracle And these are the ones highlighted in red because we found um that we often have problems filtering out duplicate bugs And also that for example having test oracle can find a number of bugs that also the ver oracle can find Simply because it also generates work losses Now I slowly want to start wrapping up. Um, I want to mention that uh TLB is a metamorphic testing approach, which has some implications namely a metamorphic testing approach in this context means that we have a SQL query that we execute and We get this result set and based on these two components we derive New SQL query that functions in the test case and also a test oracle that decides whether the test case executes as expected Now the fundamental limitation Of metamorphic testing approaches is that they cannot establish a ground truth So it might be that we cannot detect a number of bugs simply because the database management system Is consistent, but uh, but still Like buggy Also to shortly compare it with the other approaches that we proposed so All three approaches that we have proposed so far can test work losses, but only TLB is applicable to comprehensively test other features such as aggregate functions On the other hand, um pqs the first approach that we proposed is the only one that can establish a ground truth But this is also a reflecting Effort so for pqs the implementation effort is moderate But here for TLB as I also have demonstrated in the code. It's actually quite low and no work. It's even lower So for pqs and nipirac, it's the same thing like TLP where I would have to Write some custom Java code that generate the queries Exactly. Yes. So they're basically all based on the same table generator and also on the same query generator It's basically then just um configured to um to provide the test oracle that we define Okay, so if I if I did what you did for ductdb or take the cockroach one like Just those two class files would be enough to run all three tests Um, so actually it looks a little bit different for each um oracle that we want to support. So Um, okay, I mean you have to check me out, but I I understand. Okay. Cool. Keep going. Okay, sounds good. So, um I also want to shortly talk about the reception we have received so far so I would say the majority of database management systems that we tested we received quite positive feedback from the developers Especially the cockroach db folks have been very supportive and appreciative of our work So for example here a Jordan from cockroach labs even said that we are doing the database industry a great service Or here peter which um, which said that he is a fan of our metamorphic testing approaches As another example the ductdb developers they appreciate our testing efforts because ductdb is quite a recent database management systems And mark who also presented I think a couple of weeks ago Told us that he appreciates the work because otherwise users Would basically have a not as good as experience And also since we can provide more compact test cases to reduce the issues And as a last example for developer reception So pincap the database management system developing tidyp We actually participated in their bug finding challenge And in this bug finding challenge 22 of the bugs that we reported were classified sp1 bugs So the second highest severity level and six sp2 Which demonstrates that our bug Finding approach can find many severe bugs and actually when we redeem the points that we Obtained we can redeem them for I think one has 17 t-shirts. So did you get their t-shirts? Not yet, but let's see I can also redeem them for for who this so Maybe it won't be one has 17 t-shirts Then as Andy mentioned during the introduction Industry has started to adopt our approaches. So we have been contacted By quite a number of companies organizations and developers that I Already started implementing the approaches or which were interested in implementing them and of those pincap Already open sourced that tool which is called go see colanza and they support all free approaches in their system um Now last I also want to talk about bug importance So of course we found bugs of different severity levels. So also many Not so severe bugs, but also many severe bugs But I want to argue that even the less severe bugs matter and for this I have a concrete anecdote that I want to tell namely user of sqlite reported This following test case on the sqlite mailing list stating that this query here computed the incorrect result And another user confronted the original Bug reporter saying that you cannot do this world-class in production code. I mean you can see it's it's quite convoluted And the user defended themselves stating that The query was generated by some middleware Yeah, so automatically derived not user specified And we actually found the same underlying bug We reported to the sqlite developers. They quickly fixed it So at the time when the user reported this bug, it was actually already fixed on the latest development version And I believe that this shows that even bugs that might be considered as obscure They might affect some users, especially when queries are Generated by some co-generators Um, I guess I'm already over time or Five minutes for a demo Go for it Okay, sounds good. So basically I'm going out to the sqlite Sorry to the sqlite home page. I'm cloning the repository It should take one second Building should be quite trivial We only see the the powerpoint. I only see the powerpoint I'm just stopping the share and Can you see it at all? Yes, great That really can pick us. So we're now in the directory of sqlite answer What we only need to do now is we use a maven to build the project Which should only take also a couple of seconds here You can see that all the test cases execute correctly because we don't have any at the moment But hopefully soon And then basically we already have this char here that is generated And then we can directly launch it using Java And if you use sqlite answer without any option it will first print all the supported database management systems So here for example ductdp and also the different options that are supported And then basically we can select that we want to test ductdp And now every five seconds also some progress information should be output Namely, how many queries that we generate and also how many databases What we can also say that's saying that 50 percent of the queries are failing Successful statements 50 successful statements 50 percent. So that means oh, yeah, yes, okay So basically we have a very naive Database and career generation process. So most of the statements fail or or half as shown here What we can also quickly look at is the logs that sqlite answer produces Yeah, we're just using to you know And basically what we can see here is that first some tables are generated then some random actions are Selected. So here you can see a couple of insert statements Actually quite a number of insert statements And if we go a little bit further a bit Low you can see the Queries that are generated. So here's a original query actually quite similar to the mycicle example that demonstrated And then the partitioning queries here where we have the free ternary predicate variance Well, I guess that's um That's um The overview of how yeah, but how do I know which one failed that you have your log file has whatever? Oh, yeah, so now none of them failed because um, we cannot find any more bugs on the latest version of duck tp But um, if it detects a bug it will still fail with an error message and also explain what uh, what went wrong But sorry in your in your debug output when you were running it it was saying successful statements 50 percent Uh, so so that's that's basically it sends a query to the database management system But it might be semantically incorrect So, okay, okay. Yeah, correct ones, but but not semantically or not necessarily got it. Okay Okay, thanks. Okay, cool. Awesome. All right. Um, so again, let's we're we're virtual so we can't applause, but I'll You know, I'll I'll applause for everyone else. Uh, we have just two minutes left. Does anybody have any questions for, uh, manual Again, I'll meet yourself and say you are and we Hi, I'm I'm steven frost. Uh, hopefully you guys can hear me. I'm with Credit us. I think you mentioned earlier. Sorry. I was in the middle of something else when you asked, but um, I would be curious if you Um, manual have looked at either sql smith or if maybe you've looked at deppson Um, those are two different tools that uh get used pretty heavily to uh to test out postgres I'm also really happy to hear that postgres didn't have any bugs in your last round Right. So um sql smith, I guess is the most widely used tool for for, um, fasting and finding Crash bugs basically, but I guess our goals here are different since we want to find a logic box similar also with with jebson But I agree that these are very widely used and very useful tools as well Well jebson is interesting because it's actually like there are things you can do in terms of serializability of transactions and ensuring that the database is performed properly um, so I I wouldn't agree that it's necessarily just a fuzzer it it does go after some Some issues like that, uh, which I would be interested to hear if you thought about sql smith is a fuzzer jebson is more targeted, but but like right jebson is harder to set up Actually, steven. Have you guys tried the new one ellie? Um Don't think I've played with ellie. Well, actually I think I mean I'm trying to remember if that was one of the ones that was mentioned here because we were Yes, so jebson slash ellie. I mean is that something that one of my co-workers, uh, peter gagan was just working through Um with a kyle kinsbury, you're working with jebson slash ellie to work through a um, uh, a serialization isolation Serializable isolation issue in postgres. Yes, I'd love to see that run against other database platforms Yes, okay. All right, cool. Awesome. Thank you Panas, I think you are roger. You have a question I actually I will say this is panas chisances. You're a software great, uh, presentation very interesting tool Absolutely, actually, I was triggered more by one statement that you said about When you the other method when you compare three different databases Of their output and then you said, uh, possibly this tool would not work because of Sematic inconsistency in the language You haven't thought about using your tool in fact to recognize such inconsistencies And uh, uh compare The correctness quote-unquote in semantics of the different languages SQL languages were flavors I think it would be interesting but the the thing is also that database management systems deliberately implement, uh Inconsistent semantics. So it's not that every database management system wants to closely follow the the the SQL specification And also not everything is specified in SQL standards I think even created index essence on it. It's it's not um part of the specification if if I'm not incorrect So I guess it would be Or did I understand your question? Yeah, partially. Yes, but I was curious whether actually it's a good starting point to use that So everyone says like gdbc says that they offers you some common language so that you can collect and interpret A lot of back end databases How can you actually truly? Recognize that when you are interpreting you get the exact correct semantics Or what would be the least common denominator? That would be interesting to see You know using these methods Oh, it's just a thought that uh Occurred to me when you are talking is not a it's a suggestion rather than a Oh, yeah, that's That's that's very interesting. So basically to extract the common core between all the SQL dialects That might be unseen 92 or so I guess that's some some minimal one But it would be interesting to actually check if If uh, all of the database management system at least support this SQL dialect That's where did they divert it at some point in 90? So that will be interesting to see Okay, all right, Raju your quick question Yeah, I wanted to know like Who you are, where are you coming from? Oh, sorry. I'm Raju. I'm from uh couch base So like I'm interested in trying to find out how we can use this tool for uh our no SQL So and uh, you know, that's the main thing that I am interested in. So is there any next steps that you can suggest? um So we already used it from for one time series uh base database management system and also found bugs using it um, I guess it would be very interesting to also investigate whether we can find bugs in documented or document oriented database management systems like mongo to be also um, I think there will be a next is interesting step But I guess at the moment, um We we will still stick to To dive a little bit deeper into relational database version systems Okay, but thanks for the suggestion You when you said document databases, you said mongo to be he's from couch base. So You know, that's uh, sorry. I didn't hear Okay Okay, guys, uh, this has been awesome. This is uh, again This is this is a super exciting topic and I'm glad to see that everyone is recognizing Uh, you know what your tool is capable of doing. So I'm I'm excited to see how this grows and gets wider adoption Um, so again, thank you for manual for sitting up late with us. I realize it's 11 30 for you at night We really appreciate this


Going on everybody got Colin Sean here with late in sports cards were up in some hockey got upper deck allure 18 box full case number two Did a randomizer for the break had Ryan be with the stars down to Richard H with the blues? Good luck here guys. Let's see what we got Let's see what we got ourselves for that low man. He is incredibly good Bunch of spots came in while I was doing all that. Thank you guys Toolcase spots gilded spots to me. It's just because he's on the Astros Yeah, he did not have a good post season either man. He should have done he should have done better He is he is Alrighty guys appreciate you getting some spots in while I was typing in and everything there Let me see grab him home till later Update series is full get that going next. Thank you guys and six to go in our dual case MLB bowman chrome hobby and choice Jacob, I think you're getting catfish, man Cardo Rodriguez 18 box guys 18 boxes. Good luck doing some allure Yeah, don't worry about him don't worry about him I Lure good luck. Good luck. What do you tell me his name was Ricardo or Richard?  There's a black that one's gonna be Adam Henrique black rainbow not numbered got a color flow that one is gonna be of Piotr Kachetkov iced out insert Pius Souter Maddie Baneers and a Jonathan Huberdote Huberdote That's crazy Shane Cam Ackinson color of a Alex Vlasik like the pickle is Ilya Sorokin We've got a color flow that one is gonna be Carson Meyer. Why are the names so small? Goodness gracious Carson Meyer to 100 on the color flow. Get that right there I already can't I already can't read them. I don't know what they say Alrighty guys, there's two silvers in the front and back. We'll take out any rookies But the most part is not gonna be top-loaded Quentin byfield on the black a lot of color in here There's a braid in on trachy trachy York Strand It's a red of Leon dry-sidle red pulsar or red rainbow not numbered We've got a color flow of geez on that's your Demko And that one is crazy looking that is a mark stone. It is gold glitter bomb not numbered glitter bomb stone There's a rookies that one is gonna be Ronnie. It's hard Got a Nathan Smith on the color flow and check, please Check please now one is Pier Luke Dubois There's a rookie silver Just realized that this says the word rookies up top. There we go. Oh, I know Sean I'm a huge fan Yeah, these are not this is the one I was someone I was thinking of a minute ago when you said this is dope the glitter bomb a black rookie of Riley Walsh for the Alexis Lafrenier. Yeah, the numbers are weird on the blue-green Silver rookie there's a die-cut rookie that one is Bobby Brink Bobby Brink on the orange slice orange slice die-cut There is a Victor Loden These are really hard to read man. Matthias, Macelli Red rookie is Braden Tracy got a blue-green of Tomasino and Otto is going to be Keaton Corsac that is for the Golden Knights on the full rainbow Otto Not numbered there nice start for the Knights a red pulsar that one's gonna be Frederick Anderson red rainbow Another rainbow of Matthias, Macelli. Yeah, I don't know box two Rich should be live at 10 a.m. In his own thing in his own thing Fanatics live can find us on their late in sports cards black is Mitch Marner got a Color flow orange and yellow cousins Yeah, right thinking about the other day There's a black of Brady Kachuk and one is Sam Bennett on the color flow What up, it's going good Russ. How you doing, brother? Moog Moog going on Russ Like your profile picture straining kid emoji or meme There's a black that one's bow and Byron got a color flow that one is Marco Rossi Nice one there for the wild color green and yellow. It's a red of Brock Nelson Brock Nelson on the red rainbow This one is blue and purple Sebastian Ajo and blue tracks is gonna be a Nikita Akatuk Akatuk rookie, I believe for the devils on the blue line Yeah, Bobby That's faria Say black rookies of Nathan Smith got color flow of Jacob Movare and a quartz die cut. That's a sick one Tom Wilson Not everybody's favorite player cool card though on the quartz of Jonathan March or so There's a red and black Scott Pernovich base base base It not interested in there from the 80s and 90s for the most part. That's what's called the junk wax era We don't we don't buy or take any cards from there Orange slice of Andre Vasilevsky if they were thrown away probably are from there Color flow Jacob Glavis.  Yeah, yeah It's a good use for it Yeah, these are nice looking man. These are These are lower hockey look real nice. There's a black of Kyle Conner Got red and yellow of Morgan Baron. I just can't read it Some silvers Seven tops is dope That's fair. I'm down with that. It's a cool coffee. That's like right when I got out of it Green yellow Kendra Miller. Yeah, there's some good sets within that that junk wax era for sure, but Yeah, black is gonna be Conner Garland got a red and orange of Kent Johnson Really My mom did that one day at a someone Pokemon cards at a garage sale gave like 50 cents for like my life's work There's a Dylan Cousins red Blue and green is really Jonathan Dolan keep it going here Cheese black rookies is Declan Chisholm for the Jets orange and red a certain Corsac and a our Timmy Panarin still want those back. Oh, yeah No, the air cards are nice the the Frank Thomas the Bill Ripken there's a Nikita Akatuck Matt boldy Silvers a orange slice Jake Gensel. No one is not numbered on the die cut Another orange and red that one is Dawson Mercer some silvers Brayden Schneider Brayden Schneider red rainbow Cold Caulfield and Otto is going to be a Isaac Ratcliffe going to the Flyers on the full rainbow auto go to Massachusetts every summer really That was why was the junk waxes. They had so many boxes red Matt Duchain got green and red that one is Jesse.   Oh, yeah. Oh three Freakin a rods on the cover. Oh, Victor have been Rangers a rods on the cover red and yellow Kirby doc Those are the day Bobby ain't like that no more man. What's going on, Greg? What's going on Greg? There's a black of Nikolaj Eller Ellers, there's a red of Ryan suitor. Is there a bug? Is there a bug on Shawn? Greg? It's not me calling it that not me calling it that that's what it has been referred to check I call as the black and Yellow and green is Troy Terry Silvers I could choke him out with it. Oh back off, bro Got my Eckersley shirt on today. I can't get rid of the mustache until I'm not wearing it It's a bow and Byron on the red red rainbow got a red of Paul Caulfield and When it's another glitter bomb Robert Thomas glitter bomb Black is Dylan Holloway This one is man. These are so hard to read Lucas Dostle for the ducks and a quartz That one's gonna be Mike Smith on the insert University of Miami ACC championship baseball ring I thought I pleased this man. He's been here.  Oh, it was fake in that boldly Was he a baseball player? You just thought he had one. No one over there yet. All kinds of stuff All kinds of stuff. I've seen the ring and I really like the ring Made a trade and I thought I had them me lander orange slice Red and green is a Ryan suitor and he in fact he got you Red and green Dylan cousins and Otto is gonna be Brad Marchand nice one there for the Bruins Well, Brad Marchand a 35 on the auto to 35 Ain't still wet. I'm looking the wrong spot for numbering now that Two spots Interesting just trying to throw me off on this one man Trying to throw me off. I see how they go. It's a pretty good Chuck Red and yellow is gonna be Sebastian Ajo and that is sick to 299. We got a Jeremy Swamin on the runaways Bruins doing nice Swamin runaways to 299. I need box number. Oh geez. What is this box five box five a big case? Cards in here. I actually with the Taco Bell earlier the Taco Bell earlier Orange slices feshnikov and then Sean had some food to Alex new hook Runaways baneers Oh, yeah Always a move. This one's got a thick one in it. Kent Johnson on the red got a green and yellow Matt boldly What's up? Oh I got you and auto it's patch auto Connor Garland that one's going to the Canucks That's a sick one there five of ten Purple diamond auto patch nice one there for the Canucks sick-looking patch Little three-color logo. I don't work for Vancouver Vancouver with the patch auto little Cuban sandwich. Nothing wrong with that. Oh, no, nothing wrong with that There's a red marches. Oh Got a Isomont I go to a place down by the airport I feel you Harrison there's a Mitch Marner red Color flow was Chris LaTang and atomic to 299 that's going to be a Matt de Shane game skills Silvers are pretty much the base in this one guys a lot of silver so we're not gonna be top-load most of the silvers There's a black rookie of Braden Schneider red and yellow is Jordan Harris red and orange Jordan Harris Caprice off Bowman baseball We already did Merlin Update series is next. I probably still got 20 minutes or so on this one updates filled So probably 20 minutes behind total Phillip so only be 20 minutes behind the schedule already did the soccer spots are flying into a missing them Appreciate y'all. It's a black Cal macar got blue and green. That's a mark McLaughlin You see Phillip appreciate you buying some spots my man There's a black Robert Thomas There's a Peyton Krebs Peyton Krebs. I Still remember that it was a long time ago filled. I still remember that my home's auto at a prism That thing was a beast that Brad I think Brad pulled it for you. That's been a while now There's a black Roman Yosi Got a orange and yellow that one has been Myers We have one they shut down Zoe's yeah, we did have a Zoe's we did have a Zoe's there's a Tom Wilson black Red and orange was Ben Myers was close wasn't the greatest that's fair And one more trout super frack of that one.  I don't remember that one trout super Like that back of on scopatar got a blue and green chase Pearson Runaway's yo Rossi Marco Rossi Josh Norris black color flow of Brock Bozer and a Frederick Carlstrom on the die cut that one is a green rainbow to 99 for the stars No, I don't think he is I don't think there's no he doesn't have plans to at least Quinn Hughes on the black Got orange and yellow of Nicholas Abruze easy No Fair heck. Yeah So red by field got a red and orange of bunting and red and orange Isaac Ratcliffe Isaac Ratcliffe not or number 200. Yeah numbering is all over the place on these Joe Gilded's really nice man. Gilded's really nice a brain Tracy black Orange and yellow is gonna be a Lucas Dostal and check please Check please is Ryan Reeves magic Matt doesn't work here anymore Yeah, he does not work here anymore Yeah, I've been a few months now. Yeah, like two months probably Black is Jake Christensen got a red and orange of Anders Lee Orange slice insert is a coral strum Alexis LaFrenier We should you Jeff bind into our mixer Jeff Jeff red of Marco Rossi Marco Rossi red and orange is Jonathan Dolan and Otto is going to be a Lucas Reichel that one's going to the The black Hawks Lucas Reichel on the full rainbow auto not numbered It's one black Hawks magic Sean There's a red of Cam Atkinson Yellow and green is gonna be cold cold Caulfield nice young player there Yellow and green and the 299 got iced out Quinn Hughes nice one there canucks Realize there was three huses now. There's a one playing with Jack and In New Jersey Jack has been absolutely murdering it too, man. Jack Jack Hughes has been hot to start the season very very hot Crosby or Ovi I'm going Crosby I'm going Crosby a Thomas Hurtle Thomas Hurtle black got a orange and yellow of I cannot read these Eric Calgrin Eric Calgrin Lane you get on the phone with upper deck Tom to increase the font size on the front of these Black Carill Caprice off these ones are too small.  We got a Mark McLaughlin. He is man. He is love it Greg love it We're going Ovi Ovi top it. Oh man. Oh Man, we've lost it. There's a Lexus Lafrenier red orange and red and orange is Declan Chisholm and this one's going to be Orange and yellow of Andrew copped 85 Rangers. I'm looking for the big autos You can hit like Crosby and McKinnon McKell McCarr is in here. They've got There's Cola Chinook on the black Cola Chinook yellow and green is Tanner genauts. They also have McDavid. I believe on the autos But the autos in the patch autos for sure coming in about one per box on those black rookie Carson Meyer Red and orange is going to be a Zeck Zeck Hayes Orange slice is going to be a Kate Clayton Keller orange slice not numbered green and blue is Chase Pearson and Jack Quinn on the runaways Jack Hughes Quinn Hughes Jack Quinn No, no one's close to McDavid right now. Not even close There's a Caden Corsac on the red red and orange is going to be a Ross Colton and auto is Nathan Smith for the coyotes Center for the coyotes on the red auto that one is not numbered red rainbow auto Yelts Yeah, no, there's there is certainly not a debate. Who's the best in the NHL right now, man It is McDavid. It is McDavid Clayton Keller red got orange and yellow that one is going to be Alex You follow and glitter bomb rookie is going to be Kent Johnson for the blue jackets To 199 on the glitter bomb No, it's definitely not me dude Also goes by Chuck. Yeah, there's a black Bret Burt's Red and orange of Thomas Bordelo, he's got star calling everybody Chuck start calling Matt Chuck What me Not yet not yet Jacob not yet. He's not there's a black Matthew good Chuck He's got to put up a couple like 130 point seasons before that happens Jacob Peralts that man That man McDavid is the goat right now Sad a new burger at McDonald's Black is Sean courtier Red and purple of Tory Krug base base base Nathan McKinnon on the red red rainbow got a Matt Greslik and this one is going to be Yanni Gord Yanni Gord former Lightning To 50 nice one there for the Kraken Release the Kraken. It's a good question. I feel like Greg does Greg probably does Alrighty guys next when we have his update series. It is sold out. Perfect.  I Think Sean's trying to get a new a golf partner because Matt wasn't very good today Getting who's nets off yellow and green is Bobby Brink Bobby Brink Not me not me me. No, no, no, it's an orange slice is a lot for me a Got a red and orange of March on and runaways bunting. Yeah. No, that's not bad That's not bad. Shipee is a beast to man. He's been having a good postseason. It's a Maddie Baneers nice rookie there for the Kraken a Blue and purple that one's Kirby doc and auto is going to be chase Pearson for the red wings on the full rainbow auto He's got some nasty stuff.  He's good at that He has the gift Yanni Gordon the yellow and green that's dope Greg. He is very good at calling stuff man He's super knowledgeable as as you know as you know He is ridiculously knowledgeable in sports. That's a son. Oh, yeah, Jasper Brett. Oh, yeah, Jake. There's a red and orange Charlie Coil That's awesome, man. That will do Have to check them out once he's a once he starts doing that What's your furthest drive? Oh boy? Oh no three three something Jacob Markstrom Shawn does pipe that thing man orange and yellow Carson Meyer Or so why didn't you tell me there were so many cards in here there's a black rookie of Mad so guard mad so guard red and orange is a Braden Schneider That's dope Jacob Peralt Did the corn fairy tour nice nice. Yeah, my friend a rider. He actually came up through there the corn fairy and the You should be called the web.com. I think Andre's fetching a cop on the red red and orange is for the blue jackets and to 299 got runaways Michael Bunting Michael Bunting for the Maple Leafs 388 down Ellen 50 mile per hour wins. Let's go shite. We'll count it. Yeah, we'll count it Is a Matt boldy on the rookies nice black boldy Terry these guys would never hear the end of course Matt's Urslic Yeah, we don't have the we don't have quite the elevation around here to do to do that kind of thing But we have the wind Orange orange slice is Vladislav Kolacinik Of course it is orange and yellow is Ryan suitor That is almost 240 that's not bad. I ain't bad by any means Shorty Dylan Holloway Ready to swim in we got a Jake Allen that one's gonna be for the Canadians, but you stepped into that thing on the red rainbow auto Yeah, right Putting all your weight behind that thing with some stank on it. Let's go. There's a red of patty king Um green and blue is gonna be Mike Smith and To 10 wow that is gonna be Jake Allen on the Blockers blockers short print. What is that four of 10? Nice one there Canadians Nice low number there on the blockers Insert to 10 That'll do right you guys halfway Yeah I That's fair Harrison that's everybody that I know that we play with we all stink. We do the same thing. Yeah We can all make contact. Yeah, it depends where it goes Nylander Bobby Brink Yeah, Donnie is by far the best here by far one million because he has full control of his ball Here's a black team on my obviously not all the time, but For game mile. Yeah, you can Donnie is a beast Donnie is a beast Hey, let's go shite had a caddy tell you it's the farthest you've seen that's that get me going Alex the vetchkin got a uh orange and yellow of tan urge and not I probably drive that golf cart right into the lake Now you ain't gonna see another one either Yeah, Donnie's very good jake. I'm very good Donnie good He was a high school golfer all that and then was going amateur. We're trying to go on amateur and jake markstrom Lucas Raymond And to 299 we got iced out charlie coil for the brewers Bruins Bruins Bruins Round you guys next one is update jumbo sold out. We got um tops, um They're a bowman chrome dual case with two to go pewter catching a cough There's a yanni gourd red and orange and quartz is going to be barzol shite Donnie It's impressive You know, yeah, he's uh, he's accurate. It is freaking impressive Oh boy, anthony anthony what happened? We're not Blake and Yellow and green pearl Sean immediately turned to that chat dude. He heard anthony He heard anthony was here Um, allure is probably better for like the autos and stuff like that But series one is like the base like um, the young guns are some of the bigger cards for each individual rookie. That's like their It's like their flagship, uh, rookie is the young guns Those do really well for big players and stuff like that. Yanni gourd on the orange slice Um red and orange is going to be Marco Rossi But allure is really good too, man There's a lot of uh, a lot of nice color in here rookies autos because the series one doesn't have the autos Um, if you're looking for that specifically There's a red rookies bobby brink for the flyers Um blue and purple that one is going to be chrysla tang and auto is igor Uh, char on a strong ovech strong ovech that sounds right on the runaways auto for the devils On the blue auto Come on, Greg Sounds great. It probably would though. It was a pretty nice nice, uh, nice quality cards Nice quality on these Jasper bratt um, this one is red and orange of key entree miller and a Matt greg mad so guard i'm not i'm not To 65 Feel like i've met greg Two more on this row Oh for you we're walking up in the room in chicago We're just watching up There's greg What the hell hey greg Yes, you know we lost a little bit we're off the rails a bit We're really back in we're getting going no worries. Sorry. We got time to reel it. Here's a jack use Got yellow and red of um matt grizzlik at least nobody's saying anything about my pronunciation. So we're doing it. We're doing it Got a touch up on my russian That's true. There's a black of joe pavelski got a um red and orange of jacob jacob glavus Yeah, I know man. I know it's like never ending. I know Yeah, of course, it's a lot of cars man. It's a lot of cards. There's a black of um bella veer Both both both bill there There's a uh fixed willonski fixed willonski orange and you know that one's all right nailed it that one's all right I said that confidently. Yeah 52 laden boxes Well depends on if you opened them. There's a gabriel landis scott Red and orange jack camble and this is Uh michael ismont 265. I think Touch them on my polish too.  I'm gonna need some help with samuel fugemo Got a orange and yellow of riley walsh It's a rookie boldy Yes, it's a lure number two matt a lure number two Look matt get some um tops update going next sydney crossby on the black Red and orange is philip tomasino Zegras Pia debt that's right sam.  It's hard Yeah, yeah, just one second red is noa kates Red and orange is gonna be eric calgren. Yeah, you got uh columbus The blue jackets matthew the blue jackets Sharon govitch sarin govitch. All right We got a braden schneider rookie auto nice one there red rainbow auto for the rangers braden schneider on the auto Another matty baneers rookie Selling clothes. Oh boy It's a red of connor garland got green and red of igor char Sharon govitch. Sharon govitch. Let's go. Isn't the same guy Yeah, it is Joe pavalski on the glitter bomb Glitter bomb to 199 for the stars. I know I'm not on the island I'm not on island right now Alrighty box box number 12. Yeah, I've been sitting in flares. I've been sitting in flares for a little bit Yes, I was driving home and I was like, what is that in the road on i4 and somebody put flares in the road because they They're car broken down really they flared up the road.  Casey uh middle stat. That was what I thought I turned the corner.  I'm lucky I've heard but Some of these for sure. There's a vasilev ski go lightning yellow and green is going to be a karson meyer Iced out gibson bass bass bass It was like mad max man. It was fake mustaches papal, um butchenevich Red and orange is iffollow. I've been wearing fake mustache for months. You guys haven't noticed yet. Yeah What's up, karnig doing good my man. How you doing today? I love karnig. How are we doing today? There's a red cold coffee Got green and blue of marke marke kastelik Forest and orange and yellow kalaev to 85 my my favorite candy Swedish fish Swedish fish are good fan Yeah, baseball cards and candy stories.  I hope they're just chewing like what's going on, bro Chase pierce Orange and yellow is oh man vladislav kola kola nachikov It is collection of call collection of call. It's so good though. It's literally my favorite candy I get it my kids give me my birthday every year just the red flavor. I feel that I feel that I like the sour patch might be my favorite pewter swedish kashnikov There's a color flow of toroi terry Yeah, silver silver silver.  Oh, yeah And a full rainbow auto this one's braiden schneider another one another rookie auto for the rangers second schneider auto I get built to fish It brings on one tonight boys. I mean uh college when I was 21. Yeah That's what I mean.  Kyle connor on the red. I was doing cave stands Red and orange. I don't Doubt that and this one is a blocker is the 299 got igor schusterkin for the rangers Igor schusterkin for the rangers. That's what I meant for us. I meant college All righty guys six to go. Come on now Come on now. Oh boy Really Walk him right out the back Leon dry sidle back black got a mark mclofflin The best high school parties this guy right here sylvia's man keggers Everyone in miramar knows where they go There's a black swamin got a blue and purple that one is bow and byram Blue and purple Let's go Greg There's a black a brock nelson Got red and orange that one is mark castellic I'm trying to find numbers man. I'm looking at everyone. They're all over the place I think it's the pulsars that are numbered But not all no, it's a base color. I don't think yeah, the red ones aren't so if it's a base color Another one then base colors are The red ones aren't either. There's a veg melka Veg melka red and orange is mike smith and another blocker is the 299 this time. It's um juce soros for the creditors silver silver silver Yeah guys just the uh fillies left in dual case mlb just the fillies left um jacob movorere Kings red and orange is going to be a chase pierson And quartz is jesper brats or brat jesper brats just the fillies Definitely cj definitely are definitely are yeah Like I said the series wants more of a base product and then the uh young guns are what you're really looking for Then they do have numbered young guns and acetate young guns as well. Those all sell pretty well Um nick, um at bruise at bruise. There's a orange and yellow that one's going to be philip tomasino silvers A matt boldie on the orange uh orange slice die cut Orange and yellow spencer knights runaways mclofflin Eagle is here the other night right after was he right after he was here taking it blaming crud kimbrill Blaming crud kimbrill there's a red of chase pierson Got a orange and yellow of jack cambell and auto is going to be morgan baron Morgan baron nice one there for the jets on the uh full rainbow auto Winnipeg Yeah Red is jordan greenway yellow and green of um madso guard And glitter bomb is sydney crossby nice one there for the uh penguins. That's two 190 not glitter bomb crossby Righty guys five more. Just five more. They like keep multiplying. I know I know Lock five to go And we'll get um tops update jumbo going next that one is filled up Got a black dylan cousins Blue and or green and blue is mark castellic senators The peach man is a speciality cop Got a red and orange that one's going to be adam raska so I thought it was kind of strange too when he said it the first time I was like Yeah, it's it's a glitter bomb. That's what it says What it says patty cane on the black rainbow got orange and yellow of kindre miller or chiantre miller It is a glitter bomb. How are we doing spencer? How are we doing my man? Spence? How are we doing today spencer? Glitter bomb coming Spence bringing them out. There's a red tanner janet Got a red and orange of jack mcbane and glitter bomb is alex the brinkett One's gonna be for the black ox to 199 Hockey's not that tough to sport though. So glitter bomb works. Oh, yeah These dudes don't have teeth Yeah, right the glitter bombs are sick. I I agree christen. I like it Is a frederick carl strum will definitely take a uh a crossbie that one is marco rossi rookie on the um Orange and yellow I'm gonna eat a whole cow Jono, uh-oh, I don't even know which one we're talking about. Let's see if I find it again.  I'm not sure probably be a little bit cj the first one that just came out with series one So it'll probably be a couple months. I would have to think Um Before the next allure comes out. There's a samio fagamo for the kings red and orange is igor Um Sharon govitch. Yep, Sharon govitch. Let's go forest and auto is going to be mark mclofflin for the bruits On the um red rainbow auto not numbered Bruins auto there mclofflin right on right on Gotcha. Gotcha. There's a red of jared spurgeon got yellow and green of of bokonji and mama And all time this one's gonna be mark messier. I haven't seen one of these I'd be relatively short printed there nice one oilers oilers on the messy Interesting mark messier Nice one there for the oilers must be around a case it only when we've seen this case Black is going to be uh jack rosslovic. Yeah, spencer. Yes, red and yellow andre's leet There's a bunch of numbered cards Yeah, yeah, there are some um It's the pulsars the glitter bombs the the base color stuff aren't so the black's not there's nathan mckinnon Any of these that are like base color if they have the pulsar and the color then their numbered orange and yellow is bobby brink It's a black catan clayton keller Um red and orange is going to be brock bozer Yes, samuel taking one in flux richard taking one in flux jake taking one in the mixer Spots are flying. I appreciate that guys a couple in flux for samuel. Thank you. Greg. What are you talking about? Thank you richard couple in flux as well There's a red uh root hints Orange and uh orange and yellow is yanni gordon into 99 We got a matthew shane for nashville on the games uh game stills for the predators Appreciate everybody grabbing spots next one to spots a fill is our um bowman chrome dual case just the fillies to go Just the fillies hayden hogson Um, this one is red and orange of michael bunting and a quartz of blake colman Quartz for calgary Yeah, yeah Yeah, those those inserts i've seen through the numbers have been on the front Yeah They got numbers in like six different spots on these cards. It's definitely interesting It's a michael ice ice amont Um, there is uh sharon govich on the green and blue got that guy down Got that guy down I have I've seen him on uh, I've seen him on the making the calls on the espion There's a orange slice victor headman go lightning Um red and orange is lexus laffernier mani baniers runaways The boldy rookie rupee gotcha Gotcha I'm gonna remember these I think there's a pewter kachakov kachakov gains There's yellow and green owen power. Thank you shon and auto is gonna be michael icemont's For the jets another jets auto there on the uh full rainbow Not numbered auto I don't know if the wings uh wings been known to anybody recently that card looks cool. Let me know what this is It's a thick one First one we've seen at ease It's a red of jake gensel Color flow is red and orange of jonathan dollon And it's a debloons trevor zegras Gold debloons trevor zegras. That is a sick-looking card A little piece of something on there. We'll mag that I believe that is sick It's gonna be trevor zegras for the ducks. It is a little bit of thick too On the debloons first one of these we've seen Nice one there That's true spencer. That's true Red wings definitely had some good teams definitely definitely Ready I can't I can't help it with shon, but trying to be on your side spencer Yeah, those are nice. I haven't seen one of those yet cj. I like that A little bit thicker stock on it Red is krill kapritzoff You got a red and orange of robin salt and glitter bomb is gonna be a kaden korsak That one's gonna go to the golden knights to 199 199 kaden korsak a thicky-vicky Exactly Exactly garnet There's a black jack mcbane One is orange and yellow of dylan cousins and a quartz rookie that one's gonna be owen power nice one there for the sabers On the die cut rookie thicky-vicky Black rookie jordan morris or jordan harris jordan harris Red and yellow of kaleev and nookates runaways Orange is jacob markstrom orange slice orange and yellow taylor hall ice out quin hughes No, it is It is most definitely spencer dude. It really is red of mark castellic worse than the damn 60 box burn red and orange lucas rickle and auto is gonna be matisse macelli for the Coyotes on the rookie auto Red rainbow auto i was sending him the silver rookies too and had to stop there was too much going on Hey, thank you chris for buying one on each of ours each of the uh mixers or buying two spots in the mixer I should say I think the first one's phil Red thomas hertzel got a yellow and red jack cambo and green die cut is joe pavalski One is for the stars to 99 So color match there for the stars blackjack alan Uh green and blue adam raska Yeah, exactly exactly spencer. It is like that the base cards are silvers in this There's a dylan larkin black Red and orange is payasutra ice out grubauer two more It's a trevor zegras for the ducks orange and yellow is isaac ratcliffe i thought tuesday was saturday Very real Yeah Sean's been Sean's been a little confused on the day of the weeks recently Yeah, he went to froggers the other day greg and it was a saturday or he thought it was a saturday It was a tuesday. He's asking revered. He was where the games are Hayes thompson black got a orange and yellow that one's isaac ratcliffe He's so confused the box Yeah, he definitely did Black is victor arvidson Red and orange is martin netjes niggas. I don't know. I know it's weird I know it's not Daniel's killing it It's a black connor mc david nice one. They're oilers Got a green and red of philip highlander hollander. Hey, let's go man. That's six spencer Well game use hard and that's sick Ripken to 15 take that too Appreciate that man. I'm glad we got one nice one for you Brent burns on the red I've read an orange of um bokonji imama And connor garland on the green to 99 connor garland for the canucks Black rookies lucas dostel Yellow green alex ifalo Yeah, nice spencer. Very nice, man Appreciate you helping us fill those man. I'm glad you uh, they got a dub Victor lodin, there we go Uh, red, orange is owen power. I'm down. I'll get a tatted shawney arp. Yeah The pack was a cinder block. Yeah, the patch ones you can see definitely orange. Um nick abruzy Bruisy orange slice green and blue is jack mcbane And matt boldie rookie runaway Yeah, I think it's like, uh Yeah, I think it's insurance Jordan spence Rookie's red got red and orange of mads so guard and auto. This one's going to be philip hollander. That's going to the um To the penguins man full rainbow auto. I'm just just Yeah Red uh kuznets off Orange and yellow is cop and wow, I've not seen this one josh anderson that one is the pink leopard Pink leopard on the josh anderson not numbered must be pretty short printed First one we've seen the pink leopard another um appropriate one for hockey The pink leopard leopard josh anderson for the canadiens must be pretty short printed hadn't seen that one yet One more box One more box That was a uh marathon here. Let's go box 18 box 18 Populated uh Philip grubauer Red and orange is blakenberg You can give me every card It was a blue Oh, man, unbelievable I'm not doing all this. Oh, there's a uh boon jenner boon jenner for the blue jackets and a blue and purple This one is ben meyers blue and purple Not numbered there Oh nice. You got another one today. Let's go It's a debris cat Go riley red and orange is going to be a matthew kichuk Well, iam rip and person was on latin channel too, so we can yeah, you do that There's a shan cotier, uh yellow and green is um caligren And blockers blockers the 299 is carl fez melka. Bez melka black is isaac wrackliffe Red and orange is going to be jack quinn and quartz brady kichuk I like shawna to do josh anderson Orange and yellow velasic. That's not very good Chris critter on the orange slice got red and orange that one's going to be hayden um hayden hogson own power two more packs Oh, yeah Chill chillin. Oh, he's so good. It's a red matthewis macelli Orange and yellow is going to be thatcher Um thatcher demko and this is going to be lucas rickle on the quartz rookie die cut auto That's a sick one there for the um For the black hawks not numbered on the quartz rookie Nice one there lucas rickle Yeah, that's going to be shawn tomorrow. That's for sure. We like to need to get one of those wear that matthew kichuk red I'll say red and orange of um I got what he said already Like to pay rupe rupe and this one is glitter bomb braden schneider for the rangers to 199 And that will do it guys full case number two a lower hockey Let's go on over mars. What's up brother? How we doing sir? How we doing? Yeah, that'll be shawn tomorrow. So he'll have a nice change up top. Loan some Top loan some uh almanac instead for sure Alrighty guys, we had a 299 um matt dishein To uh to 99 was dishein as well. We had a 199 to brinkett kint johnson uh corset thomas schneider pevelski crossby and mark stone glitter bombs These were to um 100 on the orange and yellow mire and ratcliffe and matthew kichuk Had to 99 uh igor shisterkin carlstrom pevelski and connor garland These were to 85. We had um cup and kaleev To 65. We had a sogaard and isamont blue to 50 was going to be yanni gourd had blue um akituk akituk the 35 uh the devils Someone was quintin byfield the 10 nice one there for the kings We had also the 10 jake allen for the canadiens had a all-time messy And we had the pink leopard of anderson Autos we had base auto of lucas rickle on the die cut had base uh hollander macelli um ice moths McLaughlin that one is um baron schneider um another schneider This one was um sharon govitch Yeah, I just can't read them josh jake allen on the base. No, it's not that chase pierce. Um lucas rickle There's a patch out of that one sick connor garland for the canucks To 10 nice one there sick-looking patch on that one. Um had base auto ratcliffe red was fix uh fix Wolotsky there's a Corsac Brad Marchand and naphen smith and our one bag was a debloons of trevesy grass Alrighty there that'll do it for a protecolore 18 box full case number two. Thank you guys We'll get these uh get baseball going now


You from here? Duh, nigga, but like, well-bought, stupid-ass nigga. I know in Orlando, like, what, you ain't got no hood? You just outside the fuck? Castleberry. That sounds same.  Ain't nobody for Casablanca. That nigga by his side. Look at the real Orlando. Look at the real Orlando, nigga. Said, there ain't no Orlando, partner.  Talk about Casablanca.     Yo. Yo.  Who Casamere? Anybody in Casablanca? How to floor the niggas. How to floor the niggas. Talk to the police. Man, no pussy-ass crocker to pull me over, man. Man, I'm crocker. Get behind me, man. I told that crocker.      So get what your boy do. What you do? Boy, I took them cracker on a high-speed chain. Here's your time, man.


Hi, I'm Zor. Welcome to Unizor Education. Well, I continue the course of advanced mathematics for teenagers which is presented on Unizor.com and that's where I suggest you to watch this lecture. This lecture contains a couple of problems in theory of probabilities. I call this problems number six actually in this series. So basically, these are problems related to bias formula for conditional probabilities and I do suggest you to refresh theoretical material which precedes this particular lecture dedicated to conditional probabilities and well, let's just go and solve problems. I have only two problems, so I'll just probably speak a little bit longer about each one of them. So the problem number one. Let's say you have two events, A and B, and you also have the probabilities of these events we know and we also know the probability of conditional probability of the event B under condition recurrence of A. Basically, graphically it can be presented as this. This is A, this is B, this piece is A and B. So that's what's known. What's necessary to find out is a conditional probability of A under condition of occurrence of B. The opposite. Well, I don't want you to consider this problem as purely theoretical. Here is an example which might actually kind of lead you to appreciation of the problems of this type. Let's consider that event A is a particular randomly chosen person is a good mathematician and event B is that any particular individual randomly chosen from whatever you meet is a good chess player. Now, you might actually have heard that good mathematicians, well, in many cases are good chess players as well. So let's assume that you know approximately, at least, the number of good mathematicians among all the people you might meet randomly. And you also know approximately number of people who are good chess players. Now, if you know that certain percentage of mathematicians are good players, chess players, you basically realize the conditional probability of having, being a good chess player under condition of the person is a good mathematician. But you want to do a reverse. What's the probability of the person to be a good mathematician if you know that he is a good chess player? Well, it makes sense, actually. Now, we can just put some numbers. Let's say the probability of being a good mathematician is 0.01. So let's say 1% of the people you might randomly meet are good mathematicians. Now, the probability of the person to be a good chess player, let's say is 10%. So it's 0.1. So 10% of the people you meet are good chess players. And also assume that the probability of the person to be a good chess player under condition that he is a good mathematician. So out of all the mathematicians, which you know or you might meet, let's say 90% are good chess player, which means this is 0.9 probability. What you are looking for is you just randomly find the person who is a good chess player. What's the probability that he is a good mathematicians as well? So that's the practical problem, which we are talking about. Now, well, actually the problem is trivial, quite frankly. It's very simple and it can be solved using just the definition of the conditional probability. What's the conditional probability of A under condition of B? Basically, by definition, it's the ratio of simultaneous occurrence divided by the probability of B. So it's area of this divided by area of B. That's what will be in geometrical interpretation of this, of the probability. You know the geometrical representation of probability is basically an area of some of some geometrical figure and the conditional probability of let's say A under condition of B. It's basically a ratio of this common area divided by the whole area of B. Now, at the same time, we know all these three parameters. Now, from these three parameters, well, probability of B is known, but probability of A and B is not known, but we can find out it from here, because at the same time, I know that the probability of B under condition of A is also the ratio of the same intersection divided by probability of A. So, if you divide this area by the area of A, that would be a probability conditional probability of B relative to occurrence of A. And this, we know, we know this one, we know this one, so we can determine this and substitute it here, right? So, probability of A and B is probability of A times conditional probability of B relative to A. So, I will substitute it into this formula. So, what would I have? I will have this probability of A times conditional probability of B under conditional A and divided by probability of B. So, that's the formula. So, from this is the formula of conditional probability of A under conditional occurrence of B, if I know probabilities of A and B and conditional probability of B under occurrence of A. Now, let's apply it to this particular case. So, again, I am meeting an accidental meeting, the random meeting, the person who happened to be a good chess player. Now, my question is, is he a good mathematician? What's the chances? What's the probability of him to be a good mathematician? Well, let's just substitute it into this formula. What do we see? We see probability of A, which is 0.01 times conditional probability of 0.9 divided by probability of B, which is equal to 0.09. So, nine percent the chances are that if you have met a good chess player under these conditions, the chances are nine percent chances are that he is a good mathematician as well. Well, it's nine times greater, by the way, than if you just randomly pick the person, because if you just randomly pick the person, the good mathematicians are only one percent. So, the chances to meet a good mathematician is one percent. But the chances to meet a good mathematician among good chess players is nine times greater. So, if you want to find a good mathematician, well, go to chess tournament and well, ask around, whatever it is, and the chances are that you will meet a good mathematician among them significantly greater than if you just try to randomly pick people from the street. Okay, that was my first problem. My second problem is in some way similar and more practical, I would say. And I can even exemplify it in a very practical way. Let's say you have two events, A and B, and they are mutually exclusive, which means that the total sample space is divided by these two events. Well, as an example, which I'm going to use, event A means that the person is sick with certain illness, whatever the illness is. It doesn't matter.  These are two mutually exclusive events. They cover completely entire population. So either the person is sick with this illness or not. There are no other cases. Now, next is we have certain event X. It's somewhere here, partially overlapping with A and partially overlapping with B. Now, what do we know about event X? Well, we know conditional probabilities of happening of event X under this and under that condition. So, geometrically speaking, we know the ratio of this area, which is common between X and A relative to A, and we know the ratio of this area, which is common of X and B, relative to B. That's what we know. And obviously, we know the total probabilities of A and total probability of B. Now, what I would like to determine is conditional probability of A under condition of X. And here is a very practical example of this. Let's say X is certain test which you are running a diagnostic test for the person. You know, some person comes to a doctor for a regular checkup, for instance. And the doctor wants to conduct this particular test, X, which is supposed to identify this illness, which I called A. So, event that this person is ill with this particular illness is A, right? So, question is whether this diagnostic procedure determines or it does not determine correctly whether the person ill or not. Okay, now we need some numbers. We're talking about example, right? Okay, so let's consider the following situation that there are only 1% of people who are sick with this particular illness, which means that the probability of A is 0.01%. Now, obviously, B is the rest, 99% of the cases. So, probability of B is equal to 0.99. Now, what about this test? Now, this is a diagnostic procedure. Diagnostic procedure is not perfect. I mean, if it's a good diagnostic procedure, it has a very high probability of identifying the illness. But here is what the real numbers are. Now, if the person is really sick, then the test X shows the positive result, which means the person is sick. Not in 100% of the cases, which would be ideal. But let's say in 99%, 95%. So, the probability of positive identification of the illness, when illness really there, is 95%, which is pretty high, right? Unfortunately, unfortunately, if the test is run on a healthy person, then sometimes it gives so-called false positive, which means that the person is not really sick. But the test shows something. Well, let's say it's an X-ray. And X-ray, you know, sometimes X-ray is clear, sometimes it's not exactly clear, and it's not always obvious what exactly the X-ray shows. Alright, so let's assume that the false positive are 5% of the cases, which is not a lot, right? So, that's what's given. What I'm looking for is the following. What's the probability of the person being really sick, if the test is positive? It's a very practical situation. In medicine, I mean, you always happen something like this. Any test has certain reliability factors, and the reliability factors are these two. What's the percentage of correctly identifying the illness when it really exists? And what's the percentage of the false positive results when there is no illness, and but the test shows some positive results? Okay, so these are reliability numbers, which are quite practical. But let's use this to find this particular probability. All right, we'll do exactly the same thing. This is equal to probability of their simultaneous occurrence relative to the X. That's what we're looking for. So we're looking for ratio between this piece, which is common between X and A, divided by this area of X. That's what we're looking for. Now, both we can determine. Now, the simultaneous occurrence of A and X, I can use my existing probabilities, is its probability of A times conditional probability of X over A. That's simple. That's exactly the same as in the previous problem. Now, the probability of X, the full probability of X, very easy actually, we can see it from here. X is divided basically in two different pieces. The one which is common with A and the one which is common with B. The one which is common with A, we have just calculated. So I will put exactly the same thing. Plus, the one which is common with B is analogous. Sorry, this is A, which is P of B times P of X under conditional B. This is the full probability of X represented as a sum of two pieces, X common with A and X common with B. Since A and B completely encompass the entire space, there are no other pieces. So this is basically the formula, which happened to be called bias formula in theory of probabilities. Now, let's use this formula in our case. What do we see? 0.01 times 0.95 divided by 0.01 0.95 plus P of B, which is 0.99 times conditional probability 0.95. Now, let me write the result approximate. And this is, quite frankly, it's a striking result. You see, just judging by these numbers, it doesn't look like the test X is really like bad.  You see, 95% of the cases, if the person is really sick, it shows, and only 5% on healthy person it shows. But that's unfortunately is the problem. This is the problem, you see. 99% of people are healthy and considering 5% of them give false positive. That's a lot of people. So that's a lot of positive results, which really are, you know, misleading, so to speak. That's why I have only 16% probability of the person being really sick, if his test is positive. 16% is really low probability. It's low reliability. So no matter how you think about this test, the real result, which is supposed to be applied to real patients, is that, well, test is unfortunately only 16% reliable because there are many false positive. Okay, now I'm sorry, I have to put 05. Okay, so what I wanted to present to you is basically another application of the bias formula, especially the second problem related to reliability of the tests. I do recommend you to read the notes. They are, for this lecture, they are very detailed and that will probably help you to understand this approach to theory of probability is much better. And well, it's very practical, as you see. So that's it for today. Thanks very much and good luck.


Going on everyone Sam here with late sports cards ripping one box the sports card all-manac multi-sport personal box edition volume six Box number 54 for Ronald G. So good luck here My prize card. I own Probably 99 Ray Lewis upper deck on card auto or 2012 NT Pro Bowl worn laundry tag of Ray Lewis Those are probably the two that I really love Most that's sick dude number to 250 true purple 23 bowman chrome Brandon my aya Of the Yankees congrats on that one there Ronald. That's awesome now Brandon. Yes now known as Brando slash Brandon 89 of 250 nice one there congrats on that one there Ronald Thanks again for the sport buddy get that right out to you my man