okay there we are once again the famous Bendix diner as seen on the Jerry Seinfeld show comedians in cars getting coffee. Hasbro-Kiteson, Jersey, Route 17 South. I'm headed to meet with William H. Morrill III commercial voiceover specialists and former NFL football quarterback for more wackiness or God knows what we're gonna talk about maybe more madness who who the hell knows who the hell knows suffering suck attached suffering suck attached are you two you too yeah right yeah right well she's not here you're not here she's here you're here something's going on something's that going on and I don't want to think about it I know excuse me we have my coaster where how can have how can that be they don't make a hundred percent cotton jeans do that they're they're mixed well how did they shrink I only have one thing to tell ask her are you doing your your share to prevent swamp ass in America I don't even ask I'll tell you I'll tell you later check the web no I don't I I'm a stickler for not sharing personal business on the internet oh yeah that's why you're on the computer no I talk about politics and everything else consumer advocate but people want to don't want to be on text Jerry since Sunday and he asked me what did you go Jerry who's Jerry the guy down the shore he said I haven't heard from you what did you go on the cruise he used to say he used to say me the past so you were you with your own wife and now he's saying what did you go on the wait a minute is Jerry the guy just okay okay okay I thought well I don't know your your I lean nobody listens I don't know who oh who all your friends are but what's going on with you but wait boy to me he's questioning you the guy who never had you down to see his home and he's questioning I lean so nothing positive materialized yet but that's the wisest thing she was in the same day she's still talking about the laundry not a different day I know the way the day looped so what you're when they again so what you're talking about is the temperature of the water in the washing machine that caused it to shrink when you say low medium dry air why why nobody listens to you listen listen if you're if you're cooking let's say a pot rose dinner and you and you use her logic and you crank up the heat all the way what's gonna happen to the meal oh you know what she's smarter than I thought she was you're right it is gonna get burned you ever hear the term low and slow bar the barbecue experts the pit masters they always say that you can't go wrong with a low flame or low medium it'll it'll dry faster if you put the dry I know but where what's the name of what's the name of your place oh is that where you are skyline shape and then I told him he had no one so he made me sick with you wanted and don't everybody does whatever they want what's called a landlord why not you're paying rent it comes with the rent you pay for it yes you do it comes with your rent now that I do did they paint your when you moved in did they give you a new paint job how long we've been there okay wow wow that's a long time so most 20 years you know what the landlords seem to have something in common in this region there's a pattern here I'm starting to see it you know I'm starting to see the pattern with landlords in this part of the country your landlord the one he used to have the one he has now mine in other words what I'm trying to say is they they want they like to collect the rent right away on the first of the month but they don't want to fix anything well naturally he does he lives over here no no where he used to live back in the day he had a no good scumbag now he's got like a different kind of scumbag no no he's in a nice place he's in a very nice you know he's fine is the problem is the owner of the real estate is a piece of excrement or camel down yeah to put it nicely in public fecal matter I had some Mexican food yesterday with a with a frozen margarita margaritas on during happy hour it sure did yesterday was my birthday yesterday I went with a friend to in Garfield is a really nice Mexican place I don't like to talk about those things what 57 no not really worse 58 let's put it this way I entered a new decade you're 60 yeah see I got the background left I entered a new decade yeah that's Jimmy you're what 70 70 look 70 what is it no I don't but you're looking well you look older but I've been nice well yeah they do but they do it behind your back hold on let me ask the pendu pendu long is our friend Eileen older than she claims to be okay it's saying yes I think I'll let me do it always does it does does do women lie about their age even though they don't want mental I do women often lie about their age how old are you but they they but they don't like mental I yes that Leslie Nielsen is not is I lean 54 yes okay what about Eileen's college Eileen's college certificates is it for real I want to find truthfully Eileen has a college degree doesn't she yes you're a smart cookie I had no idea you were that that's why he loves you yeah right Eileen loves Jimmy doesn't she I'm trying to figure out the motive he has my friend Jimmy like Eileen a lot more than he lets on now the amulet is busting my chops now oh that's the amulet everybody bullies the amulet no I like I like her as a friend she's very entertaining I just said more than you let on we're not like what you're thinking uh-huh why what are you thinking I'm thinking entertainment purposes are you entertaining she's she's a compadre are you a compadre well that means that means like a friend a buddy of you know love can make the person nervous you know but see only bills allowed to have the the perfect ten exotic dancers and models and actresses but only bill can have them but I how he wants to twist it all towards bill he wants to have the best okay as my parents said have I had all my women been simply wonderful incredible boy that's wrong right now I'm not doubting that swung right into I know two other cherry was in the Miss Universe page she's Miss Iran Roseanne was in the cover of Cosmo twice so it's okay for you to bone out it's okay for you to bone out the best but it's but for me you want me to bone out the worst you had heavy women come on I had heavy problems that you like heavy no I had heavy issues you know you like heavy you know you know what the problem was used to say nobody you know what I know I will never be cheated on no no you know what the problem is the the the here comes the top no no it's gonna take a while I guess the top of the line young attractive ones are glorified prostitutes gold diggers they're gold diggers because it grows pretty she's a gold digger no no they ask too many questions about income about a man's income but they're rare so what you're saying is most beautiful women are gold diggers well they size up I don't know what to ask you about what I'm saying they they ask you most beautiful women are gold diggers they always ask you what you do for a living too too quickly in the first conversation no I don't agree with that I think when you meet people you say what's what do you do that's very that's kind of rude no stop what are you gonna say when you meet somebody what's wrong with somebody that used to do stuff but doesn't really do any a lot now what are you saying I don't understand why does somebody have to be doing something you know what I don't understand career-wise I don't know what you're trying to say they don't know what do you do well it's not a dear business why isn't it because they shouldn't concentrate on materialism concentrating they're asking a simple question what do you do yes that concentrate that's an American woman thing oh yeah that's a good wrong for anybody to ask someone else what do what they do for a living actually very rude it's kind of rude that means you're stuck you're trying to sign up your ink is it rude when you ask someone what they do for a living go to your own maybe that spirit was JP Morgan is talking to maybe it's well maybe it wasn't you know JP was kind of like big I don't know you don't know that you never met him either listen I read many stories about JP and the Industrial Revolution give me got some bad it just told you here come on come on there were scumbags Carnegie okay we're not getting all right now all right now you want to know when somebody brings up the subject of career or job are these women trying to size up the man's income I just got through asking it is it wrong for anybody to ask another person what they do for no but the motive of asking Jimmy because you so how do you want to phrase that question you you get very vague question because you do you want to be working you grew up in a corporate how do you want to be Jimmy don't stay off of it how do you want to be worded that's what I want to know it's none of a female to be worded the question you know what I don't care because I still say it's nobody's business about you know it's rude I'm talking about just flat out rude I don't agree with it at all is it wrong for anybody to ask me I could people want to do I just can care less is it wrong for anyone to ask anyone what they do for a living okay what about what about instead of that question what about hobbies and interests now is it wrong to ask anyone about their hobbies and or interests no that's perfectly fine that's acceptable because it's stupid and very ignorant rude keeps secrets from others well then they have to accept the person for who they are you find out if I ask you you know what you got a point because eventually they'll find out what about you anyway somebody said like one month long no you know I can't ask you I know you say what are you into that's the same thing what do you want to do for a living no no everything is living income career I'm talking about hobbies interests like my friends you mean Madonna have a clue about the proper and right questions have to ask of a woman he meets I would never ask a woman or what she does boy as I get as I've said an ask before some most people just don't get it do they you know if the CEO or let's say Warren Buffett because he mentions his name a lot I'm not a lot occasionally let's say he met a perfect 10 and right away she asked she didn't know Warren Buffett she asked him right away well what do you do for a living well you know what Warren Buffett should say I'm I'm a I'm a financially in a very financially independent man very I'm rich but guess what you have to sign a prenuptial agreement whoever marries me has to sign a real prenuptial agreement that's stupid argument because they hold do and they're smart and they get challenged in court and defeated now what's your next question you hide your money you can't hide it the court you're gonna get you you mean it doesn't get it there's no lawyer that you know it's not binding they're all challenged there's no lawyer that can finagle is a prenuptial agreement before marriage binding you know like bananas no because the woman is favored in court a law that's why it's been settled there's your answer what do you want to know next that's because judges feel sorry for women what do you want to know next is tight it is it is the jiu-jitsu judicial system in America bias against men oh hell yeah is the judicial system in America biased against men also so the amulet thinks it's fair to treat women special bias against men that's a broad statement you didn't say it but gets married men during a divorce or anything like that you can't narrow things down now in general they feel sorry for the woman women get screwed in court a lot they do I don't think they get no I don't think oh by the way the woman oh but when they complain and complain about being raped or sexually insulted in college or by these these coaches or whatever that favors a woman I don't think so we're gonna favor the woman well it has to be proven that it's actual sexual harassment it didn't it didn't favor the woman that's the point and do a lot of men and I've known them yes the woman yeah yeah yeah take the whole suitcase the whole suitcase no I'm gonna go through it yeah right well if there's any money missing Jimmy took it do a lot of women I mean do a lot of men yes a beautiful woman to death because they want to get in her panties a lot of men yes a woman to death because they just want to get into her pants so obvious Jimmy come on man knew that is it only an idiot a person with no absolute class that yes is a woman in death and desperate to help by the way I read a whole bunch of articles about how OSHA is coming down on Jeff Bezos Amazon warehouse many violation safety violations you're trying to show them aren't you let me ask you try to flirt with let me ask you a question do you know generally how to do laundry but you used to do it in a stream streams to beat it on rocks the rocks stonewashed animals no not your mom Jesus Christ I mean that was what 70 years ago now now stonewashed denim that's a bunch of hogwash right I don't believe hi hey well I do I think if they say still wash they probably throw a little bit of stones in these giant ones and things and throw the stones in to beat them and change they're not lying it is stonewashed but it's not in a stream or something now when you two are together and you do your laundry I want to make sure you use real yeah sure now who who's the one that officially shrank your pants who the aid again the one that aggravates you all the time the aid that visits her I think Jimmy's got to come over and do your laundry and we're all out I don't know why he has an obsession with us being a dating couple I have no idea why you know why should you mean so he can parade should my friend Jimmy Madonna go over and do my friend Eileen's laundry see all of today it's not gonna happen that's well hey said no listen more power to you okay I wish you what do you want to know Jimmy no no don't keep going on what do you want to know give me is a William H. Morrill like Andrew Anderson that he tries to get under old James P. Madonna skin am I like our friend the pro wrestler Andrew Anderson am I like Andrew Anderson in that I try to get under Jimmy's skin all right well that's good to know okay Jimmy contend tends to get very defensive doesn't okay depends on that's very good depends on the subject doesn't depend on the subject now so he's saying I'm defensive all the time within skin so that's why from now on his new name is egg okay well an organic egg has thick shell but I don't care you're his new name from now on his egg egg like Vincent Price on the back so if I say you've got egg on your face that means he's kissing you martini shake it not stir oh smooth to lose baby yeah well oh look at him you can you can I'm not embarrassed you can she's happy look at her she can he can think smooth you know what I wait wait you hear that music I know that's not gonna apply here oh oh by the way yikes they give you facts Eileen she's she didn't know that Eileen these are facts I leave she's interrogating you now I'm entirely interrogating back Eileen isn't that right Jim I'm doing that she I can use the names of every sentence to why do you notice that he said before he doesn't want to get into it when you asked him what how's it really going with you I didn't say anything about myself I just got on her by leaving and then Jim and now you get nervous no you said no you said I don't want to get into it no I didn't mean a word of what I'm doing did you I didn't say one word or what I'm gonna do something bad to Mr. John Gabriel oh yeah bad like a goat was bad Jimmy Madonna tries his best to change the subject no I'm being Sherlock Holmes deductive reason you know you ever hear that I know the kids am I anybody's fool is there more to William H. Morrill than meets the eye here's there more to me than meets the eye oh I got a good yes there is more to you I always knew that I could have told you that but but you're you're you're trying to be you're sneaky enough to withhold a lot of things but my dad has your idea now I was honest I knew that right away oh I got a good question it's gonna be something is I leans purse real heavy because there are important things in there does I lean carry a lot of important things around in her large purse that means you're oh wait a minute is I lean no no you don't have anything is I lean a hoarder plain the simple is I lean a hoarder you are you're a hoarder you are hoarding disorder oh huh oh or and you who are you don't what do you hoard magazines newspapers what because that suckers heavily why why yeah but not just do there's a reason I mean could you get to the point just a bam right you know really you're going to go 20 minutes on this answer why do you hoard there you go what do you have like 20 stacks of you know up the ceiling of magazines and stuff no I don't know I'm learning your place Jimmy has he's full of crap let me ask you a question let me ask you if you don't wear them you said you buy a lot of clothes but you don't wear them it doesn't matter Jimmy got that now the room you're so-called my guy boyfriend this is named Jerry Jerry Gallupi no Gala Gala Jerry Gala do you honestly think after eight years that he is going to perhaps marry you no he said he's gonna marry you when he turns 60 no no no forget about that I'm asking you you think God knows well let me fix God knows I'm Bill that's Jimmy that's not true no that's not true Jerry Gallupi both of you is there a God there's no God you can laugh all you want I lean simply believing in God she's believing in the supernatural isn't she it's a rest something you've never seen before you believe it that's not good so it's a cat lick thing cat lick it's it's I'm sorry but it's ignorant you believe in something that doesn't exist do you get the way what did God create man did man I wish you'd listen Eileen listen to the question you are so scared you try to interrupt this every time that's not right do you know what swamp ass really is do I lean and Jimmy try to interrupt you sometimes out of fear I'm like freaking flash Gordon Ryan afraid there you go okay mean the merciless yeah oh look a miracle she didn't take her purse with her oh we're going I lean a miracle look at dad he's pregnant unbelievable she's actually put she keeps pulling out her so do you really know what swamp ass is you know it's these people the vast majority don't have any clue whatsoever do they does William Morrill say this all the time because he wants to put himself on Mount Olympus do I say and ask this question all the time because I want as Jimmy says to put myself on Mount Olympus he likes you the amulet so far you're all for six he likes you all the amulets like you that's that you don't ask a question I know you don't I always answer good I don't make fun I ask good questions before I asked him good talking I don't know I have to think it's something he's gonna think now I lean oh I heard Donald Trump really scold the Jeff Sessions is that going to cause a lot of friction they've had for nothing but friction between the two of them it's all happy it's all heads going on every day it Jimmy there's nothing new this is just where Jeff Sessions has already months ago recused himself and I'm Trump's telling him it's old hat there there's been nothing but friction yeah so how do I ask that question when it's you just answered it I asked oh will the real well would the Republicans be surprised in this November you mean the what do they call these midterm elections yeah the upcoming November this is upcoming November midterm elections the Republican is a Republican party before to be extremely shocked by the Democratic Party or the Progressive Third Party is the third party I'm expecting a phone call I gotta wait for it I don't have to wait for you bring your phone with you oh you better give me a cell number why didn't you give me a cell number no I get the home number because they'll be calling me in the home number about different hours and stuff like that that is that's not that is an answer I don't know I said why don't you bring yourself don't look home on your phone call me on my home phone yeah no I don't know I don't understand I don't understand that logic they'll have to contact me in my home number because of the early in the morning yeah no I don't know I'm wondering why you're giving that not early in the morning my cell phone shut off early in the morning my home phone be turned on early in the morning I have my cell phone shut off because some girl calls me early hours in the morning on my cell phone but it's not early in the morning but he'll be calling me early hours in the morning but it's not early in the morning now I don't know he's calling me about something else now is he gonna call me afternoon now you're gonna roll it is okay if it's not personal can I ask you what what's so important where you don't know where it is oh my god no I don't know what is this about you don't even know the food store I'm gonna find out no I just doesn't make sense to me you should have gave your employer yourself yeah no I don't know there are many in hack and you don't know who's this guy no I do keep saying I know no I don't know I lean this doesn't make sense to me why do you give the the future your possible employer your cell phone number I guess I will eventually you keep saying we know eventually you're gonna trust this person but do you see the logic of you having to do that I'm having to rush all the way home this is how people get murdered when they talk to people they know nothing about when the guy ends up killing now she has to rush all the way home thinking I'm gonna get a phone in case they do so now because you didn't give them your cell phone number you have to leave a public place and rush all the way back home listen if your cell phone number is is hard to remember do what I do put it in the notes you know on the cell phone has a section called notes or reminders there's an it's a notepad put your cell number put down I lean cell phone number and put it you have a smartphone or or like him you have like okay like me so put it in the notepad as put down I lean you know type in I leans cell phone number and put the number in there and save it this way you have it because it's really you know to be honest with you a lot of people with smartphones don't even have a home phone number anymore because it's a waste of money why pay for a home phone number if you have a smartphone I'm using I'm using but that thing where the notepad is I'm using that to dial numbers out on to get my keypad to get the key to get the to get to get to be able to make out okay all right listen there are there are three pads that I know of on iPhone there's the contacts pad which which allows you to dial out with one click right then there's reminders let's say you have a shopping list you're going to the store you can put all the items you need right on there and then there's notes no I can even show you then there's notes there's next time I'll show you where they are because right now you got to you got to go home but please any important calls from anybody you should always give them your iPhone number and then next time remind me I'll show you where the notes are and the reminder pad is because when you have that you don't even have to write down your shopping list you just look at it and it's go I need milk I need eggs I need bread I need this I need that it's you know technology I don't write nothing down but anyway go go go yeah oh my god theologic bizarro world boggles my mind that was an experience a month six weeks ago when our current lighting ran out of here moments after Ray the kind of little out there guy left she brought out there after him we she scurried I lean says she had nothing to do with Ray did not go out there following him she telling the truth now she's Irish and they say Irish kiss people kiss them they kiss the Blarney stone it does does kissing the Blarney stone means that the Irishman is full of shit I did look at that swing brother I would love to make my my English language art diction or grammar I would love to make more articulate well I like the feel of it it's not a sex toy I know we got to go to Gabriel's I want to eat I want to order the fried calamari phenomenal they say yeah about the quality of food to Gabriel's there's a restaurant in there for everybody raised about this quality of the food as incredible as everyone raves about right I think Gabriel's the place to be yeah you admit that you're not 100% correct right but you do admit you are correct good 99 point something percent of the time I write I do really rarely rarely right because I use it in the casino and at the racetrack and it was a hundred percent accurate they followed me around but it was a hundred percent accurate you know what you know the psychic mediums on these special programs say that you have to you cannot confuse the spirit world you have to be precise in your questioning well that's what I that's what you've been saying right I know but you I'm not this is why I do it you know and then when you hear but I'm not disagree I'm not this I've been telling you for a long but I'm not disagree with you I should get to your position you get so you get so aggravated by it because you keep going when I just told the guy we came in earlier and I got we got a touch I thought I'm gonna leave you to you talk to you here I how that be no no no we talk you'll talk yeah like I a light bulb went lit up over my head and I said you know what you know what I think far too many people are long-winded as they say I agree with that oh I have a question yeah um was Bill Moro and Andrew Anderson correct when they said do not make it look like you're pursuing a woman do not make it obvious that you're pursuing you to be following women that was you cannot follow when and my friend the progress my Andrew Anderson and my son sat at the Barnes and Noble Cafe in Brampton Jersey years ago and we told Jimmy following a woman around worry where we write and saying that it's totally stupid what he was doing well because it could be misconstrued as stock misconstrued you were stalking that's wrong you made that stupid face when you did I don't know why you were doing I said why I don't know I didn't have a mirror in front of me I'm not saying was what Jimmy was doing was that stalking look at that well for a technical terminology I guess it can be Jimi Madonna be following anyone around this was many years ago by the way yeah no no and we told you stop it remember we set you to get over here right now now if if you happen to be online or doing something and a woman stands beside you and makes conversation that's a totally different story don't you you don't understand you well I'm trying to I'm trying I'm trying to pick your brains that's interacting with a woman that's totally different talking to a woman it's completely different than following a woman around in a story isn't it look how fancy goes into it look at that swim to it would vigor and enthusiasm that's the point I'm trying to show you it's vigorous things certain things and thank God they gave you a decent amount of napkins but if I brought my mother's bib in the one with the snaps and the velcro where it doesn't have a lobster on it that's why I can't wear I don't think they expect anything less about a way how are the beers I only had one of the ones I have to have the other one I think the last one I gave you had okay had the Genesee and then yinling you still having a refrigerator yes yeah that's what I had last night when I was on the show I was on two boo shows last night we did cream a tailor you're familiar with the Taylor New York State wine country Taylor I've heard of Taylor wine cream sherry you know cream sherry's they call dessert wine I don't do a pair of teeth they call it no no it's not a core it's not a cordial liquor my mom used to like oh what's the green thing oh yeah you might say no no it's all sugar no I can't stand that I can't stand those liquors no and you learn yeah any for my leech is in order now we learned about her hoarding anything we learned about her this guy anything any wine over 14 percent alcohol is considered a dessert wine a sip and wine like me Port sherry I thought you cared about booze I don't care about alcohol content it's just fun to laugh I like pure grain of 196 a sip and wine you know most booze is like black or whatever 86 booze I did find out something from a gentleman from Tennessee there's a reason why the cops go after moonshiners out in the woods getting their taxes well he said it's because that some of it is so high in alcohol you can risk your health or go blind yeah but but yeah but it's dangerous may God pure grain alcohol you imagine the alcohol content 196 it's deadly it's deadly pure grain alcohol is nothing but problems and it's completely deadly isn't it yes oh yeah yeah we had a great party at my frat house in Tennessee oh my god now you are like they said the doctor said you're not dead as you are you are in such incredible shape for football well look at look at your results from the oncologist fabulous numbers you never lost your hair during chemo another great thing so the oncologist says so far it's it's really working your new therapy yeah football and baseball has ever ever been around like mine I knew that when the NFL and and William Morrow was a victim of office politics in sports have I gone through an awful lot of politics my entire life in sports probably jealous of you that's why I knew that I've always been good to me in college all it's like the Wild West Jim yeah they're eating instead of coming up and challenging you with a gun I'm faster than you guys are gone sitting eating with friends and come up and you say I could go harder for you I said nobody care and what about the coast saying he hired his son-in-law that time that story about set of using instead of having you try out for the team school that was high school when I signed up for football that's how I'm here to start a football position as a quarterback I've already got my quarterback I said you've never seen me throw he said it doesn't matter I told my dad how do I fight that it doesn't matter it doesn't matter you've never seen me throw and I did know so what Jackie Jackie says that you should be getting it in real soon so it looks good shit when I missed him on a few outlasts in the gym when we finished the practice for me Kenny came running over he was all state in premise for premise high school he holds the record of brushing he's gone he could be a suicide but look he came over like this dynamic inertia he said I can't keep it still the nerves are all numb and they turn them over the entire back was way buffed out from the impact of the ball hitting during the process stand he said look at this my god look at this holy shit he said I have no feeling when I knocked Fred down they thought he was gonna go to New York yeah what do you so so that so a good a good game plan would be like in it to go to Gabriel's like in the early in the early afternoon right well not early afternoon early evening hours five to seven five to seven and but but Sunday it's free to close it right on Sundays okay but they still they still serve dinner at like after seven if it's 7 15 yeah like you know but they have special Wednesday at prime rib night right but you can get fried calamari probably in any any well as a main course or whatever but I think if you want to the five or six dollar dollar pricing special it's very happy hour five to seven yeah prime rib no not primary the calamari calamari yeah yeah of course when but Wednesday is prime rib night when it is especially yeah well I just I just want you to know when something's a blackboard special and discounted sometimes a restaurant will give you smaller portions unfortunately they won't because they don't people see that say fuck you are lucky you know who does that Fridays down move 17 like if you give him a coupon never give a coupon be before the you ordered a food I know but that's why they have a small print please give coupons to wait for so whatever before ordering I said right about that because I used to go there with my sister my mom and it was proven without a shower of doubt that's not fair even the vegetables were cut in half portion that's not fair it was baby back ribs one of your favorite you like baby back barbara came there he said it's true is a lot smaller when he got it don't let me go back special he said it was a lot smaller look old TV he's on yeah all blackjack Mulligan has really nice a lot of these restaurants baby back you were smaller portions when you presented coupon yeah you find out and they love to push expensive cocktails that's why I don't ask for ice Jimmy I told you what I used to get my martinis I said ice on the side even the dance clubs pull that that scam I tell them ice on the side because they just give you less of everything it's actually ice cubes a cheap filler well my god of course you're getting less hip you don't order you know that burritos most burritos are low to a white rice well so you get less meat now now there's a new thing on the food network people are putting french fries on top of hamburger patties no in the sandwich no that's been done for forever Bobby play says every hamburger needs to be crunched five that means putting potato chips on top of a hamburger right you can put anything Bobby flay hamburger you tell me one thing you cannot put on a hamburger you know what that's what one thing you can't put on a hamburger you can put whatever your taste whatever you want onion rings you know Bobby flay no Bobby Flays burger paradise you know what his burgers are they're only six ounces when I go to all these I get the eight out they're all eight ounces I think was five guys one for two years I'm sorry one third pound two years in a row these are five guys one third pound is not a bad size I've heard uh I hop is that new burgers are incredible eight I think they're eight ounces to be honest with you this new these new burgers at I hop is selling now are they actually are they great they look great on a commercial the paper in the USA today said their sales have skyrocketed and I have to be honest with you I hop blows away Denny's it really does Denny's I've never been impressed when I was in Florida South Florida I went to Denny's quite often and then I've also went to I hop up here and there's definitely definitely a difference and that's that you're about ready to go right yeah see all you'll learn things from this yeah you're gonna get some food yeah this has been a mega lab 21 production


So, applying Akutani's fixed y n theorem, we are going to get to Nash's theorem. So, this is Nash 1. There exists a y 1 star to y n star such that j i of y 1 star to y n star is less than equal to j i of y i comma y minus i star for all y i in capital y i and for all i n n. This is what we are going to prove. So, the proof idea is going to be to simply apply Akutani to R. So, what did we had just establish that y is a Nash equilibrium if and only if y sorry y star is a Nash equilibrium if and only if y star belongs to R of y star. That means if and only if y star is a fixed point of R. So, to show that there is at least one Nash equilibrium, all I need to do is show that R has a fixed point. So, and now to show that what how will we show that R has a fixed point, we will show that we will apply Akutani's fixed point theorem which means that we are going to show that if I take phi as R, all these hypotheses are satisfied. So, this is that is that is going to be the proof idea. So, we need to check a few things. So, let us go through them one by one. So, in order to apply Akutani's fixed point theorem to R, we need to check that the domain of R is close convex and bounded. We need to check that R actually maps that set the domain to subsets of itself. We need to check that R is convex valued and we need to check that R has a close graph. Once these once these are checked Akutani will then give us that there is always a fixed point and hence there is always a Nash equilibrium. Alright. So, firstly what is the domain of R? So, domain of R remember was of R is y. So, R maps R maps y to subsets of itself and so R is in fact mapping from a set to subsets of itself. But what but is this domain closed and close convex and bounded. So, what was y remember y was simply the product of these yi's right. y was the product of these yi's. Each yi itself was a set of probability distributions on a finite set. So, probability distribution that means basically it is every component is between 0 and 1. So, the entire each yi is basically in some box of dimension 1 lying in some box of dimension 1 in its own space. So, y being a product of all of that is in a hyper box of higher dimensions, but at the end of the day it is in some box or in some bounded set in its own in its space right. So, therefore so since y so since yi's is a set of is the set of probability distributions the set of probability distributions on Si this is in fact. So, this set is therefore yi therefore is bounded it is also close because you take the take a sequence of probability distributions it will converge to and if it converges then the limit will also be a probability distribution why because you can see that if I take vectors yi say yi k these belong to y capital yi and if I take their limit as k tends to infinity sorry take the limit of k as k tends to infinity as yi suppose they converge to some yi then each of these belonging to yi means that this equals 1 for all k it also means that this is greater than equal to 0 for all k and therefore if I take the limit as k tends to infinity the limit should also be should also have this property that one transpose that limit would be equal to 1 and that limit would be greater than equal to 0 which means which effectively implies that that one transpose yi will be 1 and yi will also be greater than equal to 0. So, in other words therefore so yi is bounded because it is these are probability distributions and they are all this it is closed for the same reason and then what about convexity you take two probability distributions and take a convex combination of those two probability distributions that is another probability distribution right you mix two probability distributions you get another probability distribution. So, which means that it is its convex another way of looking at this is that remember yi is simply defined by linear equations and inequalities. So, therefore it is a convex set and it is also therefore a closed set but because these they distribute the what it defines is a probability is the set of all probability distributions it automatically is bounded as well. So, in other words capital yi is closed convex and bounded ok alright. So, once capital yi is closed convex and bounded therefore yi is also being a product of such sets is closed convex and bounded. So, with this then what we have ticked off is this we have ticked off the first part. So, we this one we have a we have a function that maps a set to subsets of itself and it is a domain to subsets of that domain and that domain is a closed convex and bounded set alright. So, the first this is done now let us go to the next one. So, now we are we are looking at this one we need to ask is the function is their set valued map convex valued ok. That means what we are asking is R convex valued ok. Convexity I just argue I just said that it follows because you take convex combination of two distributions it is a convex it is a probability distribution ok. Alternatively you can see that y here you see yi is just defined by linear inequalities and equations. So, therefore necessarily for you know yi is a convex set alright ok. So, now we will be showing that R is we will show that R is convex valued it follows trivially actually that if A and B this is a standard property that if A and B are convex or they have A and B are convex or closed or bounded the product will also have inherit that property trivially actually. So, I just write I just quickly write the proof here. So, take any two sets A comma B or convex and take now you take a point and take a point take two points x1 x2 in 8 cross B this one you let us write this as x11 x12 and this guy let us write it as x21 x22. So, now if you take a convex combination of x1 and x2 you take alpha times x1 plus 1 minus alpha times x2 where alpha is in is in 0 1 ok. So, this is therefore now alpha times the first you know you get a convex combination of the first component.  So, this is in A this the lower one is in B the A is because A is convex and B is convex and so, therefore, the product is also in is in A is in A cross B no, no, no. So, the multilinearity. So, this is the set of all probability distributions the set of all probability distributions is convex if you constrain the set of probability distributions like through a Markov chain or something then you do not then it is not necessarily convex alright. So, now we want to show that R is convex valued. So, for this let us recall go back and see what R was. So, R was the product of R R i's and each R i is the set of best responses of player i ok. So, R i of y minus i is the set of best responses of player i to y minus i ok. Now this what we want to so, it is to show that R is convex valued what we will show is that each R i is convex each R i is convex valued in other words for each for if you fix a y if you fix a y then for every i and if R i of y minus i is going to be is a convex set ok. Now, why is this a convex set is why is the set of best responses always a convex set ok. So, for that we need to just carefully look at what this what is actually happening here ok. So, and this is once we understand this the next part will also follow very easily. So, if you look at the cost of player i that is listed here that is written here it is it is this is it is this expectation right. Now what we are doing is when we are evaluating the best response what we are basically saying asking is we are fixing all the y minus i's ok. So, we are fixing the mixed strategy of the other player and asking what is the best that this guy can do. So, if I fix everything other than y i here all. So, fix all the y j's other than y i then what sort of expression is this in y i it is linear in y i right. So, if you look at this expression as a if you see it is as a function of y 1 to y n it is actually it has this product of all the y's involved ok. So, it will in fact be a polynomial in y 1 to y n right because it will be a polynomial each term involving a product of n n monomials right ok. So, this is going to be a polynomial, but if I fix the y minus i then it is linear in y i clear. So, which means that is what are this these best responses the best responses are actually solutions of a linear program they are the ones that minimize this linear function over all the y's right. So, in other words this here. So, now let me go back here r i of y minus i is simply the solution set of this linear program right it is r min of g i of this over y i in capital y i ok and this here this chap here this guy is linear in y i ok. Now, if this is linear the objective is linear in y i what about the constraints which is which is this capital y i capital y i itself is defined by two linear constraints is defined by the requirement at y 1 transpose y i is equal to 1 and y i is greater than equal to 0 right. So, if you put together this these two facts the objective is linear in y i the constraints are also linear in y i which means then that r i of y minus i is simply a solutions is the solution set of a linear program or a linear optimization problem. Now, if this is a solution set of a linear program then what is this what do we know about the solution set of a linear program we know that the solution set is always going to be if there are two solutions to a linear program then every segment the entire line segment joining them is also going to be a solution right which means that if you have two points in r i of y minus i then the entire line segment joining them is also in r i of y minus i right which means what which means that if you have two points in r i of y minus i the convex every convex combination of those two points is going to be in r i of y minus i which means r i of y minus i is actually convex right. So, if you take two points in it the entire segment joining it is also in that set which by definition makes the set convex. So, which means that the r i of y minus i is actually a convex set is convex and it is convex for each value of y minus i for each y minus i.  So, what does this mean then therefore, then r itself r of y was itself r 1 of y minus 1 till r n of y minus n and each of these guys is a convex set each of these factors in this product is a convex set. So, therefore, this r of y is also a convex set and this is exactly what we wanted to show we wanted to show that r is convex valued right which means that for every y r of y is a convex set means that r is convex valued ok. So, let us go back to Cacutani again and let us say what he let us see what he has to say. So, we have now knocked off this we have shown that r satisfies this is that it is convex it is convex value.


When it comes to assessing the performance of a machine learning algorithm, there's a lot of different approaches that you can use. The one that we've been using in previous episodes is the area under the receiver operator characteristic curve, or just simply the AUC, the area under the curve. Unfortunately, the AUC is a bit of a blunt tool. Say we had an AUC of 0.7. Well, there's a lot of different shapes of curves that would give you an AUC of 0.7 to suggest that those models all perform the same. However, you might expect that at a specific specificity, one of those models might perform considerably better than the others. So my goal in this episode is to explore an alternative approach that will allow us to look at specific specificities to compare across our models. If we were to use a machine learning algorithm to predict who does and doesn't have colon cancer, as we've been doing in these recent episodes, well, we would get an output from our model that is a probability that the person has cancer, so by default, all of the reports and metrics that are given to us by the software assume a threshold of 0.5. So if above 0.5, we assume the person has cancer. Below that, we assume they're healthy. Well, 0.5 isn't necessarily the best value. It seems to make sense, but again, it's not necessarily appropriate. So what we need to do is figure out what is the appropriate value. So as an example, in the United States, for the fit result, which is a metric of the amount of blood in a person's stool sample, if you have more than 100 units of blood in a stool sample, then you're referred for colonoscopy because you're at higher risk. If you're below 100, then you're considered healthy. So there was a balance between specificity and sensitivity, again, there's a trade-off there, and that based on that balance, the smart people figured out that 100 was the right threshold. My understanding is that if you go to Europe, they use a different threshold, maybe like 50 units. And that's, again, a different weighting of what's important in terms of sensitivity and specificity. So it becomes very difficult to compare different models and different approaches when they all have different specificities or different sensitivities. What is necessary is to pick a specific specificity or sensitivity, and then let that other parameter vary so that you can more directly compare the models. We've been talking about the area under the receiver operator characteristic curve, which again, this big composite value. But what we should do is go back and actually look at the curve. Because if we looked at the curve, we could pick a specificity, and then by tracing that line up, we could look at the different sensitivities of our various models to compare our models performance directly. That's exactly what we're going to do in today's episode of Code Club. We've got a lot of ground to cover, so let's get to it. You'll recall that we've created two different types of models. One was L2 regularized logistic regression, and the other was a random forest. For each of those two modeling approaches, we use different sets of features. In one set of features, we used only the microbiome data. In another set of features, we use microbiome data plus the fit. So between the two different types of features and the two different types of models, we have four combinations. So the output that I want is a figure with four receiver operator characteristics. Now the hitch here is that we have 100 seeds for each of those four modeling combinations, which means we have 400 total receiver operator characteristic curves that we somehow need to synthesize down into four curves. So there's a lot to cover, and as I mentioned, we need to get moving here. The first step that I want to take on is building a single receiver operator characteristic curve from one of those 80-20 splits. Again, we took 80% of the data, trained the model on that, and then evaluated the best fit on the held out 20%. So let's look at the sensitivity and specificity curve, the rock curve, for one of those example files. I will do read RDS process data forward slash L2 genus 1.RDS. Again, we're going to come back and read in all 400 files eventually. And so again, we see all this output here. I'm going to call the output of read RDS model. And then with model, we can then look at trained data, trained model rather, and we can use that as input to the predict function. And so we'll take that model and we'll apply it to the test data. So we'll do model, dollar sign, test data, and we'll then do type equals probes. And what type equals probes we'll do is we'll output the probability that each sample in the test set. So again, we had about 500 total samples. So it should be about 90 samples or 100 samples in our test data. Because again, we did an 80-20 split that it will output the probability that any sample was a healthy or had a screen relevant neoplasia, again, think cancer. And so again, if we look at that, and I should be prob, not probes, we then get this data frame that has the 92 or so different samples, as well as whether it was a healthy or an SRN. I'll go ahead and call this prob for the probabilities. And then the other bit of information that I want to get as well is the observed data. What was actually observed? What was the classification clinically, not by the model? And to get that, we can then do observed as model, dollar sign, test data, dollar sign, SRN. We then get all those classifications of the people. And so we want to bring these together because we want to look at the probability of an SRN as well as the observed. So we can then look at the sensitivity and specificity of that probability threshold for making the best possible classification. And so I will make a variable, I'll call prob obs, and we'll then do bind calls. So bind columns with prob, and then observed equals observed. And again, if we look at prob obs, we then see we've got those 97 rows. And we've got two columns of probabilities for healthy and SRN and the observed. I'm only interested in the SRN and the observed. So I can go ahead and then do select, SRN and observed. Looking at prob obs, we get those two columns worth of data. Again, my strategy when I'm working with a whole bunch of data is to simplify things as much as possible, break down the individual steps. And so we're working with this one file here, but eventually we'll then want to ramp it up to all 400 files. What I'd like to do now that we've got the probability thresholds and the observed diagnosis is I want to go ahead and figure out what is the sensitivity and specificity for each of these probabilities. I'll take prob obs and I'm going to do an arrange. So I'm going to sort the data frame in descending order by that SRN probability column. And so again, we see that at the top, we have very high probability of SRN 0.67 and all the way down at the bottom, then we have 0.35 low probability of an SRN. But of course, we see that there is an SRN there, right? To calculate the sensitivity and specificity, one of the things we need to know is for each of these thresholds, each of these probabilities, how many positive values are there at or above that threshold, as well as how many negative values are there at or above that threshold. So if we pick 0.6356 right here, the second row, then we would have two positives and no negatives, right? So those two positives will be true positives. And if there were any negatives, so healthies above 0.635, those would then be false negatives. So to calculate those, I'm going to use a trick, which I've used before, that leverages the fact that a true value has a value of one and a false value has a value of zero. And I can then do mutate. And I'll create a variable called is SRN. And I will then say observed equals equals SRN. And maybe I'll put that in parentheses, just to make it a little bit more clear. And so now we have that column that's got a bunch of truths and falses. What that allows me to do is to create a variable I'll call TP for true positive, and then use the function cum sum, which is the cumulative sum of those rows in is SRN. And now what we see is that we have, these are all true positives, right? So at this value or above, we have four true values. And so the number of true positives at that threshold is four. What we need to handle is this fifth case where we have a high probability, but it's healthy, right? And so that would then be a false positive. And so we can then do TP equals cum sum. Exclamation point is SRN. And so exclamation point means not. So if it's a true value becomes false, false will become true. So that false will become true. And then to the right there in our FP column, we will then have a one. And sure enough, we get that one in that column. And that is our number of false positives. So we have the column for the true positives and the false positives to get the sensitivity and the specificity. We need to know the total number of positives and the total number of negatives in the data set. To get that, we can do prob obs. I'll do count on observed. And that gives us healthy and SRN the number of ends. So I'd rather have healthy and SRN be columns so they're easier to deal with in that code chunk below. So I'll go ahead and do a pivot wider. And I'll do names from equals observed and values from equals N. And so again, we now have this table that has one row healthy and SRN. And I'll go ahead and call this total. So now what we can do is we can say sensitivity equals TP divided by total dollar sign SRN. And so again, this gives us our different sensitivity values. So instead of directly calculating the specificity, I'm going to go ahead and get the false positive rate, which will be FP divided by a total dollar sign healthy. If you needed, you could then say specificity equals one minus the false positive rate. Great, we can go ahead and clean this up a bit. And so I'm going to go ahead and do a select on sensitivity, specificity and FPR. And this gives me the three columns that I'm going to need going forward. So at this point, we have the sensitivity and specificity data we need to create a rock curve for this initial file. But again, we've got four different models. And we've got 100 seeds from each of those. So it's time to begin to think about how we're going to kind of compartmentalize this or chunk it apart into functions, so that we can easily replicate this across our 400 RDS files. The first thing that I'm looking at is this chunk of code down here that we use to calculate the sensitivity, specificity and false positive rate. I'm going to go ahead and cut this and put it way up here. And I'm going to use this to create my first function. And so we will call this get sends spec, and I'll call it lookup. And I'll explain that in a moment and function on data. And we need a closing curly brace down here. And we can go ahead and tab that in. And again, instead of prob OBS, we want data. And so we can test this with get send spec lookup on prob OBS. And we get that same output. So I can then say send spec lookup. And that is going to be again, that function we just ran down here, which was that get send spends, get send spec lookup, good. And so now we have this variable, send spec lookup. And what I'm trying to think about is that I'm going to have the same data, but for 100 splits for each of our modeling comparisons. And so I don't anticipate that I'll have the same specificities for all of those, right? So imagine that I had, you know, say 0.97 here as a specificity. Well, what would the sensitivity be? Well, it should be 0.133. Well, how do we how do we get R to know that? I think this is going to take a few tricks. And I think I have a pretty good idea of how to do it. We can go ahead and take send spec lookup. And for the time being, I'm going to do a mutate on diff. And I'll do specificity minus x and x is going to be the actual specificity I want. So I'll do x equals what I say 0.97. And now we can look at our specificities. And we've got this fourth column, which is the difference, right? And so there's a point at which it goes from being positive to negative. And what I really want, though, are those specificities that are positive, because then I want the largest sensitivity. Does that make sense? So again, what we can look at are the diffs that are positive, right? So that's these four rows, because those are all positive. And what I want is the largest sensitivity. And so that's going to be the sensitivity I want to output. So instead of using this diff, the way I've written it, I'm going to write it as a filter instead. And I don't need diff, but I can do specificity minus x greater than zero. And again, I get these rows. And now what I can do is top n on sensitivity. And I can do n equals one. And then I get one row out at the bottom here, right? I'll go ahead then and mutate my specificity column to be x and my FPR to be one minus x. So this seems like it could be a nice encapsulated piece of functionality that we can make a function that would take in that specificity threshold, right? So we can then do get sensitivity function. And then we'll feed it x, we'll also feed it in our data frame. And so I'll call that data. And so then data will be here for get spends, look up, just kind of add some white space here and close that out. And good gets sensitivity on let's do again, 0.97 just to make sure we get the same value.  And we get those values that we saw before. And again, we could change this to be say 0.9. And again, get a different sensitivity and false positive rate for that 0.9. Okay, so hopefully you're seeing where we're going with this, and that we could actually then iterate get sensitivity over a whole bunch of values of specificity, right? That's exactly where we're going. So I'm going to go ahead and grab this function and move it up here. I always like to have my functions at the top of my code. And again, what we could then do is we could have a specificity vector that will be seek 0 to one by 0.01. And we could then do map DFR on specificity, get sensitivity. And then we'll also give that sense back look up. And so then that will create, as I said, a data frame that's got all of my specificities, as well as all of my sensitivity values. So something I'm noticing looking at this is that we have 287 rows, should only have 100. And that we have, you know, some specificities appear duplicated in here. So let's go ahead and take that 0.88. And let's go ahead and see if we can't do some debugging here. So if I replace that 0.90 with 0.88, we do get two rows out. So to solve that, we can come back up to our get sensitivity function. And I can add a distinct line. And that will only return distinct or unique rows from our data frame. So now if we come back and we rerun that at 0.88, we get one row. That's great. We can go ahead and delete these. And let's go ahead and rerun map DFR. And now we get all we get 100 rows in our data frame for those 100 different values. One other problem I noticed is that this ends at 0.99 not 1. So again, if we come back to our code, I think we can fix that by doing greater than or equal to in get sensitivity. And then if we go ahead and rerun this, we now have the sensitivity specificity for all values of specificity from zero to one going by 100. So now what we'd like to do is then replicate that across all 400 of our RDS files. What I'm going to do because we currently have lines 31 to 44 as just kind of freestanding code is I want to turn this into a function, right? And so then what we could do is we could take this function and we could use map DFR to iterate that over all of the files and life will be easy, hopefully. Okay. So what I'll do is I will say get pooled sends spec. And we will then give it file name. And I'm also going to give it the specificities. All right. And so we'll do a curly brace there. And we'll grab all this code, tab it in a notch out of line to get some nice white space. I feel like Bob Ross when I'm talking about white space. And so I'll grab that specificity and bring that down here. And so now we have this function. And we can replace this with file name. And this then becomes file name. Right. And so that all should be good. Create that function in memory. And so what I can then do is get pooled sends spec on this function. And we'll also then give it specificity. And I'll define specificity before get send, get pooled send spec. We now have a function that we should be able to iterate over all 400 of our data frames. One little thing that I noticed, though, is that this data frame that's outputted doesn't tell us anything about the type of model that it came from, right? So we could theoretically get 101 times 400 rows and not know where it came from. We need to go ahead and mutate this to go ahead and put in a tag indicating what type of model was used to generate those data. So at the end of this map, DFR, I'll go ahead and do a mutate. And I'll say model str replace on the file name. And we'll do, let's see, processed data forward slash. And then I'm going to capture the first part of that. And then we'll stop with the number. So we'll do back back D to match a digit, the star any number of digits RDS. And we'll replace that with back back one. So again, back back one is what was matched inside those parentheses. And now we see we've got L2 genus here in that final column, we're now ready to iterate this function over all 400 of our RDS files. To get those RDS files, we can do list dot files path equals processed data.  And we'll do full dot names equals true. And so now we see we've got those 400 files, right. And so that all matched. And we're ready now to feed this in to map DFR. And we will then give it our function name, as well as specificity. Okay, processing those 400 files took about five minutes to run. So my code certainly isn't fast, there are probably ways that we could speed it up a bit. But for now, I'm just going to keep going. But one thing I totally screwed up on was I forgot to save the output to a variable name. Well, you have been very patient, and you are still watching this video. So I am going to give you an extra special tip for watching all the way through.  Well, this is something I do very regularly, right? So there is a special variable called last dot value. What do you think last dot value holds the last value, right? So I'm going to call this pooled sends spec. And I'll do period last period value. And so now pooled sends spec has all that stuff, right? So this isn't totally reproducible. You know, I'll still, you know, at the end, perhaps want to rerun everything altogether just to make sure it works. But you know, it saves me another five or six minutes rerunning everything. So again, that that cool tip again was period last period value to get you the output from the last statement that you ran in the console.  Good job. Now we'd like to go ahead and plot our data. But before we can make the plot, we need to go ahead and group our data and summarize it grouping by the modeling approach, as well as the specificity. And what I want is a line for the median sensitivity, as well as the interquartile range on the sensitivity. Let's go ahead and get to that. So we'll go ahead and pipe this into a group by model and specificity, sensitivity equals median sensitivity. And I'll do L quant quartile, quantile on sensitivity and prob equals 0.25. And we'll get the upper quartile by doing prob 75. All right. And then I'll go ahead and do a dot groups equals drop. Right. And so now what you can see in the output here is that we have the model, the specificity, the sensitivity, the lower quartile and the upper quartile. And we're in good shape. Now we're ready to plot the data. So we can go ahead and feed this into ggplot aes x is going to be one minus the specificity, and then y is going to be the sensitivity.  And then we will then pipe that into I guess it's not pipe, we'll add that into geome line. And we will also then do geome ribbon. And I want to do a color, right. So I will add to this another aesthetic, which will be color equals model. And maybe I'll go ahead and put these things on separate lines. And I'm going to put the ribbon behind the line, right. And I think I will also do an alpha, let's do 0.25. I don't want the ribbon to be the same color, the same intensity color as the line, I want the line to kind of pop out, right? We'll save this with gg save to figures. And I'll do rock curve dot tiff width equals five height equals five. Wonderful, we have our plot with our four rock curves. One thing I noticed is that the ribbon geome ribbon uses the fill for the color of the ribbon, not color. I think the color might actually be the the boundary. So let's go ahead and then do fill equals model. And that didn't change anything. And so it's interesting that the legend is giving me the right colors. It's giving me the fill as well as the border and the line. And what I think is happening actually, is that if you look at my summarize statement, quantile is operating on this sensitivity, not the sensitivity of the full column. So what I need to do is I need to grab that line. Pretty sure this is what happening, what's happening. And I need to put the sensitivity down below. And again, we have our ribbons of color, we have our solid line. It seems that the color aesthetic with ribbon is the border of the ribbon. I don't want that. So let's go ahead then. And I will go ahead and put fill model down in an aesthetic for geome ribbon. And I will do color model as an aesthetic for geom line. That looks a lot better. One other thing that we could perhaps do, let's go ahead and do theme classic. And we'll also move the legend to be inside. We don't need all that white space over by the legend. So we'll do theme legend.position equals C. And then we give it coordinates on the x axis and the y axis. So I'll do 0.8 and 0.2. So it looks a lot better moving the legend inside. It gives us kind of more space for the plot within the panel. One thing I'm noticing about this that I want to check out is that the line, you know, if you kind of look at this purple line for the random force genus with fit, is that an angle? Usually rock curves are stair stepped. So what I want to try, something I haven't done a lot because I don't use a lot of these types of plots. And so instead of geom line, let's do geom step. So geom step instead of geom line will give us a step shape to our curve because it shouldn't be kind of a diagonal line. Sure enough, now we get that stepped feature to our curves that you would expect for something like a receiver operator characteristic curve. So that looks a lot better. I don't use geom step a whole lot. And so it's kind of cool to see it being used here. But I think that is the more appropriate geom over geom line. Again, what I hope to show you was that we can look at a receiver operator characteristic curve. And instead of looking at the composite metric of the area under the curve, we can look at each of the individual curves, we can perhaps pick a desired specificity, you know, you could come to a specificity of say 0.8, then draw up and see what are the sensitivities of our models. And that then allows you to begin to have a dialogue about, you know, relative performance of models, as well as, you know, what's more important specificity or sensitivity. Those are trade offs. And, you know, in some cases, there are no right or certainly no easy answers. I know within the oncology field, they have constant debates over these types of things. What's more important specificity or sensitivity. And, and it's just hard, right? If nothing else shows you how I think through kind of a complicated problem, right of how do we take 100 rock curves that all have different values for their specificity. And how do we then bring them all together so that we can reflect the variation across the 100 seeds for four different models. Thanks again for sticking through. I hope you enjoyed that period last dot value tip. I know I use that a lot. If you've got other tips for other people, by all means feel free to put them down below in the comments. I'd love to hear what kind of tips you know about that you don't think anyone else knows about. And we'll all learn from it. Anyway, keep practicing with your R coding skills. And we'll see you next time for another episode of Code Club.