team. We're a relatively new team, and I wanted to share some of some information on the sort of direction that we're going in with our team and the work that we have been doing and thinking about doing and kind of just have a bit of a discussion about your thoughts on that, your feedback, your input, try and help guide this work from your experiences. So this is going to be a fairly interactive session. So if you're in Feedloop or YouTube, please join through to Zoom on Feedloop if you want to participate. I'll try and also look at Feedloop chat and like the etherpad and stuff, but I think if you're in, if you're in Zoom, that's going to be ideal. So the title of the talk was about improving mobile web for experience editors. The sort of more specific lens that we're working through is talking about content moderators. And so let me just quickly tell you what I mean by that. So for us. Oh, thanks, Andrew. Yes, I'll, I'll try to speak slow. So for us, content moderation means processes which review, report, or make changes to the contributions of other editors. So that might mean processes like patrolling recent changes, reporting bad articles, using admin tools, undoing edits. This isn't just limited to administrators, if I say moderators. I do also just mean active editors that are kind of doing things beyond just writing article content. I want to make it clear that to some extent, all editors are moderators, you know, most Wikimedia contributions are building on the work of others. And so to some extent, when I'm talking about more moderators or helping moderators, I'm also just talking about helping editors as a whole. So earlier in the year, we published some user research, we spoke to a few dozen editors across a bunch of different Wikimedia projects. And we published this report called content moderation in medium sized Wikimedia projects. So we hear a lot, I think, from the biggest Wikimedia communities about their problems in terms of content moderation. And we wanted to make sure that we were also thinking about how these sort of slightly smaller communities that are still needing to do a lot of content moderation work, and try and figure out how they were experiencing that and what problems they had, so that we as a product team could sort of figure out some ways of helping there. We focused that research in the Tamil Wikipedia and Ukrainian Wikipedia communities. But as I say, we did speak to editors from a wide variety of places, as well as reading a bunch of previous research and requests and that sort of thing. And the main thing that we learned was that mobile web was pretty bad for content moderation. So we have this project page on MediaWiki that you could go and check out. We basically decided that we observed that as we were speaking to all of these content moderators and active editors, they were either saying, I don't want to use mobile web because it kind of sucks or I can't do the things that I want to do on it. Or if I do use my mobile phone, I sort of switched to desktop mode and I pinch and zoom through the desktop skin. And I'm also an active editor on English Wikipedia, so that didn't come as a surprise to me. I always exclusively edit on desktop. But I think it speaks to an area that we're really sort of missing in terms of our focus because something like half of our page views to Wikimedia projects are on mobile. And there are a pretty wide range of communities, whereas many as 40, 50, 60% of editors are primarily editing for a mobile device. And if they're getting frustrated, if they're not being able to do the things that they want to do, then we're missing something in the sort of experience for them that we can hopefully improve. So I'm going to get to a point in a minute where we can sort of chat about this a little bit, but I just want to quickly explain some of the work we've already done over the past three to six months or so. The first project we picked was a fairly small one. There's this sort of overflow menu on mobile web when you're on a page. There's obviously an edit button, a history button, the watch list button, but there's some other functionality that sort of gets hidden away behind this drop down menu. It includes things like moving the page, page information, linking to the Wikidata item. It also includes administrator tools like protecting the page. And while that was already implemented, the only way you could get to it was by turning on the advanced mobile contribution setting. And so we thought that that maybe wasn't ideal and there might be a way to bring that to all editors so that they have access to those sorts of features like moving a page without having to go and find some setting that isn't hugely intuitive. And so we made some improvements to that menu. We added the block user button if you're an administrator and you're on a user page. And we also did some research with newer editors and found that menu wasn't very confusing to them. It was kind of understandable. And so we decided to roll that out to all editors. So now if you're a logged in editor and you have a user account, then you should see that drop down menu on every page now. The second thing we've been working on is preferences. At the moment on mobile web, there is no link to user preferences, which kind of surprised me when I realized this. You can open the search bar and type special preferences and then go through to the page. But that's obviously very unintuitive if you don't know that page already exists. And that's obviously a super critical page for editors. It includes functionality like changing your password, changing your email address, updating your notification settings, turning off banners. And so we thought that even though maybe it isn't specifically a moderation feature, that seemed pretty high priority to us. And so we're currently working on a new design for that page. And then we're going to make sure that it's linked in the interface so that users can kind of access that page and take a look at it. So I've seen at least one or two questions in chat already. And I just want to kind of open up for maybe quick 10 minute discussion on mobile web in general. I don't know if you want to put this in Zoom chat or in feed loop chat, but maybe raise your hand or something if you try to use mobile devices to edit on a regular basis. And then I'd love to hear what your experiences are. What problems do you face? You can feel free to either put that in chat or unmute if you're on Zoom. I do see there's one question already about whether we include blocking users in the scope of our team. And I would say not specifically. There are other product teams at the foundation like anti-harassment tools and trust and safety tools that kind of spend more time thinking about blocking users. We did end up adding block user to that dropdown menu because it was missing. But I think any further than that we're unlikely to sort of make changes to how users get blocked. So that aside, feel free to unmute or stick in chat if you have thoughts about your experiences of using mobile web at all. So Cintra Abdul says, I use mobile devices to edit. I'm curious what kind of edits do you make? Are they sort of big edits? Are they small edits? Are there typical activities you do on mobile that you don't do on desktop? Randall, go ahead. Hi, Sam. Nice to hear from you. Yeah, generally, as a more experienced sort of user, of course, there are sometimes when you have sort of don't have access to your computer and edit from mobile phone is sort of very necessary. But yeah, it's generally nice to see sometimes, for example, now a Discord server on Ukrainian Wikipedia, we have this user that only edits from mobile phone. He's kind of a new user. So all the time we hear some like sort of arguments why is this there, not there, something like that. So yeah, it's kind of it's important to get sort of this perspective of new user maybe. And yeah, it's it may not be very obvious sometimes. But yeah, yeah, generally, we need to make any improvements, I think, in mobile devices. Yeah, it's nice, nice work. Yeah, thanks. Yeah, absolutely. Thanks for sharing. Abdul Rafi, do you want to speak? Hi, Butch, go ahead. Okay, I posted my question on the chat. Actually, this is an ongoing issue that I've gone through. This graph chart, I've been already made about 1000 articles using this template and I cannot I am not comfortable undoing this. I'm using this graph chart and it's visible on the mobile browser, but on the on this Wikipedia app, it's it's not visible. So charts can be seen there. Gotcha. That's good to know. Yeah. And so this is an important point is that we're the scope of our team is to think about mobile web. So through a browser, the apps are kind of distinct and there are other foundation teams that work on those. But I'll make sure to take that to them and see if there's not already a fabricated ticket or something that they could look at. I'll make sure they know about that. I'm just scanning through there's a whole bunch of comments on on zoom. Yeah, Michael, you mentioned that you don't enjoy editing from mobile because it is kind of limited in terms of functionality in the interface. I wonder what kind of functionality do you find is missing? What is it you're trying to do that that you aren't able to do? For example, I handbooks, but when I try to switch into the editing mode, I seem to only have access to only the sand, the info box plus the intro. But the general article is not visible on the device, even when I try to pretend the orientation of the screen. Still, I seem not to really have the content, entire content of the article displayed. And as well, the font screen really is the interface is really small if you are really, if you really want to do like reasonable editing and any other projects or data. I believe it's something because font screens are always small in size. Yeah. Yeah, absolutely. I think I've seen discussion on that as well that on mobile you can only edit sort of section by section. So if you hit that main edit pencil, it actually tries to get you to edit that like top section and you can't edit the page as a whole. I think that's something we could definitely look into because yeah, I've also had that frustration when I tried to edit on mobile. Renvoi. Yeah, I just wanted to sort of find some maybe at least one example of things that should be maybe looked into. Yeah, that this user was sort of reporting to us. So yeah, it's interesting, but I don't really know how because I didn't really do it myself on mobile. But in fact, it is quite hard to find this sort of move from Wikipedia to Wikidata from mobile phone. Yeah, there was some mistake in sort of template infobox on the right side of the like, yeah, if you're on mobile, not maybe from the right side, but yeah, so user would try to sort of change that and yeah, moving to Wikidata wasn't as easy as maybe should have been. Maybe, maybe not. So yeah, that's something to maybe think for. Yeah, thanks for sharing. Jung Jin? Yes. Hi. Can you hear me? Can you hear me? Yeah, we hear you fine. Oh yeah. So I'm Jung Jin from Korea Wikipedia by the way. So I have several experiences that share I'm not sure it's gonna be maybe duplicate because I joined the session a little bit late. But yeah, so what I was feeling about personally about mobile web is like, I can, as an administrator in Korea Wikipedia, I cannot able to use my word of control the administrative access to the mobile. I know the editor side, I'm pretty sure other peers explained many concerns, but like the administration side, I know it is like the most major problem because like for example, if I want to protect or block, it is kind of like very messier interface so that I cannot able to using it and I cannot able to using like somehow the desktop mode. So like the how I using the Wikipedia mobile is like the I just like to manually convert it to the desktop mode and using it that way because of the administrative action. I know Wikidata is another problem, but I know previous person already spoken, so I'm not gonna do repeat again and yes. So the biggest problem also is the mobile interfaces basically just look like just like the reader readers interface and I don't feeling like that does not encouraging any people to edit or like that's not giving the throwing the message that you cannot you can able to editing or those those things. So I think that's kind of important concept in order to do in order to get abstract the mobile especially because nowadays many people are using just only the mobile not the desktop interface to just like the wrapping and the contribute over the project. Yeah, I think those are great points and it's got it segues on great to what I'm about to speak to. I just want to speak to one question that Amir put in chat about the strategy or planning to have a responsive interface that is you know not a separate website you know at the moment we have this dot m and there's this very hard line between sort of mobile and desktop. I think that's a discussion that we should have and where our team is talking to the reading web team who are doing the desktop refresh work at the moment. We've been speaking to them quite a lot about this work and I think that's something we're going to keep talking with them about. I think they'll be in the best place to work on this rather than our team we have quite a small team. But yes I would love to see that as well I think it would make considerably more sense than this sort of hard line desktop versus mobile paradigm that we have at the moment. But okay thanks for sharing your input I'm going to move on a little bit here and actually build a bit on what Jungjin just shared which is the mobile interface is very much a reader focused interface. I do personally agree with you there and I think that's because that's what a lot of the work has been focused on in the past on mobile web. We've historically at the foundation I mean been thinking primarily about readers when we've been making changes to the mobile site. I will say the reading web team did some great work on advanced mobile contributions a few years ago but we kind of want to build on that and see if we can make this better for editors as well. So one of the big questions that we have then is like how can we increase the number of mobile first content moderators and if reading is good on mobile and you know I think making your first edits is also pretty good on mobile we have the growth teams structured tasks that you can kind of engage with and finding the edit pencil I don't think is too difficult. But past that you know how do you then become a member of the community and an active editor and a moderator. And that question is partly spurred on by some data we were looking at. So we pulled some data on desktop first editors and mobile first editors so by that I mean editors who make more than 50% of their edits on desktop or mobile. First of all you can see that there is a lot more active editors on desktop than mobile which maybe is a concern in itself you know if we have half of our page views being mobile readers then I might personally expect that number to be higher for mobile. But even off those active mobile editors they are much less likely to participate on talk pages much less likely to participate in project namespace pages much less likely to have an advanced user group like patroller or rollbacker and almost no administrators primarily edit from a mobile device. And yes these are numbers from all projects combined. So of the $3,200 administrators 99% of them primarily edit from a desktop device and that number really stood out to me. It really feels to me like if you're a mobile first editor you should have you know something approaching an equal opportunity to become an administrator and be successful in your project and be able to help out in that way. But at the moment almost zero editors almost zero administrators primarily edit from a mobile device and if you look at mobile only contributors that last column so that's 100% of edits on mobile web then that's sort of even even lower across the board. And so this is kind of one of the questions we have is like we want to make mobile better for active editors but we also want to think about can we bring more editors through that pipeline from making their first edits to being an experienced active editor patroller administrator in a way that can then support the community more broadly. So as I said mobile first editors are much less likely to be moderators and administrators and again I want to spend five or ten minutes just having a chat. What do you think the barriers are for new editors in terms of using those moderation tools on mobile? And when I say moderation tools I mean things like reporting bad content that they see or undoing edits patrolling. What do you think the biggest barriers are to those editors? And how can we help mobile users discover editing tools and discover moderation tools? Perhaps thinking about this in a way that isn't going to increase the amount of vandalism for example you know not making it too easy to just go through and make very rapid edits for example. I'd have to hear your thoughts on that. Young June I see you have a hand up but I don't was that from before? Do you have a something to share this? But yes but the only thing I can add for a point is like some like a user maker gadgets are not like comfortable or an accessible via mobile like that's the point I can pop popping up. Yeah thanks. Yeah thanks. Yeah there's a comment in chat from Vera about it being hard to find out about tools when you're only familiar with the mobile experience and I think that's a great point. I think sometimes when we as experience editors when we think about it we we already know what tools are available and so we might go looking for those tools that we're familiar with but if you're a new editor and all you've done is you know make an edit and you haven't spent much time on the desktop site you don't even know what tools are available to you and so we definitely want to think about how to make that mobile experience inviting and sort of explanatory you know and set it up in a way that teaches you a bit about how all these tools work and there's a comment about how many mobile first editors are very active i 100 plus edits a month I guess that's a group likely to become admins. Yeah I think you're I think you're right and we haven't dealt into that data any further than just looking at those active editors but I would really like to speak to those very active mobile editors who who aren't moderators or administrators and see what they think a comment about having a twinkle like interface on mobile would go a long way I think that's a great idea that's something that we did look at one way we were doing our user research and I think even for desktop some of the twinkle functionality really is like very very useful and very important and I know that that's broadly an English Wikipedia only tool I know that other communities have imported some of it or tried to import it but yes I would love to see that and I think it would be especially useful for mobile okay well I'm going to move on from this slide there's going to be some more opportunities to to share your thoughts because I want to share with you sort of a theory that we have and this is a fairly new theory so I would love to please please be critical of it and and tell me what you think but we have a theory about how new editors become active members of their Wikimedia projects and it relates to four key pages and these key pages I've been referring to as kind of revision navigation they're the pages that help you navigate the edits that other users are making because we think that after an editor has made some of their first edits you know after they have copy edited an article they are interested in or they've done a structured task from the newcomer homepage at the moment I think they don't get then guidance on on where to go from there and we were sort of thinking about this and we think that there are sort of four key pages where a new editor learns some really fundamental things about how Wikimedia projects work and so these are the page history which lets you see every edit that is made to a page if you're a brand new editor that might be a novel interesting thing to you that oh I can look at the edits other people have made specifically they can go through to user contributions and see every edit that an individual contributor has made they can then from those pages go through to a diff and actually look at those individual edits and then of course they can use their watch list to track edits to those pages and so our theory is that those four pages are sort of core to graduating a user from they've made you know their first edits to they understand sort of how the wiki works a bit more broadly and how they can navigate those pages and see what other editors are doing and then maybe engage in content moderation if they see the undo button or they find the recent changes for you that sort of thing and so we've been thinking then and I'll show you some some things in this in a minute but we've been thinking then about okay if these four pages are important then what are they like on mobile at the moment and how could we improve them but first I just want to give a couple of minutes do you agree with this idea that these four pages are sort of critical to to becoming an active editor are there other pages or workflows that you think might be missing here I'd love to hear your thoughts so there's a comment in chat saying that when I give talks I usually show recent changes first and then show how page history and user contributions are filtered I think that's a great point we we haven't included recent changes here but I can absolutely see how that is a how that's a key element when you say that you show recent changes first is that is that brand new editors people who haven't edited at all or is that kind of to people who have started editing and Darcy says these aren't the only that scroll these aren't the only lists used for content moderation recent changes cleanup backlogs yeah I completely agree this definitely isn't isn't you know everything that an editor needs to do content moderation and we're trying to focus really on the first steps that they can take after making their first edits and making sure that there was sort of key core components are good yeah that's a great point and the recent changes is important but it doesn't scale well for very large workies I completely agree with that that's something we found in our user research when we spoke to contributors from the larger workies they said I need filters for recent changes I need auras I need you know tools like huggle for filtering edits and then we went to small workies they said recent changes is fine I can see every edit that has happened to my workie if I filter for anonymous edits I can see the last week all on one page and so yeah I absolutely see that Darcy asks the newcomer portal on the user page seems to work really well is that on mobile yes it is also mobile actually think it might have been developed as kind of a mobile first experience and then brought to desktop even and so mobile editors can get that that new home page and I think we've been speaking to the the team that develops that tool I think there are opportunities to for editors who have done some of those recommended edits sort of take them then through to more content moderation or reviewing patrolling sort of contributions because at the moment they're mostly sort of building on existing content I'd love to think about what those sorts of contributions might be like Nick comments that too much space is wasted on mobile so it gets sold destroying to browse through more than a handful of recent edits that's interesting yeah a lot of white space is something we've noticed as well and I think it's always a balance between and I think if any of you saw the talk earlier from the reading web team about the new vector desktop interface it's it's sort of a similar issue we we want information density for experience editors but also we want it to be navigatable and sort of clear for people who aren't so experienced and so it's it's constantly a trade off cool well thanks for all your input on this and I'm going to move on to the next topic which is assuming that these four pages are pretty key where do we start as a as a team in terms of making improvements and so the first step that we think is most important is improving the diff page really all of these pages and all these workflows come back to looking at an individual edit and seeing what what changed and in our view the current diff page on mobile doesn't really support a new editor in understanding what is happening the page hasn't changed for a very long time it was designed a long time ago as far as I understand it and at the moment it's missing some pretty key features and links so the one that the communities have raised a lot is the undo is missing you can thank a user for that edit but you can't undo it I know that some communities have a gadget that allows you to undo but otherwise that button is just not there you also can't get through to the page history you can't get to a user's contributions very easily your watch list is a little bit hidden here and so this really stood out to us as something that needs improvement yeah michael points out that the diff also doesn't work with red green color blindness absolutely not only is that diff kind of a mess to pass that was just someone copy editing a paragraph if you're red green colorblind then that's actually really hard to to figure out on desktop of course we have that two column view and it's yellow and blue and the obviously the position helps left and right but on mobile that green and red is the only way that you can understand the difference between added and removed text so just to be clear this is the current state this isn't this isn't our changes this is what it looks like right now and what we think needs improving um Darcy asks is the diff algorithm the same as used on desktop um it is different because on desktop you have a two column view and so um the algorithm only needs to to show you what was removed over here and what was added over here on mobile the algorithm needs to put those things together and as you can see from that example there show you what was removed and also what was added right next to each other um and I think it could be improved a lot um I've seen other inline diff engines there's a tool called wiki diff I think which has oh sorry wiki ed diff also uses inline and it looks a lot better than this one because it breaks up less of the words so you get more of a removed sentence and then more of an added sentence rather than being like every other word so I think we can make a lot of improvements to that that diff engine so that that's clearer to to look at and so uh having looked at this and and sort of decided that this isn't great and we could make some improvements to this we've started making some very early sketches this is you know absolutely version 0.001 um of of kind of our explorations here this is very early days but we made some very simple wireframes of what a new diff could look like on mobile so um some of the key things we were thinking about here is obviously getting rid of that red and blue uh sorry red and green diff change and making it probably yellow and blue given that that's how it is on desktop but we could also introduce some more elements there like crossing out the old text so that even if um color blindness is an issue you can still tell what was added and what was removed compacting some of the top of the diff so that um you can see the diff more easily um straight away making sure that there are links to the page history to user contributions and of course making sure that there's an undo button um on the undo button also we were talking about the fact that if you undo an edit and this is true on desktop and on mobile it opens the full editor so that you can make a partial change you know if you wanted to tweak the edit as part of undoing it um but we looked at some data and found that in the vast majority of cases when someone tries to undo an edit they undo the whole edit um making a partial edit is pretty rare and so we were thinking especially on mobile why not have it so that when you click undo you can just edit uh edit your edit summary and then confirm the undo right you don't have to open the editor at any point um though you can if you want to um so this is just kind of an early exploration we're not um we haven't tested this with anyone yet I think you're more or less the first community members who are seeing this um and so I'd love to get your feedback on you know what do you like or dislike about those sketches um how do you feel about the inline diffs and what do you think about that undo um process what what comes to mind for that I see that Amir has an immediate thought that this is dangerous do you want to expand on that if if I'm allowed to speak um there's a reason um revert and undo uh uh separate on desktop uh revert was much earlier revert was there when I joined in 2004 undo appeared around 2007 or so and it may or may not be the reason but uh 2007 is also the time um if you if you check by data uh when it became harder for new editors to join and stick maybe it's related maybe not but it happened more or less at the same time that the undo button appeared so there is a theory uh that uh that was one of the reasons it uh became harder to join uh Wikipedia so uh this will essentially give the revert permission to everybody uh so that's uh there should be some very careful thought dedicated to this yeah thanks for showing that I completely agree um I'm always wary about making it easier to do these sorts of things because on the one hand you have you know as an experience editor you might think yeah you know I wanted to be able to do this as quick as possible um but making it too easy is also dangerous because we know that the process of having your edit reverted is not a good one um when you get a templated message when you get that notification that says your edit was reverted you're much less likely to continue contributing and so I completely agree if we were to do something like this we would want to both be doing a lot of user testing and also a lot of data um monitoring to make sure that as we gradually tested it and rolled it out and ab tested it that we weren't seeing like a hugely negative effect there on uh on retention uh Mike how do you add a custom edit summary in that since the box is all filled with the automatic text uh yeah that is a question that we'll have to answer um I mean I think this would probably work potentially in the same way that it does at the moment where I think maybe if you click into that box it could the text could be left aligned right so you can add your comment at the end like on desktop um that's something we'd have to explore um like I say these are very early initial explorations and so that those kinds of questions I think we'll get into if we if we move forward with this there's a link in chat to a good gadget for partial reverts that's really helpful um I will I'll take a look at that um I think like other product teams we're often looking at gadgets and user scripts for inspiration um you know if if editors are already um fixing a problem themselves then that that's a good signal that we should consider that approach in the software by default um and so I want to just then uh open up a bit more broadly um given that we've got maybe five or ten minutes left um is there anything else you want to share with us about content moderation on mobile um are there any tasks that sort of stand out to you as maybe well suited to mobile um or maybe things that are just too difficult to do on a phone that that we almost shouldn't bother trying to trying to make fit into a mobile screen size um yeah anything else that you think our team should should be aware of fixing coordinates on nearby articles is a is a fun a fun micro task yeah oh and young yeah how this looks in a tablet um is a great question because um I think I think we often forget about tablets um we think about mobile screen size we think about desktop let me forget the tablet uses the mobile skin but obviously blown up a bit so um those will definitely as we start making you know better designs and um more detailed designs definitely tablet screen sizes will be a part of that um and yeah nick making the the link to switch from mobile to desktop more visible that's an interesting one because on the one hand I agree uh that that could be beneficial on the other hand I think ideally mobile would be good enough that you don't need to switch to desktop um we're actually currently implementing some some data logging so that we can monitor what uh where and when those clicks the desktop happen because I think that'll help us figure out which mobile pages aren't working um or don't have the functionality they should do um and so we will um we'll be thinking a bit more about that but and yeah michael I've heard that that link sometimes doesn't all work um so that's something we'll uh we'll also think about um okay so we have a few minutes left um there's just one final thing I wanted to say then um quick comment yes Darcy uh notes that even if mobile is good enough to never need to switch to desktop users should always have preference I agree I'm not saying that we should get rid of that button at all um certainly you should always be able to switch if you need to um so if you want to stay up to date with the work we're doing and have more input in this um we have a page on media wiki it's a moderator tools there you can find links to the project pages for the work I talked about that we've already done but also that we're planning to do um that work on diffs and revision navigation isn't quite up to date um today but I'm hoping to update it in the next couple of weeks to sort of have more of those wireframes and the things we're thinking about there um you can contact us directly probably the easiest way is to email me um but also you can find me around the wikis um we also have a newsletter which is brand new we haven't sent any issues of that out yet um but you can find that on media wiki at moderator tools slash newsletter please go there and sign up either with your own user talk page or with a community venue that you think would be interested and we're planning to send that out maybe once every few months um just with sort of updates on what we have done what we're working on and sort of where you can give us more input and give us more thoughts and feedback on on what we're working on so um that's all I have um I appreciate everyone's input I think this was a really interesting discussion I'm going to go and copy the entire chat into a document and make sure that I have that available to me to to read through again later um this was really helpful um to our work so thank you everyone for coming


Hi everyone, Joe for Jaspy's casebreaks.com coming at you with a Diamond Icons random number block randomizer featuring these teams right here. That's how it works. I think most of you here are familiar with it. Thank you. There are the numbers right there. Let's roll it. Let's randomize it. Six and a five. 11 times for names and numbers. One, two, 10 and 11th and final time. After 11, we got Rob down to Sean Healy. 11 times for the number is one, two, 10 and 11th and final time. After 11, we got five down to one. Now remember, zero gets any and all redemptions including one-of-one redemptions for just that, this group of teams down here. But one will still get live one-of-ones. Rob, you got five, Joel with four, Rob with zero and three, Dale with six, Jeremy with five. Joel with eight, Anthony with nine, Rob with seven and Sean with one. Let's order these by number and coming up in the next video will be the break itself. Jaspeyscasebreaks.com. Bye-bye.


Don't start without yourself yeah, I have a somewhat unusual background I started with a PhD in nuclear physics, but was never able to turn that into employment and so Through a variety of random walks. I ended up in the new medical school at McMaster Doing research in medical education pretty well unqualified to do so But over time I've learned a few of the tricks of the trade and over the last few years I managed to call myself a psychologist So nuclear physics that was that's quite a step Yeah, it was an easy step towards things like measurement and statistics because you had to know a little bit of mathematics to do physics but Learning how to be a decent psychologist and do decent research in psychology did take an awful lot longer and took a few very good Mentors like my dear colleague Lee Brooks yeah so part of the the research that That you do and that I Suppose in medical education that a lot of what it's based on is is categorization is diagnoses How do how do people generally diagnose? I mean it's really kind of a categorization problem, isn't it? So yeah, what what is what is categorization more generally and how do people how do people do it? Well before we get there, let me back up a second and give you an idea as to how we got into this mess I was originally hired to look at clinical problem solving and the idea was the diagnosis was not a categorization task It was really a matter of some kind of problem-solving skill And our initial quest was to go after some kind of general skills that people acquired as they became experts What emerged even back then in the late 70s was that in fact the general skill really wasn't very general At one level was too general and that everybody did it the same So that it didn't discriminate experts from novices at another level is too specific because success on one problem had very little to do With success on the next problem, right that led to thinking about well Maybe this isn't the way to characterize it at all and yes, you're absolutely right diagnosis is a form of categorization And I think for me personally the breakthrough was to work with people at McMaster who were really into what it's called concept formation or categorization and Some of the models about everyday categorization categorization really had enormous power in explaining medical diagnosis So to go back to your initial question Historically psychologists have been interested in categorization because it's the basis of communication. I Can't talk about a tree or a dog or a cat or a butterfly unless I have some senses And you have the same sense as to what the term butterfly stands for and so Categorization I think has a long history in psychology. I Think it's fair to say that initially the thinking was that there must be some implicit rules that governed each category But a moment's reflection on say a beanbag chair tells you that the implicit rules of chairs are unlikely to be necessary and sufficient Right, so what makes a chair a chair is normally someone would think Four legs, maybe wooden chair, but that that sort of rule might not be all Encompassing it might not define what everyone means by a chair I suppose no rule will encompass what everyone means by every kind of chair because you have to consider all sorts of possible legs Including none at all. Yeah, or cats or dogs or cats or dogs and so on I mean cats includes everything from house cats to lions and dogs include everything from Chihuahua's to Mastiff's And to come up with any general rule simply doesn't happen The two more prevalent views one is that essentially we average our experiences into a prototype And that prototypes are distinguished as having more of the features of the category and fewer of the features of other categories So we can play experimental games and find that a carrot is the prototypical vegetable and a robin is the prototypical bird and Penguins because they don't look like because they don't fly and they don't have feathers are very atypical So the older view is the prototype view that essentially our experience gets averaged into internal prototypes a More recent view and one which seems to play out really really well in medical diagnosis Is what's called the exemplar view that essentially says as you walk around the world you gather examples and every natural category as you learn you acquire examples and the the active Conversation or the active diagnosis ultimately amounts to matching the incoming information with some prior example and in memory That has many of the characteristics uniquely of the of the new stimulus So applying that to dogs and cats for example if I'm trying to recognize a beast that's running toward me It's probably not going to be on the basis of a prototypical dog or a cat but instead on the basis of The previous examples of dogs and cats that I've encountered before sure Glenn reger actually has a very nice example that he says suppose you grow up in the Yukon and all you see is Huskies How many Chihuahuas do you have to see before you recognize a trawl Chihuahua as a dog in the answer to one? And in fact, I I have a talk about this whole issue of medical diagnosis and one of the One of the key features and the one that grabs the audience Inevitably is the video of my year-and-a-half old daughter Where we show her playing the fish for priced toy with dogs and cats and BS skinner would be happy as can be because What she's learned is that if she puts that thing in that slot then mommy gives her a popsicle and tells her how wonderful she is But then we show her this an adult magazine a house and garden magazine Which contains a bunch of photographs and two of them have dogs in them and she points to the two dogs And what's interesting about that is that she can't say dog Right. She's a year-and-a-half old. She's pre-literate.  Yep So it only takes just a few examples to be able to get the job done. It seems so. Yeah, obviously I mean the mystery would be what exactly is that similar similarity matching How does that come about and I think then you can invoke More fundamental models of the nature of memory which is Associationist or connectionist which basically says that we are somehow in a very rapid and unconscious way matching on individual attributes And and seeking connections that way That's far too basic science and psychology for me to play with but that's I think a pretty decent model Yeah, and so what about your doctors so when we're when we go from everyday classification or Categories like dogs and cats and tables and chairs And you only need just a few exemplars to be able to get the job done Did the docks work the same way? So if you're learning about skin lesions or mental disorders or More complex types of categories. Is it just take a few exemplars there as well? well clearly medical diagnosis is a bit more complex in part because the Yes, for skin lesions we can imagine a picture that's kind of like looking at a picture of a dog But for something like multiple sclerosis, it's much more abstract than that secondly self-evidently the stakes are higher Right, we don't want to make mistakes and so what you see coming about in the clinical encounter It's what has been called since the 1970s the hypothetical deductive method Which is within a few minutes or maybe even seconds. We don't know the timescale a physician will Advance a number of hypotheses as to what the diagnosis will likely be and then we'll go on a systematic search to gather information By and large to confirm one or another diagnosis Clearly what differentiates the expert from the novice is not how many hypotheses or how early But what are the hypotheses and expertise resides entirely in generating better hypotheses? Even going back to our original studies in the 70s We found that that early hypothesis generation was enormously powerful Basically, if you thought of the right diagnosis in the first five minutes your chances of getting there was 95 percent If you didn't your chances were 20 percent Wow Yeah, and that then leads to the fundamental question that's kept me interested all these years Where do those hypotheses come from right and exemplar models are a very powerful way of thinking about that sure that essentially prior experience is available to you and So the idea then is that you?  Yes, certainly something like dermatology We've actually documented that a dermatologist looking at a skin lesion if he gets it right it takes them eight seconds Wow, there's not a lot of room for analytical processing in that process One of the a couple of games I play with clinician audiences quite often one is to ask them How long after you graduated before you felt you were competent? Now to the average person on the street You think that would be ruled out of order from the outset they're confident when we graduated them aren't they right? The answer is no The answer is routinely five years post graduation you put that with five years of training And then you say the ten years and ten thousand hours that everybody's that's now become common street talk yeah The second thing that's really intrigued me to illustrate the power of this method.  Yeah But a very recent study actually just published this year showed that You turn to the emergency doc and say is this one sick or not? And he makes a judgment in seconds and that is about an 85% accuracy in terms of predicting whether ultimately when they get to the Wards they end up with a very serious diagnosis The doc with his experience can recognize sick from a mile away So you mentioned that process of categorization and diagnosis and so on is is fast and accurate We can do it very quickly Is that all there is I mean there seems like as doctors it seems that they should be able to I mean That's what they do in medical school isn't it is is learn the rules and and They're they're very careful and deliberative in coming up with these diagnoses Sure I think in every clinical encounter there's really two fairly distinct phases the initial phase of hypothesis generation which Is so effortless it just seems to be too easy And then a second confirmation phase where they go back to the rules the 29 causes of anemia the signs and symptoms of pernicious anemia And so forth just to confirm that that's what they're dealing with now Because it seems so effortless that first phase is often called just pattern recognition and it's kind of denigrated It's people sort of think that you know, you're not really doing your work You're not working hard enough and in fact that whole idea that that first System and it's called a dual processing model than that's called system one that that rapid generation of hypotheses There's a kind of a prevailing wisdom that that's bad for that that leads to errors and that what we should do is teach people analytical processes to correct those errors and that the second phase is actually the fundamental phase in terms of reducing errors This has been popularized a lot by a guy named Kahneman who's got a Nobel Prize psychologist He's written a book that I think many people know it was on the New York Times best seller list for at least a year called thinking fast and slow His claim is just that that essentially fast thinking system one thinking leads to errors And that we should be cautioning people doctors and everyone else to be more Slow rational thoughtful and that that would reduce the errors that happen in human judgments. I happen to believe that's wrong And I've accumulated a number a bunch of evidence to suggest that indeed that's wrong We encourage one group of physicians working through cases to go fast We encourage a second group to go slow and take their time We find that the group that goes slower indeed takes longer on each case But study after shoddy study has shown that the accuracy is exactly the same in both groups So the notion simply by slowing down and being thoughtful and analytical You'll solve the problem and the errors will go away seems to be simplistic at best Sorry, I think we should wait just like that for that high pitch. There we go So if the goal then is For accurate judgments, I mean we see this across a few fields where The ideal is to slow down take your time make sure that you're doing it correctly. Yeah It seems fairly counterintuitive that that would actually that wouldn't be as good as making a rapid judgment as that the fast sort of judgment that diagnosticians are making are as accurate as Diagnosticians or decision-makers who take their time. Is that really? I mean is that yeah, it's almost like there's a trick in an egg thing If I ask you what's 12 times 12 the answer is a hundred and forty four It's rapid and it's accurate if I ask you what 17 times 17 The answer is I don't know but I guess I can work it out The answer is actually 289 by the way only statisticians seem to know But that's an example where the slow process is actually more error prone than the fast process It seems I agree And in fact the I think it's safe to say that the majority view is that the slow process is going to be Ultimately higher benefit in the long run, but that's against the evidence This has really been captured in medicine where a book called the reflective practitioner by a guy named shown Was published probably ten years ago and argued basically that if every physician learned to be reflective in the course of their process And at the end of the day looking back on things that the world would be a better place It plays out particularly well in health professions because like any professional they are basically autonomous They have clients they make very serious decisions about those clients And they have no peers in general looking over their shoulder to see whether they're right or wrong So there is undoubtedly a need for physician for practitioners or any kind of professional to keep up to recognize their strengths and weaknesses The notion therefore that reflection being able to reflect on your process is somehow going to achieve that Has real cash right sure Unfortunately, it doesn't have a shred of evidence Right, so the opposite are people good at assessing their own ability. I mean it is There's I've read a little bit in in the field a colleague of ours Kevin Eva has done some work on on the role of self-assessment And it seems at least given his preliminary work that That we're not very good at it. That is we don't seem to be We don't seem to be very good at assessing how well we can do Various things is that is that right? You've published it's not there's nothing preliminary about it Kevin should be here to speak to it But I think I can pretty well paraphrase what Kevin would say.  Yeah. Yeah, I think the evidence is utterly overwhelming that people can't self-assess the way out of the paper bag Sure, in fact Kevin has a nice demonstration of this if I think of all the people who are watching this video I'm gonna ask them all a simple question Are you in the bottom half of driving skills? Hmm now half of you should say yes But having seen this but perform having done this in live time probably one out of a hundred will say yes Right half of you by definition are in the bottom half of driving skills, right? The evidence is absolutely crushing that people Cannot assess where they're at. They basically start with the premise that they're at about 70% and then go up and down from there Hmm You're better than I am, but I'm about 70% so you must be 80% yeah, it was captured by a Psychologist whose name I sadly forget, but he said it so beautifully. He said How can I know what I don't know? When I don't know what I don't know right and it's that simple Do you have no way of judging what the universe of that of that domain is and so your only guess is to say I guess I got 70% of it even experts Experts of course are the exception because they do know the domain Now at some level know the experts fail too, but in terms of judging your overall knowledge base Yeah, the people who know it all are the ones who can accurately determine that they know it all right But that doesn't help an expert judging his own performance on us on a specific case particular for example We have some evidence. Yes, actually we have a fair amount of evidence in medicine that Physicians perform better than chance and the more expert you are the better you perform But still their hit rate in terms of I'm confident in this diagnosis versus I turn out to be accurate in this diagnosis It's probably about 60% hmm. They're far from perfect. Wow. They're better from better than chance, but far from perfect. Yeah Yeah, interesting So how do they improve as an expert? You mentioned that we Going slow doesn't necessarily help being deliberative And there's this compounded issue of of self-assessment that we often don't know how good We're actually doing it any given case. I mean, what does that mean for improving practice? How do we get better? How do we? The the goal of this course that we're taking is called the science of everyday thinking So given your experience In in the field of expertise and in medicine. How do we improve everyday thinking? You improve by knowing more So just the accumulation of experiences Yeah, it seems so tantalizing it would be really nice if I could be very prescriptive and say well if you do this that and the Other thing then you'll be much better At some level, I guess there's got to be a bit of a germ of truth in that I mean this course wouldn't exist if we didn't think that there are some that being explicit about everyday thinking and The fact that the traps and everyday thinking Wouldn't help people think better and and to some level that's true, but that's generally Locking into what's been called general problem-solving strategies, which are not very powerful Sure, that's going to help a bit I suppose reflection is going to help a little bit, but it seems that all of these strategies To sort of generalize horrendously are good for about a 10% improvement. Okay It's not zero, but it's not night and day or black and white either Mm-hmm and very clearly the single best predictor of how good you are is how much you know about the domain not What problem-solving skills you bring to bear on it? We began there that was wrong Yep, and so then would you suggest in in getting more experience in gaining and kind of accumulating knowledge Is it just a matter of studying the domain more of getting more experience? So if I'm a if I'm a novice Diagnostician in order to become an expert in order to become an expert in the true sense is it just a matter of Working hard and studying the rules and getting as much exposure to a wide variety of Examples within that domain. Yes, we could but we can do better than that This really segues into a whole other branch of educational psychology The question is are there strategies we can do to optimize the acquisition of knowledge? Yeah, and again, we're talking about two kinds of knowledge formal knowledge and experiential knowledge Yeah, and we're now beginning to discover and I can't take any personal credit for this one This isn't my domain, but people like mayor Bjork Rotiger Have been working hard on taking models of the nature of of the mind short-term working memory long-term memory associative and Turning that into very prescriptive and powerful strategies to enhance the efficiency of learning right things like An obvious thing like mixing up the practice the examples from across multiple chapters so that you have to try and figure out Which is which it turns out to be an extremely powerful strategy for learning Yeah The idea of transfer which is being able to take knowledge that you've learned in one context and apply it to another One it doesn't happen at all as easily as we think it does but to Psychologists of a device strategies to make that happen better So I think this is moving much more into the instructional educational psychology into things There are things we can do to very much enhance the efficiency with which you acquire the knowledge You need to get the job done as a as a diagnostician or as a human All right. Well, thanks Jeff. That's a good one to finish Go away My name is Jeff. I think about reasoning


Amen. So keep your place there in 1 Samuel chapter 14. We're going to get to a story here in 1 Samuel chapter 14 in just a few minutes. There's a lot going on in this chapter, but we're going to look at a story right away at the beginning of the chapter. This morning the topic or the title of the sermon this morning is loyalty. We're going to talk about what the Bible talks about loyalty. I was thinking about the subject I'm reading through this story and others in the Bible. The Bible actually talks a lot about loyalty. Loyalty however in our day today is a rare thing. I believe that the loyalty has always been a rare thing, but you see some amazing stories in the Bible showing extreme loyalty in the face of very difficult situations. Worldly teaching though teaches against loyalty in almost every way. Looking at just all the isms that we have today, from feminism to secular humanism to all these different worldly teachings today, they actually teach against being loyal. I'm going to show you that from the Bible this morning, but look, loyalty, all that to say this, loyalty is a trait that you should have. It is a trait that is a very important trait for someone who is as we would call someone of good character. Loyalty is something that you should have. Before we get into the sermon, let me just give you a prerequisite on loyalty. There's a hierarchy to loyalty that the Bible talks about. I mention this all the time whenever I talk about these hierarchies of leadership and following. We went through a series on that just recently, but basically even when the Bible talks about hierarchies in our lives from children to their parents, the Bible always puts in that caveat that children obey your parents in the Lord. So obviously the hierarchy of loyalty is always with God on top. The Bible talks about that with children in Ephesians chapter 6. The Bible talks about it with husbands and wives, wives to submit to their husbands as unto the Lord. The Bible says in Ephesians chapter 5. So obviously we aren't to just be loyal no matter what to some person or human being or whatever. It's always God is our ultimate loyalty. God trumps everything. God trumps everything. Even in Romans chapter 13, the Bible is talking about government and the structure, the purpose of government. Romans chapter 13, the first couple verses there are some of the verses that are really like people really work those verses hard to just get you to say that no matter what you need to listen to the government. I've heard that preached not by anyone that I associate with, but I've heard some pretty crazy sermons preached from Romans chapter 13 saying you just need to do whatever the government says no matter what. But the point of Romans 13 is really two points where the Bible says let every soul be subject unto the higher powers so there's no power but of God. So really the point of Romans chapter 13 and those first few verses there is that number one, the point of government, God gave a structure of government to mankind to carry out proper judgment to punish evil. That's the first point. The second point is God is always the highest power. We're always to obey the higher power. So when we talk about loyalty this morning, just understand applying this to loyalty that our first loyalty is always to God. So we're not just to follow somebody or just have friends that are just taking us against what God wants. That's not proper loyalty according to the Bible. Even in Colossians chapter 3 when it talks about servants and masters, it says, servants obey your masters according to the flesh. Look, you're going to have masters according to the flesh in this life but then it says fearing God. The Bible is very thorough saying that, look, God is the ultimate authority in our lives. So if any loyalty that we're having, and look, many people will do this, they'll just be loyal to a fault to where they're going against God being loyal to a man. This is when Peter said we ought to obey God rather than men. So that's not what I'm talking about. What I'm talking about this morning is finding the proper people to be loyal to and then actually being loyal. And I'm going to show you three points this morning on loyalty that'll show you how everything we're taught today. You wonder why is loyalty so rare? It's because everything that we're taught today is basically teaching people, adults, children, generations to just not be loyal, to have no loyalty. Turn to Matthew chapter 14. A lot of people will hear a sermon like this and they're like, oh, you can't just be blindly loyal. But here's the thing, when we look at our ultimate loyalty being loyalty to God, it's impossible to be blindly loyal to God. And I'm going to show you that in a second. Look at Matthew chapter 14, look at verse number 12. So we're just talking about some prerequisites that God is our ultimate authority and is our ultimate loyalty. Look at verse number 12 of Matthew chapter 14. So Jesus here is rebuking the Pharisees. Look what he said. In verse number 12, then came his disciples. Jesus just rebuked the Pharisees and said some things that were just going to really offend them. Then it came his disciples and said unto him, Knowest thou that the Pharisees were offended after they heard this saying. But he answered and said, Every plant which my heavenly Father hath not planted shall be rooted up. He's basically saying to his disciples, like there's a reason that I'm rebuking these people openly. There's a reason that I'm just tearing their faces off every time that they are here. It's because they're not from me. They're not teaching what my heavenly Father taught. He's like they are teaching something that is contrary to God. Look at verse 14. He says, Let them alone. He says they be blind leaders of the blind. So you can't be a blind follower of God because the Pharisees were blind leaders of the blind. They didn't know what God wanted. They were teaching something different than what God said and then they were leading people that also didn't know what God wanted. So look, it's impossible to be a blind follower of God because in order to follow the Lord, you have to know what he wants. You have to know the Bible. You have to know his word. So look, you can't say, Oh, you blindly follow the Bible. That's a contradictory statement in itself. The Bible tells me what God wants and I follow that. That's not blindness. That is just listening to what God says. That's just being loyal to the Lord that's in front of us here. So it's not possible to be blindly loyal to God. Somebody that says, Oh, you just blindly follow the Bible. It's an oxymoronical statement that they just said. No, I'm not blindly following the Bible. I'm reading the Bible. I understand the Bible and I'm following the Bible. It's that simple. So look, that's all that is a prerequisite to the three points. I want to give three points on loyalty today.  Go to Proverbs chapter 17.  It will serve you in many different ways, especially in your spiritual life, which we'll look at at the end of the sermon. Look at Proverbs chapter 17. The first point I want to make on loyalty this morning is this. Loyalty is long lasting. Loyalty is not a short term thing. Look at Proverbs chapter 17 and look at verse number 17. The Bible says, A friend loveth at all times and a brother is born for adversity. So just notice that at all times in Proverbs chapter 17 here. The Bible says that a friend, you know, loveth at all times. This is a friend that is loyal all the time. You know, there's a time function to this.  Friends. I mean, real friends. You know, it's the type, look, friends are the type of friend that you should have, by the way, should be loyal for a long time. All right? This is why, look, this is why, you know, it eludes so many people.  It's not like you can have a friend and be loyal for a year. And it's ironic because the very definition of loyalty itself, meaning, you know, steadfastness or faithfulness as the Bible would say, it requires, look, if you're just loyal to somebody for six months at a time and then you're loyal to somebody else for six months at a time and then you're loyal to another friend at six months at a time, by definition, you're disloyal. That's the irony of the definition of loyalty. So, look, loyalty lasts a long time. So a test is, do you have friends for long periods of time? You know, that's a good test in your life.  But do you have long lasting, you know, friendships? Have you worked at the same place for a long time? That's another, you know, just a, it's a, it's a sign of loyalty. So loyalty is not something that is short term, is the first point. Now go back to 1 Samuel chapter 14. Go back to 1 Samuel chapter 14.  Okay? Go back to 1 Samuel chapter 14. So the first point is this, loyalty, the first point is a simple one. Loyalty is a long-term thing, okay, is a long-lasting thing. Now here's where it gets difficult. Look at 1 Samuel chapter 14. We're going to look at the story. Of course, we're at a, we're at a, there's a lot going on in this story, but basically Saul and his son and all the people of Israel, they're in a bad place. They're in a, you know, this war against the Philistines. If you read the chapter before, they don't even have any weapons. You know, they're making their, their farm tools into weapons. So they're in a bad place with this war against the Philistines. The people are afraid. The people are scared. And Jonathan has an idea. You know, Jonathan is a very courageous, a very brave young man. He's a very brave, I don't know exactly how young he was here, but he was, you know, he's not someone who was a coward, and he comes up with a pretty bold idea here in 1 Samuel chapter 14. But that brings me to my second point on loyalty. Loyalty is selfless. Okay, loyalty is something that is not necessarily going to be good for you at all times. And this is where loyalty, this definition of loyalty in these examples that we see in the Bible. Look, in the Bible, there is so many stories in the Bible of extreme loyalty. And I believe that's why those stories are in the Bible, is because those are extraordinary situations, especially when we look at this armor-bearer that was with Jonathan. Loyalty is selfless. Loyalty is something that is not going to do you well personally all the time. And this is where we run into problems with the teachings today. Because the teachings today are what's good for me, what can I get, what's best for me, you know, as a woman, as a man, as whatever, you know, as a human being. All these teachings today are all focused on the individual person, the individual self. And loyalty is not going to get you there. Loyalty is selfless. Look at this story, look at verse number six. Look at verse number six. Think about it, we have an whole army up against the Israelites, and it's not looking good. They don't even have weapons. And Jonathan said to the young man that bear his armor, come and let us go over unto the garrison of these uncircumcised. It may be that the Lord will work for us, for there's no restraint to the Lord to save by many or by few. He doesn't even say we're going to win. He says, hey, you, one guy, you know, the guy that carries his armor and apparently fights as well. He says to him, hey, let's go over to this massive garrison where there's probably hundreds, maybe even thousands of soldiers, just me and you. And let's go fight. He's like, and maybe it'll work out. He's like, maybe it will work. And what does this guy say?  You know, how many people would just be like, all right, let's go? Most people are going to be, you know, questioning that, being like, ah, have you thought this through? Have you thought about this? Look at verse seven. And the armor bearer said unto him, do all that is in thine heart. Turn thee, behold, I am with thee according to thy heart. He just, right away, he's just like, I'm with you. Look, that's loyalty right there. That is loyalty to his friend, to Jonathan. He's like, let's go fight this entire garrison of soldiers by ourselves and the Lord may help us. Maybe. I mean, look, it was a humble thing for Jonathan to say. He didn't say, for sure God's going to help us here. He was being honest. I mean, there was no argument. There was no suggestions on maybe doing things a different way. Hey, maybe you want to grab these guys over here? They're not doing anything. Maybe they can go help us. There was no thought for, hey, what about our, what about my life, Jonathan? What about your life, Jonathan? There wasn't even thought for his life. Maybe we should wait for the entire Israelite army, but it gets even better. It gets even better. Look at verse number eight. So look, the guy doesn't question. He's just like, I'm with you. Let's go. Jonathan continues. Then said, Jonathan, behold, we will pass over. I mean, this is like the ultimate test of loyalty right here. He's like, behold, we will pass over unto these men and we will discover ourselves unto them. You think, all right, all right, Jonathan's going to grab his armor bearer and they're going to like put on their ninja outfits and they're going to like do the ninja thing, right? They're going to go and like sneak up on people and be like commandos and that's how they're going to defeat these Philistines. But Jonathan's like, no, we're going to go over there and we're going to announce ourselves.  You know, Jonathan, you know what's going on? He's like, not only is it just going to be me and you, but we're going to go over there and be like, hey guys, hey, over here. He's like, we're literally going to go and announce ourselves. We're going to discover ourselves to them, all right? There's no ninja tactics going on here with Jonathan. But look at verse number nine. And if they say it unto us, Terry, until we come to you, then we will stand in our place and we will not go up unto them. But if they say thus, come up unto us, then we will go up for the Lord, hath delivered them into our hand, and this shall be a sign unto us. And then they went ahead and they did it, and both of them discovered themselves under the garrison of the Philistines. And the Philistines said, behold, the Hebrews come forth out of the holes where they had hid themselves. So even the Philistines knew that all the Israelites were just cowards hiding in holes. This is why Jonathan did this, by the way. And the men of the garrison answered Jonathan and his armor-bearer and said, come up unto us and we will show you a thing. They're like, come on over here. They went and they announced themselves. They're like, come on over here. You know, we'll show you something. And Jonathan said unto his armor-bearer, come up after me, for the Lord hath delivered them into the hand of Israel. And Jonathan climbed up upon his hands and upon his feet and his armor-bearer after him. And they fell before Jonathan and his armor-bearer slew after him. So they just, they slayed dozens of men, these two guys. And it works out.  So look, true loyalty is selfless is what I'm trying to get you to understand.   And you think that, you know, loyalty is just never going to cost you anything.  Because loyalty is going to cost you something. Because by its very definition, it is selfless. That's what you need to understand. This armor-bearer would have never followed Jonathan if he was worried about himself. It's that simple. She'd say, how did he do it? Because he was just loyal to Jonathan. That's it. What Jonathan was asking him to do? Like it wasn't against what God wanted. Jonathan was just saying, hey, here's the plan. You know, God could help us in this plan or he could not. And he's like, you know, let's go. He was a very loyal person. He was not considering himself at all. So the second point is loyalty is selfless. Turn to Proverbs chapter 27. Turn to Proverbs chapter 27. And that leads me into my third point on loyalty. And that's this. Turn to Proverbs chapter 27. Difficult times are needed to reveal true loyalty. That's the third point that I want to point out about loyalty this morning. Think about the sermon on sorrow last week.    I guarantee you 99.9 percent of people, if you ask them, is sorrow good or is sorrow bad? People are going to say sorrow is a bad thing. But look, everyone thinks it is so bad to go through difficult times. But look, the reality is there's so many good things that come from it through difficult times. One of the values and one of those extreme values of tough times, of tough situations, is that it will reveal who is truly loyal, who has loyalty. I mean, think about it. What if you had a group of friends? Just think about it with your own friendships. What if you had a group of friends and half of them were really disloyal but you just didn't know? You didn't know who it was. Like that would be very disturbing. Who would ever want a group of friends where half of their friends say you had 10 friends and five were really truly your friends and five were not your friends. But you didn't know who was who. Well, I would rather have no friends than have that kind of group of friends. But that is why difficult situations are necessary or one of the reasons that difficult situations are necessary because difficult situations will reveal true loyalty. Look at Proverbs 27 in verse number six. I had a friend one time that I mean, I've told this story before. I don't want to be the guy that tells the story over and over again, but it's relevant here. I mean, I had a friend one time. I had a group of friends. They were my roommates when I was 18, 19 years old and I got in a difficult spot one time with a few of these roommates and one of these friends ran away and I was appalled. From that point forward, I was disgusted with this person. To this day, I'm disgusted with that person. Like 25 years later because I was shocked. I was shocked. I got in a difficult situation. It was no fault of our own. And it's like, hey, you know, we got to go to work here and like one runs away. And I was just like, wow, that's unbelievable. Look, that's not royalty. But you know what? It took that difficult situation to unveil, you know, look, that's who he was the whole time. That's who he was. I'd known this person for three, four years at this point. And that's who he was the whole time. I just didn't know. I just didn't know. So that's a valuable thing about difficult times. Look, think about Jonathan and the Armourbearer.  But look at Proverbs 27, verse number six. The Bible says, faithful are the wounds of a friend. But look at this last part. But the kisses of an enemy are deceitful. You see, the reason that the kisses of an enemy are deceitful is because the reason it's deceitful is because you don't know, in this verse, you don't know their enemies. They're kissing you. They're friendly to you. They're, you know, they're being your friends. You just don't know you thought they were your friends. That's why it's deceitful in itself. So you say, how do I find out? So I got these 10 friends. How do I find out? It's one of the best ways to find out who your real friends are, who the ones that are loyal to you is because, look, it's easy to have friends when everything's going well. You know, this is where the saying fair weather friends comes from.   There's going to come a time where maybe it's not pleasant being my friend. There's going to come a time where it's just not pleasant having that particular friend at times. This is David's situation in 2 Samuel 15. 2 Samuel 15, David is being kicked out of his own capital by his own son. His own son betrays him and is taking over his empire, taking over his kingdom and kicking him out. And look, these men that followed David, these people that followed David and went with him, look, things were not good. They're being hunted. That army, you know, armies where people are coming after them, trying to kill them, people are cursing them, but they were loyal and that's how, I mean, David knew that these people were loyal at that point because at that point, and here's the thing, it was David's fault. It was David's fault that they were being kicked out. This was a punishment from God and these people were just like, you know what, we're loyal and we're going to go with him. Turn to 1 Chronicles chapter 12. Turn to 1 Chronicles chapter 12, but here's the thing, I mean, nobody would want, nobody really wants fair weather friends. So if you don't want fair weather friends, you know, I think that a lot of people are probably just like, put their blinders on to this. They just like being surrounded by people and maybe they even know that they have fair weather friends, but me personally, I don't want fair weather friends. I'd rather have no friends than fair weather friends. I'd rather have zero friends than have 10 friends where five of them are just secretly enemies of mine. That's just me. But how do we know who those five are? The way you know how those five are is when tough times come. That's how you know. 1 Chronicles chapter 12, look at verse 16. This is where David is taking over his kingdom. Saul has just died. Saul and unfortunately Jonathan has also just died and David is taking over the kingdom. He's now king and he's clean in house. He's going to war against the Palestinians. He's, you know, taking back, you know, the lands that Israel has lost. And look what he says in verse 16. This proves right here that going through tough times will reveal who is truly loyal and it will also bring those people closer to you. Look at verse 16. And there came the children of Benjamin and Judah to the hold unto David. And David went out to meet them and he answered and said, if you come peaceful unto me to help me, my heart shall be knit unto you. Now here's what's interesting. This is what David is saying.  He's like, my heart will be with your heart. He's basically saying, are you loyal to me? But you know what helping David involves here? You know what the help that David is asking for here? The help is going to be some tough times. That's how David could say that his heart would be knit unto them, because he's saying, you know what? I'm going to test that loyalty. He's like, because we're going to go through some hard times. I'm not going to ask you to help me to move furniture. He's like, I'm not going to ask you to help me, you know, do some yard work. He's like, we're going to go to war and maybe you are going to die to help me. He's like, then my heart will be knit unto you. He could say that honestly because he knew that after going through that test of battle, going through that test, that struggle that they were about to go through, he knew that that would prove their loyalty. And if they wouldn't go, then he knows who they are. So it's an interesting, you know, test, but look, they prove themselves through fire. So that's the third point, is tough times, difficult situations will show you who is loyal and who is not. In your life. So look, it's a good thing. Struggle, even though you and I may not stand up and say, we want to go through struggle, we want to go through sorrow, we want to go through all these tough times, there's so much good that comes from it. There's so much good that comes from it. So that's just three simple points on loyalty. And you look at, there's a lot of stories of extreme loyalty in the Bible. You know, you could pick and choose stories all day long. There's also a lot of stories about extreme, you know, antithesis of loyalty, which is like Judas, right? Judas, who just like literally sells out God, literally just turns on God for what? For principal, for no, for money. He just sells out the Lord Jesus Christ. Even Peter, even Peter, you know, tripped up and was not loyal.  Peter basically said to Jesus after Jesus told him, like, you're going to deny me.  And then of course, Peter failed at that. No, we know that Peter completely redeemed himself and was loyal, literally unto death, you know, with the rest of his life. But look, loyalty is all over the Bible. It is something that we should definitely have. And it should be a long term thing. So who should we be loyal to? That's the last part of the sermon tonight. Look at this morning. Turn to Proverbs, turn back to Proverbs, chapter 27. Who should we be loyal to? Who should we be loyal to? For how long should we be loyal? I mean, a long time or is there any limitation to that? You know, who and when should we be loyal? Turn to Proverbs, chapter 27. The Bible says, you know, look, the first people that you should be loyal to is, I'm sorry, go to 2 Corinthians chapter 6. I messed you up there. The first people that you should be loyal to is your friends. You know, that's one group of people that you should be loyal to. You should stand by your friends through good and bad as it is fit in the Lord. Okay, don't forget that ever. You say, well, my friends are into all sorts of things that, you know, I don't feel like I should be involved in. Look, as it is fit in the Lord.   This is where the doctrine of separation comes in. This is where the doctrine of separation, like maybe you shouldn't be, you know, yoked up with people. In 2 Corinthians chapter 6 verse 14, the Bible says, be not unequally yoked together with unbelievers. For what fellowship hath righteousness with unrighteousness? And what communion hath light with darkness? So the difference between being friends with somebody who is unsaved and is just into all sorts of sin and someone should be, you know, who should be, you know, you should be loyal to is basically an unrepentant sinner that's out in the world. You should not be loyal to that person because you're not being loyal to God if you're in that situation because God wants you separate from those things. But now, people will take this too far and say, well, because I have a friend that makes mistakes, then I should not be loyal to this friend. No, look at David. David made all kinds of mistakes. As a matter of fact, a lot of the mistakes and a lot of the punishments that David went through, the people that were loyal to him and around him, went through those punishments because of things that he individually did. But they were still loyal. See, David though, he was not an unrepentant sinner. David got right almost immediately whenever he messed up and made mistakes. He was extremely humble. What's second Corinthians chapter six is talking about is some unbeliever living in the world. These are people that we should not be yoked up with, especially unequally yoked, where we are put in positions where their lifestyles are coming upon us. That's what the Bible is talking about. So first of all, you need to have the correct friends. You need to have the correct friends. And this is where Proverbs 27, 17, you know, comes in and says Proverbs 27, 17 where it says iron sharpeneth iron. So a man sharpeneth the countenance of his friend. So look, we should have the type of friends where, you know, we're not to be separated from them, serving them or being loyal to them is fit in the Lord. And look, we should be sharpening each other through that relationship. All right, turn to John chapter 15. You say at what cost? You say at what cost? What happens when it starts to cost me being friends with this particular person? Well, as long as it's in the Lord, just like the armor bearer, you know, you know, dealt with, as long as it's in the Lord, look at John chapter 15 and verse number 13. As long as it's in the Lord to be friends with this person, look, your loyalty should basically have no limit is what Jesus even says himself. Look at John 15 and verse 13. It says, greater love hath no man than this, than a man laid down his life for his friends. Look, I'd say at the point where if you and I are friends and you literally being my friend is literally going to cost you your life, I'd say at that point there's some struggle going on somewhere. I'd say at that point it's not all, you know, rainbows and sunshine. Something has gone wrong, but the Bible says that, you know, you should be loyal to your friends up to death is what Jesus says. It's like there's no better love than literally just giving everything for your friends. And look, as long as it's in the Lord, that was the armor bearer right here. The armor bearer is like, hey, is this going to cost me my life? He's like, I'd rather, he's like, I'll take the loyalty. I'll take the loyalty rather than being a disloyal person and saving my own life.  I mean, if there's ever a situation that would cost you your life, it's, hey, me and my buddy going up against 100 other people. But he was just loyal to the end. So look, you should be, as long as it's fit in the Lord, choose your friends correctly and then be loyal to your friends. It's very simple, all the way. That's the definition of loyalty. Here's another one. Turn to Proverbs chapter 31. Who else should we be loyal to? How about this? In our marriage we should be loyal. We should be loyal in our marriage. I find it interesting in Proverbs chapter 31. This is super interesting.  If I needed to be the ideal woman, all I have to do is really just Proverbs chapter 31. It's all right there.  Just Proverbs chapter 31, this virtuous woman in the Bible. But I find it interesting that the very first thing that is pointed out about the virtuous woman in verse number 10. In verse number 10 it says, like, we start on the virtuous woman. It says, who can find a virtuous woman? Her price is far above rubies. But look at verse 11, the very first thing. So it says, this virtuous woman, this perfect spiritual woman, it's like she's so rare. But what's the first trait? The Bible says the heart of her husband, the safely trust in her so that he shall have no need of spoil. Look, the first trait mentioned here is that he trusts her. He completely trusts this woman. Why? Why does he trust her? Look at what it says in verse 12. Because she will do him good and not evil for the first couple years of their marriage.  You know what the first thing? Look, the first trait mentioned about this virtuous young woman is that she is loyal. She is loyal to her husband. Look, a lot of young people, a lot of young people are looking for the wrong things in a spouse. A lot of young people, they have, look, both young ladies and young men, they have this long list of things that they want to have for this perfect spouse. They're looking for perfection in many different ways. But notice the first thing mentioned about the virtuous woman. Look, I'm here to tell you, young people, it's much less complicated than that. It's much less complicated. This ideal woman, her husband, has total trust in her. Why?  Her whole life. Young ladies, this is what men want. Even if they don't know it, this is what men want. They want loyalty. As men, as young men, go through this life and they lead, they rise, they fall, they walk, they run, they trip and stumble. They want someone who will stand by them. That's what they want. Not chastise them, not criticize them.  Not, you know, destroy, you know, the ideas and the things that they want to do. Look, the armor bearer never questioned this arguably foolhardy plan of Jonathan's. He just went. That's what young men need is they need an armor bearer for a wife. That's why the virtuous woman in Proverbs 31, the first thing that is pointed out is that she is loyal. Look, you could argue that this is the only thing that a spiritual young man needs in a life. You could make that argument. You say, how? Because if she is loyal, everything else will follow. That's how.  She's trying to save herself the suffering. She's trying to save herself and her family, you know, the suffering of what she thinks is a foolhardy decision. But let me tell you something, ladies. The trade is not worth it because he will not trust you. It's not worth the trade. This woman in Proverbs 31 is the opposite of the feminist today.  So young men, you say, what should I want in a wife? Loyalty. That's what you should want in a wife. You say, but how do I know? How do I know? Well, Matthew chapter seven says that a good tree can't bear evil fruit. So does she have loyal parents? That's a clue. That's a clue right there. All you need is loyalty and everything else will follow. Ephesians chapter, what about the men? Women are like, well, shouldn't my husband be loyalty? Well, in Ephesians chapter five verse 25, it says, husbands love your wives, even as Christ also loved the church and gave himself for it.  And pretty simple there. When? All the time. That's when. So look, it's much simpler. It's much simpler in marriage, especially, and it shows us like what we should be looking for, you know, before we even get married. How about this one? You know, the Bible talks a lot about masters and servants. This is why I got to have a King James Bible too, because this master-servant relationship is just like a real thing.  We all have different levels of authority that were underneath, that were above in our lives. So masters and servants is an important thing.  You know, it's completely lost in all the modern versions, all this great teaching about how to handle authority in your life. And guess what? Being loyal to authority in your life. How about your employer?  This is why you see that there's no loyalty today like with people that go to work anywhere. There's just no loyalty at all. It's the same principle though that should be applied. Find someone, it's the same principle that you would apply to your friends. You know, find someone, the Bible says in Proverbs 28 that there's unjust gain, meaning there's a just way to make money. There's a place that you can go work that is just, that is a decent, honest way to make money.  So find somewhere you can work that is, that will provide just gain is the first thing. Find friends that you shouldn't be separated from and then be loyal to them. Find somewhere you can work and look, here's another thing people will do. They'll be like, well, you know, my boss, he's not an independent fundamental Baptist. It doesn't say that your boss has to be a saved Bible-believing Christian. It says that you have to make your money in a just way. If he's not saved and he's, you know, but what you're doing is an honest living, there's nothing wrong with working there. You know, you go and you build something, but the guy that you work for is a Hindu or something, you know, but he's just, he's in an honest living, it's an honest trade, it's honest gain. The Bible says you should be loyal then. You know, you should be loyal. Look, they pay you. They train you. Many times people hire people and they don't make money on them for like the longest time. I don't know, it probably depends on the trade, but I know, I know friends of mine who are business owners that literally just stopped looking for people to hire because it's just, it's just costing them too much money. They train somebody for an entire year and then the person quits. They train people up for six months, a year and a half.  But it's a real problem today and it's all about loyalty. It's all about loyalty. Look, like, well, you know, they just weren't this and they just weren't that and I just feel like I, I deserved another dollar an hour. You know, here's the thing, you don't deserve anything. That's the way, you know, you need to teach your kids. You deserve nothing. You deserve nothing. You say, I deserve better. No, you don't deserve anything. You don't know anything. You know nothing. When you, when you know things and when you gain skill and gain knowledge, look, the money will come. All that other stuff will come. The key is, you know, be loyal and look, loyalty will be rewarded, especially in those situations. So look, loyalty is something, as long as we are just keeping God at the top all the time, we need to find the right people in our lives to have relationships with. And then we just need to be loyal to those people. It's very simple. It's a real problem today. And look, it's a character trait that we should all have and it'll strengthen, it'll strengthen all of your relationships. It'll strengthen your friendships. It'll strengthen your friendships, you know, especially in church. It'll strengthen your marriage. It'll do well for your, your work life. But here's the main thing. God wants us to be loyal. This is why you see so many places in the Bible where words like diligence are used, where words like faithfulness are used.  In Proverbs chapter 20, actually turn there. Proverbs chapter 20, look at verse number six, I believe it is. Proverbs chapter 20, verse number six, the Bible says a faithful man, meaning a loyal person. Look at verse number six. The first part of that verse is most men will proclaim everyone his own goodness. So remember how loyalty is selfless? This person is, he's just proclaiming himself. He's, he's focused on himself. And then what does that translate to? He's not loyal because he's focused on himself because loyalty itself is selfless and it's going to cost you something in your life. And look, here's a really sad thing. If you are not a loyal person, you will raise children that are not loyal.  They will not be loyal to you. Loyalty, you know, is something that has passed through the next generation. And guess what? This is why you see many people. Ultimately, loyalty is why you see so many people that struggle with loyalty where they completely fail in their spiritual life. Because ultimately, they can't be loyal to anyone. They also can't be loyal to God. Because it's just, it's just a trait that they don't have. So it's so important that we, you know, evaluate ourselves. Do we have the right relationships in our lives? If I do, I need to be loyal to those relationships. Why? For my own personal relationships, for my own spiritual life, yes, but also so my children will understand and see that example of what it takes to be loyal. Like, oh man, you know, my dad is still loyal to that one person in that situation. Even though that caused him a lot of pain and suffering to be there with that person when they went through those things. A lot of people told him that, you know, maybe he shouldn't have done that or maybe we shouldn't have done that. But as long as like they weren't being disloyal to God, you should be loyal to those friends, loyal to those relationships, loyal to your spouse. And the best thing out of it, and the best, you know, it just goes along with everything else in this Christian life, is that this will be an example to our children. And the next generation, they'll be loyal. But that's why this next generation, we're seeing they just can't be loyal to anything because their parents weren't loyal. Nobody ever taught them how. Nobody ever taught them how. And then hopefully, if we're loyal in our lives, we'll have that point, you say, oh, I just give everything and just be loyal no matter what. But we have that point in our lives where God will say to us, you know, well done, now good and faithful servant. Like he says to the person, you know, after the person with the five talents that makes five more talents, he says, well done, now good and faithful servant. So, you know, that is the goal in our lives. And then we can pass that on to the next generation as well. And look, here's the thing. Eternal security is like the ultimate demonstration of loyalty to us. The fact that there's nothing that we could ever do, we could go and we could mess up for the rest of our lives. We could go and we could trip and fall on our face every single day of our lives. We could actually go and like not even like want to serve the Lord for the rest of our lives, and God will still remain loyal to us. I mean, that is the ultimate loyalty right there. That's the ultimate example. Everything that God teaches, he shows us by example first. And eternal security is the ultimate, just the demonstration of, hey, this is what loyalty is. I will never leave thee or forsake thee ever. Let's bow our heads and have a word of prayer.