Yeah, well, we're back.  And Matthew James Bailey is back. He joins us from where? Colorado, is it? Yeah, that's right. Good evening, Jake. Great to be here. Great to have you on, Matthew. Matthew was an AI expert, and he's qualified in many, many things that spring out of AI-written books about it. The World 3.0 was the most recent one, which I have, which is a great book, to have you understand that there are very few limits to AI. Unfortunately, we don't use it enough in the, what do you want to call it, the social management government context. And if you don't mind, Matthew, I'd like to talk to you about that today. Because you're always asking about the ethical guardrails on AI, and of course it needs ethical guardrails. But you wanted to take an analysis of Hawaii first. So let's start there. What were your thoughts about how AI could be involved in issues in Hawaii now? Well, first of all, how does the self-determined future with artificial intelligence, does it want to be driven from the agenda of the mainland? Or does it want to self-determine its own future and honor its cultures and take them into the future? And to create a prosperous Hawaii, where artificial intelligence is working well for the different communities within Hawaii, and also helping the businesses, education, and other sectors like the military, et cetera, to move forward into a new tomorrow for Hawaii? So the question is, does Hawaii want to take control of its future? Or does it want to be beholden to the mainland? Yes. And that's a really good question. It's not clear that anyone, aside from you and me here today, will ask that question. But there it is, though. There it is. It's determining policy by AI. It's not mathematics. It's not figuring out some numerical data. It's determining much larger issues. And so when you ask about what Hawaii wants to do in the world today, whether it wants this identity or that identity, AI is perfectly capable of doing that. And so you say that, and then you, by implication, you're also saying, what about the country? Yes. The country needs to determine where it wants to go. And AI could help it do that. But again, the same problem, aside from you and me asking that question, who else? Yes, so quite things have happened since we last spoke, Jay. So the National Security Council on Artificial Intelligence, I was invited to a private roundtable with them and ambassadors from all four countries around the world. And they put together an incredible proposal, $32 billion, Jay, of how to mobilize America to be able to become a global leader in AI and to protect the future of democracy abroad, basically against the Chinese threat, which is outdated in the media, quite frankly. And also what's also happened, Jay, is that the state of Ohio have effectively digitally succeeded from the federal government with new data ownership laws for the citizens of Ohio. So a lot's going on, Jay. You know, it's funny that you say that, because tomorrow we're going to have a show about Pegasus, you know, the NSO, the Israeli software that somehow got into the hands of the wrong people who were supposed to go to responsible licensees and then it got out. And then now it's being used for surveillance. And it strikes me, and I really like, I would like your view of this, is that, you know, if not so much of accumulating data anymore, that's like the last generation. It's processing the data. It's making sense of the data. It's learning larger lessons from the data, millions and billions of pieces of data. And the secret is not so much gathering it. We know how to do that. Pegasus can do that from every man, woman, and child in the world. It's what do you do with it then? You agree? Yeah, absolutely. Guess how much data is generated globally? 2.5 quintillion bytes of data is generated per day across the world. Jay, I can't even count those zeros. It's astronomical. So you're right, collecting data is something that we're really good at. And to your point is how do we take that data and put it into ethical AI models, order to be able to make our systems more efficient, to be accountable, and to start to do well in society and help us leap beyond the challenges of today of inefficient human-centric systems. And government tries its best, but it's pretty inefficient, into kind of more of a streamlined society where we're actually kind of moving ahead in a fantastic life and a flourishing experience with the environment, right? AI itself, Jay, maybe in about a year's time, we're cognitive AI, AI that has some kind of reasoning like the mind. AI might help with government management and government decisions. They might even choose who's in the government. Now that would be interesting, wouldn't it? Yeah, and it could make policy decisions. But let me go further about my day, Matthew. A couple of hours ago, we had a show about Facebook and why Facebook has failed to identify the disinformation that is being parlayed on its platform. And a user, a viewer sending a question says, how can you tell, how can anyone tell whether a given piece of data, a report or some posting on Facebook is true or not true? I said, easy. He talked to Matthew James Bailey. He'll tell you how to do that. He used AI to determine a threshold of whether that is true or likely, not to be true. Now you may have to have a human committee to look over it on the second step, but on the first step, it's really easy to identify suspect information. And we could solve the problem with Facebook and all the social media by using AI to identify false statements. And we'd be a lot happier for it. Do you think it's possible? Would it work? Would it be helpful? Yes, it would. It would take a little bit of time, but certainly the mathematical models can be developed, Jay, to be able to train AI using the data to look for false signals. As Mark Zuckerberg used this in his question, he talked about signals. And so, yes, we can use AI to detect these signals. Now the question is, Jay, how do we deal with speech? How do we deal with the First Amendment where people genuinely have a different view maybe to the mainstream view that's been adopted by most people, i.e. get a vaccine, right? How do we deal with that First Amendment? Because what I'm seeing out of the US government is government working with Facebook to control the narrative on social media. And that is a First Amendment issue, isn't it? Yes, it is. But you know, one solution on social media and on media in general is that we have to remember the First Amendment separates the media from the government. In other words, it's the government's obligation to allow a free press. It's not your obligation or my obligation, which creates a bit of a problem in the sense that we may not have the same view of truth. However, there is a way to fix that hole in the boat. And this has been considered by Congress, but then as you may know, Congress doesn't do anything. So they didn't do anything about this. This is the provision that would allow an individual to skew the platform or erroneous information that was posted there by somebody else. Private law student, private law student. And it makes the platform responsible. Right now under the FCC rules, I think, and other law, the platform can say, not my fault. I've merely repeated what they posted. It's not my fault. If that changed, then the policeman at the elbow would be the consumer, the newer, who says that's false. And I might add, by the way, that everybody's asking, why is the right-wing media now saying it's okay to take the vaccine? Why are they doing that? They changed their position. Why are they doing that? And I think the reason is, there's four or five reasons that come to mind, but one of the reasons is they are afraid that somebody who listens to them about not taking the vaccine will die. And that person's lawyer will say, well, you know, you listened to Fox News and you died. And they knew that what they were telling you was not true. So you have a cause of action, just like that voting machine case.  Well, it's the same thing here. I think they realized, or somebody pointed out to them, or maybe there's already a suit by the family of someone who died after taking advice from them on COVID. And that's why they're changing their tune. But in any event, you know, it seems to me that the answer is not government action, except insofar as the government's saying, if you, Matthew, or me, Jay, we feel we've been lied to and we were lied to a detriment on that lie, and we can go after the platform, and that's a personal post to it. I would change everything. Yeah, it would. And in the book, we talk about personalized AI, or digital buddy. This week, I was speaking to the UK's leading wireless group, literally geniuses, in invention of wireless technologies about the personalized AI. And it was great. It was a really good PhD discussion, but they loved the book, which was great. When personalized AI comes in, then that will ensure that kind of a truth barrier between what the digital world is saying and what our own truth is, and ensuring that we get the right truth and detecting whether we're being misled by false facts, example, or news, or we're being misled by a particular agenda from a particular big tech company, for example. Personalized AI will assist us to actually recapture, if you will, critical thinking and the ability to actually choose without too much influence from this big world and from this world that is rich in contrary views, should we say.  I use the term disinformation because I think that's really what it is. It's not misinformation, it's lying. Disinformation. Sure.  And now AI, as you can tell me, AI works instantaneously. AI is watching, you know. So AI is watching what he says. And so, you know, it's like in Hawaii, we have ratings for restaurants. So if you're considering going to a restaurant, there'll be a little green sign posted. You know, this is good or not so good, you know, a gradient of good. Well, you don't have to take it off the platform. You don't have to shut down the platform. You don't have to exclude anybody. All you needed to do was warn the public on a gradient basis. And I would say, Matthew James Bailey's AI has decided that this is on a scale of A to D. This is a D. You shouldn't really believe this or vice versa. And the bottom line is the person stands advised, what do you think? Yeah, it'll go further. So the personalized AI will say, hey, Jay, these are the nutrients that are really good for you today. This is the food that you like. This is how far you want to travel because of your, maybe your diary or whatever. Let's look at the restaurants within those kind of parameters of choice. Where can you get a table because you enjoy these type of views or you can type this type of AI can manage that for you. And there's no reason why people don't, we can't do this today. It's not that difficult. The key to success are two things. One is proper data governance so we can liberate more data ethically, train AI ethically, and then it can make a decision. So that is important data governance. And secondly, is that need to start thinking about the well-being of society and citizens as opposed to the bottom line moment from moment. And this is the new mature mindset, Jay, I believe we need going forward, which is about a kind of maturing as a human species and starting to redefine what value really is and what the true currency of human living really is. Yeah. So let me introduce another thing. And that is, and there's many things to follow, but the next thing that comes up is Michelle Goldberg wrote a piece for the New York Times a couple of days ago. And she wanted to, it was an op-ed piece, and she wanted, they don't call it that if they call it an essay or something, but she wanted to deal with the issue of why people are in Trump's base, why they follow him mindlessly, why they, she is talking specifically about a group of people called the always Trump state people who will travel across the country to attend his rallies. They don't have a pot to pee in, but they travel across the country to attend his rallies. As it's interesting, why do they do that?  And he fills them up with something and it's demagoguery, it's demagoguery and it's authoritarian government and it's a tyrant, but that's the mechanism, that's what works. And by question to you, and I never said this was a rose garden, this is a hard question. What can AI do when you have 70 million people who are being confused that way? And largely because their lives are insufficient, they're lonely and they have no friends, you know, in a nuclear society, we don't have as many friends as we did 10 or 20 years ago. We live alone, we watch the television, we sleep, we eat movies, it's not as rich as it was. And so there's plenty of opportunity, plenty of vulnerability for a guy like Trump to take over our thought process. What can AI do about that? Well, first of all, is that there's people outsource their sovereignty because it's easier to follow than do critical thinking and make decisions for ourselves. And it goes on both sides of the aisle, okay? And this is symptomatic of the human race in general, is we outsource our sovereignty to others. And that's fine, but is that good for our well-being? The other thing is, Jay, is that AI can help us to discover things we're really good at that make us feel passionate and make us enjoy doing and being in life, okay? It can help us to find our gifts, maybe it's in the arts or gardening or being in community work or being a theater or whatever it may be, it doesn't really matter. AI can help us to find our gifts at which point then you're nourishing the individual emotionally, at which point then they're no longer distracted from this empty space inside that is wanting to be fed by an outsource, outsource the kind of message, if you like. Does that make sense? Yes, yes, yes, exactly. And let me take it one step further, if you don't mind. So Joe Biden and Don Lemon today, right now in Cincinnati, they're talking about these adjustment things that the government has done, would consider doing, may do later and all that. And for example, there's an issue about whether if you pay a high unemployment compensation which the government is doing right now and in fits and starts, you are searching people going back to work. So these are, it's numerical, but it's also policy and you wanna tune things in our world, in our country so that you give them the right benefits. You start the benefits, you stop the benefits, you change the benefits so that when the economy is at a correct time at the perfect time, they will go back to work. Problem now is that arguably some people are not going back to work. So a lot of small businesses can't find their stand. They can't really do the job, even though there's a demand. And so I'm thinking that AI could get in there on a governmental level and say, okay, well, should you give them $300 or $250 or $350? And can you change it? When do you change it? And then when you've done it, you have to get data and see if that's working. And if it's not working quite right, you adjust it again. Talking about 330 million people, you can get the data, you can figure out what the economy is from town to city to county to state and then you can adjust this so you can create a very adjustable economy. And furthermore, you can beat inflation if you make it smart enough. I suggest to you that the Matthew James Bailey AI machine could do that, we could develop policies and we could coon the policies so they work perfectly. Am I right? Is it possible? Yes, yes it is. And I mean, if you look at Wall Street and the Stock Exchange, they're already using very complicated machine learning and AI algorithms to manage the whole fiscal kind of moments in America and globally. So complicated models like that in terms of determining how to operate in markets which would be internal markets within the US is certainly possible, it ain't difficult. Well, it's a big task, but it's not rocket science and all it takes is investment and the right mindset. This is what I think we need to do Jay. I think we need to excite the American people. It's time to excite the American people. It's time for a mature leadership to take America into the future and to say part of that future is gonna be with artificial intelligence. So let's look at our educational programs. Let's start teaching people critical thinking and getting them ready for AI of the future and AI jobs of the future. Let's start working with small businesses and bring them into AI centric businesses so they can compete with Amazon and maybe do better. Let's start looking at what's destroying the fabric of American society or its diversity and let's start using AI and a new mindset to start to move us into a new paradigm. And that kind of leadership I think Jay will work. It does take time, it does take investment but it ain't impossible. All it takes is a new set of mathematics and the right approach and we'll be able to do this. The problem America has, Jay, is that it's governance of data and it has no national governance of artificial intelligence by the way. It's behind. Europe has a policy where every single AI must conform to four levels within two years. That's for 550 million citizens. Well, I think it's about 550. It might be a bit less than a few days. So there are regions equivalent to the size of the US that are tackling AI to make sure it does well for the societies within that region, this case in European Union. And to be honest, within the US, there's some great initiatives part of the national security and artificial intelligence. There's no leadership. There's no leadership into this AI centric future, Jay. And that's what we need. Well, let me go to the next level of my inquiry with you, Matthew. So right now, it's a long shot on making any changes, good, bad, or otherwise. And because the federal government, especially the Senate, is blocking everything. They're blocking infrastructure, which is but a motherhood kind of issue. It's unbelievable they're blocking infrastructure. But one thing that we should be doing, and many countries have learned the benefits of doing this is to know the truth, to know the truth. So for example, if I wanna find it out with Paul Proctor back when Cambodia, and I had AI and a lot of data, I could find out what was going on. I could find out what led to it and who was involved in it and what they did and how it affected the country and the sociology and the culture and all that. I can learn a lot. Likewise, in any remarkable public social event I could do an investigation using AI.  You get all the facts, you organize your material and the conclusions that come by themselves. It's not hard to do it. And it's a lot easier to do it with AI. So for example, we're gonna have this, some people say a circus in terms of Nancy Pelosi's commission. She's also got some strange members on her quote, bilateral house commission. But suppose I said, forget about that. That's all political. It's all, this is Birkus. Why don't we let the Matthew James Bailey AI machine go in there and give it all the data we can possibly give it. The videos, the television footage, all the articles that have been written, all the statements that have been taken, hither and yonk, just feed it as much as we have and let it decide what happened, who was responsible. You know, the extent of the event, the extent of the problem, the connection with, you know, our democracy and government and what can be done.  The AI is not gonna be concerned with that. You can make it clear and clean. So my question to you is how far away are we from finding the truth? Not only here for January 6th, but in other places where there have been war crimes, atrocities, governmental failures, to find out and tell the people, because they don't know necessarily, because there are doctors who would prevent them from knowing, but if you put an AI investigation in place, you could advise them fairly and squarely what happened? What do you think? Well, I think we'll see artificial intelligence assisting government to be more efficient and making government more accountable, Jay, and also the efficiency of their policies and suggest changes. So we'll definitely start to see that. Whether that passes through Congress, who knows. So we'll start to see that. We'll start to see AI as a digital policeman, protecting people and doing crime investigations. And so we'll start to see that emerge and we'll probably see AI monitoring the quality of Facebook and social media in terms of how well is it doing and what is the impact on society whether it's positive or negative. So we will start to see AI in a kind of a guidance role within different aspects of a nation from government through to services within society and how citizens are kind of experiencing that. But we don't want to enter into the Chinese approach where everybody's surveyed and monitored. We don't want to go into that because that is potentially non-democratic, but it is non-democratic. And it's also 1984 in reality. So there's a fine line to go, Jay, where we keep AI democratic centric, which is what I talk about in the book, to assist us to advance into a better future. But we don't want it to be 1984 Georgia all well, do we? No, because there are unforeseen, well, maybe does that make sense under Orwell, foreseen consequences, but even things that he did not foresee. That's right. You know, we could have the 20th century and, you know, Adolf Hitler all over again, but much worse, and if Hitler were alive today, he would be looking for AI, run Germany, and he would be using it. He certainly would. We'd have Alan Turing inventing a new AI to help us to get past this kind of all these issues, which is kind of like a bit of a war in effect, re-controlling and getting our sovereignty back in the digital world. I can see Alan Turing working on some very clever mathematics for us to regain our sovereignty again. Absolutely. So one last question to ask you, and you mentioned a minute ago that, you know, Europe, with its 500 and some odd million people is ahead of us. And I would venture to say that China is ahead of us because when China puts its head down on something and it did put its head down on AI maybe 10 years ago and it deployed old kinds of resources to being best in the world on AI as well as hacking on my dad. So, you know, they have their ahead of us, but let's assume that we put resources into this. Let's assume though, we make up our, let's assume that the government reads your book. What is it that the world 3.0? You have it handy, hold it up so we can see it. Do you have it? Sure, I do. That way, yes. Inventing world 3.0, okay, it's in there. So let's assume that we all get the message about the value of AI in this country, in this country. To solve all the problems you and I have been talking about but many more, what are we missing here? What are we missing to get there? What are we missing? What do we need to have to catch up with Europe and China? What do we need to solve all these problems? AI for the United States is a little behind the curve. How do we get ahead of the curve? So a couple of things here. First of all is that the US has invested $50 billion in a semiconductor manufacturing in the United States of America. And that's very important to keep semiconductors that will literally be the AI brain J in our computers within domestic shores, that makes sense. And so that's a very good move by the American government and Intel and Nvidia are probably number one in the world for AI and semiconductors. So the US is doing some good stuff and also about raw mineral supply J that make the semiconductor, so it's doing some good things. What I would say about China, there's a couple of things I wanna say about China. First of all is that they are using AI to ensure that children are not spending too much time on gaming. So they're actually looking at the mental health which is an interesting thing. And they announced recently a research lab dedicated to AI in environmental and sustainability applications. So some interesting things going on in China before we start raising the flag and all that kind of thing. What this country is missing is leadership. It's missing leadership into a new tomorrow. That's what it's missing. There are wonderful, there's some incredible debates in Congress and the house that I watched. Not all of them, it's not any government institution. Some of them are really quite silly. Some of them are really quite profound. But what we lack, I believe is a mindset take us into the future. And every region, every state needs this mindset. Leadership take them into the future because J AI is not going away. And so having leadership that takes the cultures and the citizens into the future where everybody's nourished and there's no reason why we can't do this, J. That's what we need. That's what we're missing now. And the European Union is starting to show signs of this. Well, that does drive me to one more question. I'm sorry, Matthew. That's okay, it's fine. I really enjoy talking. And the nature of energy here in Hawaii, one of the big touch words is distributed energy. And that means if you're going to do clean energy and you want every household to have a solar panels on the roof. So then it's distributed. It's not a hub and spokes out to them. It's they have their own control of their own technology. And I'm thinking that this would be made easier in the United States or anywhere if it could be organized on a distributed basis. Meaning, I have a little town in the middle of Missouri. I'm picking them because they have a lot of problems with COVID right now. A little town in the middle of Missouri and they have the ability to run AI for their healthcare system or their water system or the sewage system. And they don't have to bring in the big boys from New York or San Francisco or LA or anything. They can do it themselves. Now, is AI and I know that computers are getting better and the transfer of data is getting easier and the software can be passed along from place to place. Is it possible in the future? Do you think this will happen where AI becomes distributed where every little town in Hamlet will have opportunities? Absolutely. And the book talks about how we can achieve this democratically and ethically. And the key thing is states and counties should self-determine their future with artificial intelligence rather than beholden to government agenda, okay? So yes, AI will be distributed everywhere. And I've talked about this in films. In fact, we won an award recently for one of our films, talking about this actually. So J, it will be everywhere. The key thing to ask is this, do we want AI to be part biologically integrated with us or not? And I don't think it should be biologically integrated. I think the human experience, AI should be sitting next to it but integrated within it. I'm not sure about that. I mean, we're talking about transhumanism now, James and kind of cyborgs and kind of the ball collective. But AI will be distributed. And the reason for that, J is very simple. It can make decisions faster at the edge. It can make decisions quicker in our lives. We can understand us better for our benefits under our sovereign control. Having AI close to us can have tremendous benefits so long as we do it ethically and we do it with meaning and purpose and sovereign for the individual. You know, you raised one more issue and I promise this is my last question. One of the problems that I have in our world today is that we don't seem to be able to remember what happened before. The old thing about a he who forgets history is doomed to repeat it. And so when you have these policy organizations like Congress, those guys can't remember what happened two weeks ago. And the same thing with state legislators. And for that matter elected officials are even judges. You know, they're younger than we are, Matthew. They really are younger. Sorry. But the thing about AI, it seems to me is that AI and look back, it can remember. It can draw upon data that is from before. So it doesn't lose the benefit of the human experience gone by. What do you think? Right, so AI actually, this is a really good point. AI can be protected never to forget, right? Now it can be influenced. So what the important things from the past can be diluted, but you're right, AI could be a tremendous storage mechanism. One of the big issues that I've seen on social media this week is AI replicating Anthony Bourdain's voice as part of a media film, I think it was. And the who hard, if you don't mind me saying that on social media was really interesting to watch because some people said, well, that's okay. It's cool. The other one was like, no, that's terrible. And so we are going to see this remembrance issue becoming part of a social dialogue, Jay, where let me ask you a question. If in 50 years time, maybe you and I will be speaking, but it won't be you, it'll be me. It'll be our digital avatars, our personal bodies having a conversation. Now, this conversation about remembrance of ourselves and it being kind of eternal, that's an interesting philosophical question, right? Yes, yes. Well, there's a Holocaust project. I think it was started by Steven Spielberg that does that. What happens if they interview these Holocaust survivors, they get the answers, they get the words, they get the facial expressions and the body language and then the fellow dies. And from the video and the audio, what they collected in response to these many, many questions and answers, they can recreate a conversation with this person. It's quite amazing. It was on 60 minutes. And I'm sure that from the time it was on 60 minutes till the time you and I look at it next, it'll be way better. And so what you're talking about is already to an extent already happening. Oh, Matthew, I really enjoy these conversations. I can't tell you how it opens my mind to talk to you. And I hope we can do it again, promise me. Yes, of course. I would love to be back. Thanks for the opportunity, Jake. Aloha. Matthew James Bailey. He was very well-called by person in AI and we're looking forward to talking to him again. Aloha.


So this is the user testing crash course. I'm Jeremy Krigel. I'm the UX director at a company called Omnisfel. We do pharmacy robotics, which is like automating medicine dispensing, inventory management, et cetera. So I saw a keynote number of years ago, wonderful writer, Kathy Sierra, asked the audience a question. She asked, which would you rather have people saying how great your company is or how great your product is? And so some people raise their hand for companies. Some people raise their hand for product. And of course anytime someone asked you a question with only two options, it's probably a trick question. The answer is always C. The answer is how great they are because of your product or service, right? So there's things that you use that are frustrating. You hate them, but you have to use them to get your work done or try to accomplish something and your customers are the same, right? They want you to succeed. In fact, they need you to succeed so they can be awesome.  If we look at ability on the left and time on the bottom, she puts these two thresholds.  I'm not very effective. And then at that higher level, there's kind of that passions threshold or when you get into flow, like where you're really good at something, you're effective, you feel great and you lose track of time, that's when you're thinking, I rock, I'm awesome.  How do you do that? We've got to talk to them and understand how effectively they're using your product or what problems they're having that you could be solving. And as obvious as that is, and I'm sure most of you are thinking like, when I said that, like, well, of course, we know we have to talk to people. My experience has been is that there's a huge reluctance to actually get out there and engage with the people we're trying to serve. I've seen this in many, many different companies over and over again. And I get it, right? Talking to people can be daunting. We're really busy with other things. And if we find out things that, you know, where we're wrong, we might have to do rework and we're on a calendar. And it's this is out of sight of our comfort zone all kinds of reasons. But if you have a plan and you can be a little more comfortable when you kind of have some more confidence in how to navigate things, I find it can be a little bit easier. So I'm gonna talk about just a couple of things to get started and then we're gonna dig into the 13 tips to make it far more effective. So the first and most important thing is you have to have a focus, like why are we doing this? What do we hope to achieve? What are we trying to learn? What assumption are we trying to test, right? So a couple of questions you can ask are, where are we foggy? And that's FOG for facts, opinions and guesses. What do we think we know to be true that we need to validate those are our facts? What are we sure? We're kind of sure. We think that's true. And again, we might want to validate it. And where are we just guessing? That's probably where there's a lot more risk. What are our known knowns, known unknowns and unknown unknowns? Again, the things we think to be true that we might want to validate, the things we know we don't know that we need to investigate. And then there's the things that we haven't even thought of. We don't even realize we don't know. And for me, what I found, what I think is interesting is that the known knowns and the known unknowns seem to be kind of shared across the industry. Everyone kind of knows the same things as people move around. You read the same reports, yada, yada. And we kind of also know what we don't know. So the unknown unknowns, that's kind of the interesting part. That's where a lot of the insights that lead to innovation come from. And kind of finally, one of the last questions that I like asking is kind of like, where if we're wrong, we're in trouble. Let's just put it that way. So the first thing you want to do is make sure you're really clear on your focus. What are you trying to learn? What assumption are you trying to test? What do you want to get out of talking to people? You need to have some guide so you can make the session productive. So that's your why. And then, okay, who do you need to talk to to learn that? And the thing to remember is you're not your customer. Even if you were at one time, once you have joined a product team, once you've joined an company and you're making products where you've been someone like you, unless you are making products for other like product owners, designers, whatever your role is, you're no longer your customer. I'll give you a great example. A number of years ago, I worked at a startup that was serving physicians and it was founded by a former surgeon. Now that gentleman knew far more about what it was like in the day-to-day life of a physician than I will ever know. But at that point in his career, every day he spent as a CEO, he was becoming further and further removed from the day-to-day life of a doctor seeing patients. And so again, even if we were in that role before, we're not now, now we're product people. So we really wanna try and get as close as we can to the people we're trying to serve. So let's, who are those people? And one simple way is we can, let's try and define their graphics, demographics, psychographics, pharmacographics. Demographics is kind of the data that describes people, age, gender, income, all the data. Psychographics is kind of more about the mental landscape. So kind of in marketing, sometimes they refer to it as AIO activities, interests and opinions. What do these people do? What do they believe? What do they think? What are their hopes, fears, et cetera. And pharmacographics is kind of like demographics for the workplace. So what's the industry? What size of the organization? What's their role? How experienced are they? How senior, et cetera. And you might not need all of these things depending on your problem. Just to give you two quick examples. I worked for a customizable jewelry company many years ago. We were doing a study around engagement ring buying by heterosexual males. And so the demographics, we kind of knew a rough age range where a lot of people were getting engaged and income was important at that point. Psychographics were really important because we knew they didn't know a lot about jewelry shopping and they were really uncomfortable making some of these decisions considering their lack of knowledge. Pharmographics were completely irrelevant, didn't really care what industry, where they work, et cetera. And now working in pharmacy robotics. Demographics, if I'm looking at a director of pharmacy in a big hospital, the demographics might be less important. And they certainly be at a senior level you're at a certain age range, but it doesn't really come into play in terms of helping us make better decisions. And that's really the thing here. You wanna be able to make better decisions. Psychographics are relevant in terms of what's important to them, what they spend time on, where their focus is, how they adopt technology. And pharmacographics, what the size of the hospital is, what their level of experience, that's really important. So again, different problems will lead to different emphasis here. So now with that, I can create some assumptions. Again, I'm making some guesses about who I think I need to serve so I can start the conversations. So that's why I have my why, I have my who and then there's what am I gonna do? So one of the most basic things, right? We do is usability test and we put a product in front of someone, we give them a task to do. We see how effectively they can complete that task and that gives us a sense of where we need to make improvements.  You probably have observers too, preferably in a different room. We'll get to that later. There's also unmoderated testing where you can just put stuff online with a script. It's good for confirmation but you don't really get to explore things. Again, I'll talk a little bit more about that later. Another technique that's relevant in today's conversation is ethnography and that's more about discovery. Usability testing tends to be weighted more towards validation with a little bit of discovery and ethnography is much more open-ended. So I'm interested in this problem or this space. So you wanna go spend time with people, watching them do what they do now to understand what's working and what's not, how they think about things, then you can figure out how you might craft a solution that fits in with that. So those are the two things that today's techniques are gonna focus on. Now there are far more research activities you could do depending on the problem that you're gonna learn. You might put out a survey, you might do some cards and sorting, you could do a tree test but there's lots of research out there. But again, we're gonna really be focusing around these real-time individual interactions. But again, when you're planning, I thought I'd throw this out there. This is a variation on a canvas that Jeff Patton created but you can start with your hypothesis, your bet, what you believe to be true. What are the assumptions baked into that and what are the questions that you need to ask? Of those, what's the most important thing you need to test or learn now? That's number three. Number four, what are all the possible ways you could either test that assumption or fill in that learning? And then five, what are you going to do? And ideally that's the thing that you can do with the least amount of work that will give you the maximum learning. You're gonna still wanna maximize your ROI there. And then lastly, number six, what did you learn from that? And this kind of, there might be more than just what's gonna fit in the box in terms of report, deck, et cetera. But at least now you have sort of a one-pager that closes the loop. So if you have people joining your team or if you wanna go back and reference activity you've done in the past, you have these quick summaries you can go to. So in that first part, you have to have a protocol and there's lots of stuff out there. So I'm not really gonna go into a lot of the depth on this, but it's gonna have four basic parts. An introduction of why you're there and what you expect and setting some ground rules. Some context, which is usually questions to frame the activity and put the person in the mindset of what you want to talk about.  Again, lots of stuff online about that so we're not gonna dig in deep. I wanna spend most of my time talking about the nuances that make facilitating these conversations effective. All right, so here's number one. You wanna keep the environment comfortable so people don't feel like a research subject. And I kind of frame this as you wanna be just on the warm side of neutral. If you're too encouraging, you run the risk of creating an environment where your participant, instead of giving you their honest thoughts, we'll start thinking about what do I need to do to make this person happy. They're giving me all this positive encouragement and this positive feedback and now I really like this person and I wanna tell them what I think that they wanna hear and you don't wanna hear what they think you wanna hear. You wanna hear what they actually do and what they actually believe. So a little bit warm, but not too encouraging. And you don't wanna be cold either because that can be kind of like distancing and a turn off and then people might not be open to sharing. Just the warm side of neutral. Again, don't offer encouragement either. You can give a complimentary feedback like say someone says something like, wow, I really appreciate how you articulated that. That was very clear. Thank you for sharing. So you can participate in the conversation that way, but if you say something that's more like, oh, that's great, that's right. Something that implies there's better and worse feedback than again, the participant might try to figure out what you want and try and give you that so that they feel like they're performing. So that is taking that neutral attitude, maybe slightly positive. One other thing I'll say when you keep the environment comfortable, and this applies both in-person and online. You can imagine, earlier I said that you need two people. If you have a lot of people participating, especially in-person, it can really make that person feel like they're being studied. So if you have one person with a half dozen people huddled around them, that can be kind of uncomfortable. So especially for in-person, only have the facilitator and the note-taker in the room, and then other people can be, you can stream that content to another location and people can watch from there. Even online, try to, you know, if there's ways to hide the participants so that they aren't maybe aware of how many people are in the room watching them. They're a little easier online because once you get started, they'll sort of forget about all those other people, but you definitely want everyone on mute. You don't want anyone else chiming in. That is actually a role that the note-taker can play. If other observers have a question, keep a Slack channel or some other messaging channel open to the note-taker, people can message their note-taker and at that appropriate time, that note-taker can interrupt the facilitator. Excuse me, I wanted to ask a follow-up question and ask the questions on behalf of the other observers. So that's how we help keep it comfortable. They don't feel like a research subject. Part of that is that neutral attitude, just slightly positive, right? All right, that's one and two. Number three, you want people to think out loud as much as possible and that's because we want to understand what their mental model is. So anytime someone goes quiet, I'm going to ask them a question. What are you thinking? What are you looking for? What were you expecting? Something that's just going to keep them talking because the more they talk, the more I understand. So anytime someone's quiet for more than a few seconds, ask them a question again and talking.  This is one of those things we often are very clear about upfront that we want you to talk constantly and if you're quiet for even a moment, we're probably going to ask you a follow-up question just to kind of understand what you're thinking. All right, next one. You want to keep it simple in terms of language. Clear language, no jargon. Even if you're in an industry that has a lot of very specific verbiage that you would assume your participant knows, it's better to minimize it's use.  That will tend to make people shut down. So even if there is a specific term, try to say it in a general way that's more accessible in case they don't are familiar with that term. Keep it open. So you want to use open-ended questions as much as possible because they expand the conversation.  Well, that's yes or no. It doesn't really tell me a lot. I can open that up say like, well, what did you have for breakfast? That's a little bit wider. I could say, tell me about your morning routine. That's even broader and I can find out whether breakfast was part of that if I care about breakfast. Now, especially when we're not practiced at this and even when we are, it's not uncommon to end up asking a closed-ended question. That's okay. You can follow it up with an open-ended question.  Well, what is your typical morning move like? You know, then you can open it up from there or how does that compare to your normal daily routine? And then they'll, they can start talking about that. So even you can always follow up a closed-ended question with an open-ended question. All right, this is one of my favorites and this is from Byron Holtzblatt, talked about this in their book, Contextual Inquiry, which is still a great foundational text for how to engage with people and really learn a lot about what they need. And that is to think like an apprentice. Usually when we think about research, we think about the sort of the scientist, you know, when they're white code and they're clipboard or whatever, studying the lower life forms. And so like we're doing the research, we are obviously these superior beings. We are studying these other people, you know, our little organisms that we're gonna, we're gonna help out, right? Help benevolent of us. We really want to invert that. They are the experts in what they do, right? So we're not the experts, we're trying to learn from the other, these the people that we want to serve. So we need to think that as an apprentice, they're the ones with the information that we need to get. So we need to invert that hierarchy and think of ourselves in that lower position and put the people we're talking to in the higher position. Now along with that, what someone else says is true to them. It may not be objectively true, but it's true to them. I'll give you an example. I'm gonna go back and get into the jewelry company. When we were doing that research study, one of the issues that we were aware of was that guys don't really have a lot of experience in how to correctly identify the ring size for the person they want to buy a ring for, especially if they want it to be a surprise. So one of the questions during the study we would ask people is, you know, during the process and when they get to ring size, how do you know what her ring size is? And one person's response was, oh, that's easy. It's the same as her shoe size. If you don't know, there is no correlation between ring size and shoe size. Possibly there might have been that coincidence, but needless to say, the folks in the observation room thought that was hilarious, which again, that's why it was really good. The observation room was a couple of conference rooms away because you really don't want them, the participant to hear a lot of laughter and sometimes they will say funny things and think that that might be related to them. So observation room, farther away, good idea. Anyhow, so while on the face of it it was like, this person said this kind of wacky thing, isn't that abusing? When you step back a bit and say, well, what does that mean? What that demonstrates is, we have a really long hill to climb to make sure that people understand this particular concept and how to be effective, right?  So some of these things are pretty important. And again, we wanna make sure that we understand what we need to do. So what they say is true to them. Take it and learn from it. When you're asking questions, you wanna avoid hypothetical questions. They're unreliable. Observing action is the best. Watch people do what they do. That's gonna be the most reliable thing. This next is have them recall something they've done in the past. They will leave things out sometimes intentionally and sometimes unintentionally. Sometimes we have an image of ourselves that we wanna project and so our brain will filter so we maintain that image. That's why observation is best, but recalled will give us some important parts of the story. And then lastly, we get to imagine what you would do in this case, but if you're not in that scenario or you haven't been, it's hard to be, it's hard to get reliable data in that case. Better to say, how have you solved this problem in the past? So I always think of the introduction of like 3D TVs. And because they were really touted like, oh, this amazing technology, 3D TVs. And I can only imagine they must have had a lot of sessions. They brought people in and they sat them down. They said, isn't this amazing? And a lot of people went, oh my God, these are amazing. And what they probably didn't say is, well, tell me about the last time you bought a TV. And because they probably would have found out as well. Yeah, well, a few years later, HD TVs had just come out. So I'd upgraded my standard definition to a high definition. And well, how long was that take? Well, I had that old TV for seven or eight years. All right, well, if the last, you just got the last one three years ago, how likely are they to upgrade again? So again, you start to understand how someone has solved the problem in the past. And that's setting aside that 3D makes a lot of people dizzy. But anyway, how people have solved the problem in the past gives you a much better sense of how they might solve it in the future and how your solution might fit into that. Next tip is to follow the thread.   And you don't wanna just kind of boom, boom, boom, boom, go through the questions. The goal is not to get to the end. The goal is to learn. So when someone says something that doesn't quite make sense, ask more questions about that. Those can be those unknown unknowns that lead to different insights. I mean, sometimes you end up with an idiosyncratic participant who just has a different view on the world or whose context is just different from everyone else's. That's fine, you can still learn from that. But sometimes someone says something that has kind of profoundly changes how you look at the problem. And that can lead to some really interesting solutions. So this is why I like moderated testing versus the unmoderated and just putting something out there and letting people go through it again. That's why that's more validation. You only get the answers to the questions you ask. Same problem with surveys. So when someone says something interesting, you can't follow up on it. So be curious and that's why we have multiple participants. So if we don't quite get to every question with every participant because we followed some interesting threads, you'll still get the answers you need because you have enough people that you're talking to. All right, moving on. Echo, boomerang and mobile. These are three questioning techniques that can be really helpful. A conversation is kind of like playing catch, right? We throw the ball back and forth and that's what keeps the conversation going. But as a research facilitator, I don't wanna add content to the conversation because that has the potential to introduce bias. And that's what I wanna remove or I wanna eliminate my bias. I don't want any of my bias into the conversation. So these are the three ways to continue the back and forth without adding content. So Echo is basically saying the same thing back to them. So someone says, so what I'm gonna do here is I'm gonna be looking at this data to see if this thing is true. And you can say, oh, so what you'd be doing is looking at that data to see if that thing is true. And they go, yeah, yeah, yeah. And then they'll continue. So they said something in pause, which indicates it's your turn. You took your turn, but you didn't add anything to the conversation. But you took your turn in the conversation, now it's back to them and they will continue their story. Boomerang is very similar, but it's related to questions. You don't really wanna answer questions. You wanna, because again, that's now adding meaning. So someone says, the comment, what does that button do? Oh, well, this button is gonna do this thing and that thing, no, that's not what you wanna do. That's bias. You're telling them. And now again, you imply there's a right answer. Instead, you send the question back to them. What would you expect that to do? What would you like it to do? What would be useful to you? The Boomerang, the question back. You don't answer it. And the mumble technique sort of inspired by this 70s TV detective, Kojak, who mumbled a lot. But the thing with mumbling is you take advantage of this sort of social phenomena where people become uncomfortable with extended silences or pauses in a conversation. So if you're not sure, especially what to ask next, you can sort of mumble your way through it. So what you're saying is you need, I don't even know what I'm gonna ask, but you just kind of slowly and maybe a little awkwardly start to form your question. And it's gonna be weird and they're gonna fill in those gaps. Try it, it's brilliant. So echo and repeat back what they said, don't add. Boomerang, a question back to someone and mumble to kind of keep someone off talking, especially if you're not sure what kind of follow-up question you wanna ask. Don't give people instructions. Your job is to learn not to teach. This is really hard for product people because it's, you know, product people have created the product. They know it really well and they want to show someone. So if you see someone doing something that could be done better, there's this impetus to jump in and say, oh, let me show you a better way to do that. But again, now you're creating that impression that there's a right answer. So just help them understand, ask questions to understand why they're doing what they're doing. And then at the end, you can say something like, hey, you know, I noticed that you were trying to accomplish this. There's another way in our product to do that. Would you be interested in me showing you? And then you can do it then. You've already learned and then you can offer some suggestions. The no answer is like, what would you expect? This kind of gets back to the Boomerang. Don't answer their questions. And kind of the last one is don't make excuses. And I find this to be most common either with startups or anytime that you're testing a prototype that is not completely built or something where there's bugs. People start going, oh, I'm really sorry. We meant to get to this. We haven't done that. Again, if something doesn't work in your prototype that you're testing, it's a great opportunity to ask questions. Someone goes, I'd click on that. And they want to click on that. And it doesn't do anything. Maybe that wasn't in the flow that you're trying to test. You'll say, oh, I'm sorry. We didn't build that yet. You can say, oh, what would you expect that to do? What would you expect to happen when you click that? How would that be valuable to you? And you can ask those questions, understand what the person needs, and then you can redirect them back to what you're trying to learn. Those are the main tips. The last one is not really a tip. It's just like, we should record everything. Always inform people first, but even with a note taker, you sometimes just end up with notes that you go back on and you're like, I don't know what this means. So at least you have a recording that you can refer to. So I always tell people, I'm gonna record in 25 years. I've never had someone say no. I had one person in 25 years who asked me to delete their recording after two months, which I did. I agreed to that. By then we had already had everything needed. I was happy to delete the recording, but no one else has ever objected. I do clarify that it is only used for internal purposes and will never be made public. All right. So those are my tips. We got about what, 15 minutes left, 14, 16. So we wanna try this out a little bit. And it's gonna be, we're finally gonna do one round. It's gonna be fast. And so you're not gonna get this right. But what we're gonna do is we're gonna break into groups. I think we'll do groups of four. How many folks do we have? We've got 12, oh, we've got 12. Maybe two to three groups, whatever makes sense here. So pick one person as a facilitator.  Or in this case, I say in quotes, usually the notetaker would be taking notes on things the participant said. Since we're not really testing something, we don't really care about what the participant's saying. What I want you to do is observe how the facilitator is asking questions so you can give them feedback. So you're gonna be more taking notes on the facilitator. So if we're at groups of three or four, you can have multiple, multiple notetakers. What I want you to do is what we're doing, getting the breakouts is just bring up, I believe it's housing.com, is understand as a housing site in India. And so that as a participant, your job is you're searching for a place to live because everyone's done it. And so if you're the facilitator, ask them a question or two about what they're looking for, what they would wanna do, and then just try and get them to do some initial searching through using that site, ask some questions, try to keep in mind some of the things that we're talking about. We'll probably do about five minutes. If you wanna screenshot this, these are the list of tips, if you wanna have those as a reminder. So I'm gonna give you five minutes once we do the breakouts to have the conversation. And then we'll do a broadcast just to let you know that that time is up. And then at what I want for another like three minutes or so is for the people who are playing in the note-taker role to give feedback on the facilitator. And then we'll bring everyone back. And if anyone has any other questions for me or on user research in general, we'll take those questions. All right, any questions before we start if you wanna unmute and ask one about what I just laid out? Anyone have any questions about either what I went over, user research in general? Hey Jeremy, yeah. Thank you so much for the session. Yeah, it was quite interesting. And it was real. When I said real, I can't read it in a book. Actually, it's like a real time. So I had a question in my mind like, how do we master it? Because I'm sure you have collected it over a period of experience. Like so it's a several maybe decades of experience you have collected it and it is working for you. Now a lemon like me, when I would like to really explore this, what are the tips you recommend to start with? Sure. It's actually easier than you'd think, but it does take practice. But there's lots of ways you can practice at the professional level. If you're recording your sessions, you may want to listen to your sessions twice. The first time you might be taking notes and going back and go, did we learn what we intended to learn from this? And that's your goal from a business perspective. The second time you want to go back and listen to it, you want to be thinking, how to evaluate yourself as a facilitator? What did you do well? And how would you facilitate that differently? Like, oh, you know, I asked that closed-ended question there, I didn't follow it up. What open-ended question would I ask? Okay, I would ask this thing if I would do that again. Or, hmm, you know what, they said something interesting. I didn't follow up on that. Okay, that'll start to create those triggers so that you can, you know, you'll be better prepared for future conversations. Now, the other thing that, frankly, you can also practice this is every day because the not-so-strange truth in life is that people will think you're a much better conversation list the less talking you do. Because people love talking about themselves. So if you give someone the chance to talk about themselves, they'll love it. So you can use all these techniques. When you're just talking to a friend, family, and you're asking them about their day or something that they care about, think about, well, how do I go deeper in this? How do I ask more interesting questions? How do I, you know, use these techniques to get them talking about this thing that they're interested in? Because frankly, most of us, when we're having conversations, we're only half listening and we're really thinking about, what am I gonna say next? And so when you're thinking, when you're just engaging with, again, friends, family, colleagues, whatever, you can put that aside and go, I don't need to contribute to this conversation. How do I learn the most about this other person that I'm engaged with? Also really hopefully a conference is, you know, when you get a chance to meet other participants, then you want to learn about their professional experience, all kinds of opportunities to practice this. Mm, thank you, thank you. Yeah, it's only that silent sometime. As you told me, the silent is the uncomfort. Yeah, the silent one is a little bit harder from a social perspective, but you'll still find people will fill that gap. Again, we love talking about ourselves. So given the opportunity, we'll do it. Most of us, not everyone. So Jeremy, like nowadays, like as you have rightly noted down also, that most of the time we assume that, yes, I know about my user, I know about my customer. And even if we start interacting also, like as you rightly said, like we are busy in actually asking the questions and collecting the surface level information. Now to go deeper, to really understand what they're supposed to tell, but you actually told that like, I want to hear what actually they want to express that about the product. So is this like only through that opening the questionings or allowing them to space and talk, like is that the technique? Let me see if I understand you're asking, is the only way to get that insight through the open-ended questioning? Yeah, the deeper into that customer expectation, like how do we go deeper when they do not express, when they do not tell, when they are not even comfortable also to spend time with us? Yes, so spending, the spending time is an interesting one. But there are different ways to do that. In the longer version of this, I talk a little bit about recruiting participants. And sometimes you're gonna pay people, right?  Now, depending on how hard it is to get that type of person, you may have to pay them more. A more common profile, you will have to pay less because there's more people out there. But if you wanted like, I'm looking for orthopedic surgeons, well one, they're highly compensated and there's not a lot of them, so you're gonna be paying more for their time. Now, that being said, I think you can often get away not paying people at all by appealing to ego. Like if you approach people to say, you know, look, we're building this product for people like you, and we think that you would offer some really unique insights that would help us make this product better for you and people like you. So you're doing a couple of things there, right? One, you're telling the person you're unique and special, which is true. And they will give you some great insights that might really help. But everyone likes to think of themselves as, of course, I'm very insightful. I have great things to share and you're validating that. So yeah, I'd love to talk to you about that and share my brilliant insights. But the, and the second part of that is, if I use that product, I certainly wanna make it better for me, right? If I give you feedback, it's more likely that it's gonna be better for me. So there's a bit of self-interest there. So sometimes you can appeal to that and not have to pay people at all. And in fact, I sometimes think you can get better feedback that way. So there's just different ways you can approach people and you have to kind of make it about them. Why it benefits them to talk to you, not why it benefits you. Like, hi, I need to spend time with you because I need to get my job done because my boss is on me and I need to produce this thing by this. Like, I don't care, that's your problem, not mine. But if you can make it about why I should care, then I'm more likely to say yes. Thank you so much. If you, one thing I will share, if you wanna get in touch, if you have any questions, you can reach me at J-E-R at m-s-m dot works. That's email on Twitter at Sonark, S-O-N-A-R-C. And I also have a podcast called Saving UX and that's at S-U-X for Saving UX, S-U-X dot live. So if you wanna reach out or find more stuff that I'm up to, those are the three ways to do it.