Welcome, Aloha. Thanks so much for joining us on Think Tech Hawaii and we're going to have a fun one today. We have with us two very, very experienced professors of law and Davis, Professor Emeritus from the University of Toledo School of Law current adjunct faculty at Washington Lee in Charlottesville and David Larson, one of the pre minute professors at Mitchell Hamlin School of Law in St. Paul and past chair of the American Bar Association section of dispute resolution. So, gentlemen, your challenge for today is you have been hired as consultants and you're only given 28 minutes to come up with a set of objectives and a set of factors and criteria to achieve those objectives in college admissions at both the undergraduate and graduate level. Yes, I'm grateful to have the opportunity. What objectives do you want? What are your top priority objectives? So, can I start with my original list vision? Okay, so I have an original list vision, which is in the grant creating Phillips Exeter Academy. John Phillips in 1781 said he wanted students from every quarter. That was his in his grant of starting that school so the mission would be to call it our missions policies to have students from every quarter. Whatever that means. Okay. Old school. Now there were there were men until 1970. Okay, boys. You get what I'm saying. Okay. So, by that, do you mean just geographical diversity? Are you talking about every quarter of socioeconomic or educational or. Ethnic or. I wouldn't, and since Harvard and UNC decisions, I wouldn't go too far, but I would say in every quarter in quarter meaning the most broadest vision of what quarter could possibly be. So it could be international. It could be geographic. It could be different rural urban. It could be farmers. It could be the plutocrats. It could be poor chef profits, whatever every quarter we would look everywhere. This is the vision. Yeah, I think it's a, you know, if it's me, I'm just thinking and maybe this is a little child idealistic, but I'd like to have a class that reflects the diversity of the population. You know, that, that, you know, I don't want to quote it doesn't have to be exact, but I would like the faces in the classroom to be representative of the faces of our citizens. And that's kind of where I start. Now it's an interesting time for colleges because there's some recognition down there's some push now, and maybe you don't need a college education. Maybe you can get a great job by getting different kinds of training. And right now, the technology's speared. There are companies that are saying we're not requiring college educations. Now it's just going to associate degree in this area and a willingness to work and will train you for whatever else you need. And there was a news program up here in the Twin Cities within the past week of a company that's doing that.  So that's one year after you would have completed your college education those three years because they only went to associate college for two years. So I'm not sure how this cuts but as I'm thinking about diversity in college and now what's the best path for everybody. I'm not sure anymore that the four year degree is the best or necessary path for everyone. And then I'm not sure how that cuts in terms of our diversity representation and our population. But I'm not sure I want to be in the position where I'm saying that you, we want you to go to the four year college because we think this is best for you. And don't consider these other options, which may be a lot quicker path to success for you. So I just think that that muddies the water a little bit fact that maybe the pathway to success isn't necessarily the four year college anymore. Maybe it's another path. So we've got a couple of concepts out there we've got maximum diversity in all senses. We've got representative diversity, and then you might have questions of do you mean locally representative regionally representative nationally representative internationally representative. So the qualifying adjectives for each of the categories we come up with are themselves either broadening or narrowing types of qualifiers. But there seems to be at least a sense that that diversity that enables representation of groups as diverse as possible as people on people as diverse as possible. It is an objective. How might you evoke and prioritize criteria that might help generate that kind of interest that kind of motivation application and selection. And at the Harvard case that was recently decided justice so to my are observe that. Well, so people are discussing these race neutral opportunities that are available is not in so many words, let's not kid ourselves, the whole goal of the so called racial neutral alternatives are to get a racially diverse population. If we're being honest, why would we exclude a more direct reflection on racial membership. If the fact is all these other alternatives are trying to do essentially an end around to get there anyway, because I make any sense to me. Now, if that's our goal and we believe that's valuable and and we're willing to accept other pathways to get there. Then why not just accept this pathway and and and not pretend that we need to do something else. And that's a great question. Can you generate an outcome that is both diverse and balance in a way that enables representative participation for as broad a range of groups and people as possible. And if you do that, are there ways other than using those factors that would enable you to measure that diversity. There are, you know, two that have been posed have been wealth and, and not just parental income don't know that's, that's been criticized as not closely enough correlated, rather use wealth and maybe neighborhood poverty level that you put those two things together, and you might be capturing a greater percentage of underrepresented populations, but now if that's what you're going to do, you can use like wealth and neighborhood poverty level. You're going to the reality is you're going to use those preferential criteria, you're going to have to use them more aggressively, because it's not going to be as targeted. So the fact is, even though you're saying you don't like people using preferences, in order to achieve those that desired goal of representation and understanding, you're going to, you're going to, you're going to collect a wider group than the one really you're targeting at. You're going to need a more aggressive use of those other other preferences to get the diversity you want. So again, that, that idea that why are we doing this. Why are we, who are we kidding. That that's just a far more inefficient way to get there and you can do it. But why should you have to. Well, what, why would we do something efficient. My goodness. What, what, what, what's the interest of doing something efficient. I'm just playing with devil's advocate here but so one of the things that I was thinking was that if you use neighborhood, you know, like zip codes or something like that what you're doing is you're just like, basically, enshrining the resegregation right you know what I mean. I mean, in a way, if you're going by that kind of a criteria it's like, please stay in your poor neighborhood you have a better shot than if you go into a richer neighborhood even though you know the American dream was like to go to get a better house in a better neighborhood and all that stuff. I mean it's a kind of fun one. Personally, if I was on an admissions committee at a law school or college, I have the absolutely perfect solution. I'm certain you will all agree. So here is the solution. You have what I call a clean team. Okay, or a clean officer who is now a new position at your admissions office right. Now, according to the decision, the student who is applying as a preceptor student can write anything they want. So they write everything they want to my hypothetical student rights. I am the descendant of 10 generations of black sharecroppers in the Mississippi Delta, having suffered incredible racial oppression all my life and I'd like to come to your school. The clean team takes that and then they just put it to an algorithm which eliminates any references to race. So I am a descendant of 10 generations of sharecroppers who have lived through great oppression down here in the Mississippi Delta. I don't see any race in that. Okay. So then now cleaned up they take that essay and now they give it to the admissions committee that makes the evaluation about that particular candidate whether they would be an interesting person for this all diversity representation all quarters thing like that. I think it solves the problem and but those essays could also have anything about disability. They can have anything about their religion. They can have anything about their gender. Let's see what else can they do. They can talk about their political views. They can throw anything they want. And just one of the words is anything about race. The algorithm would make it disappear. And then you would have this now cleaned up essay being given. By the way, you could have a senior member of the faculty that's still going on the admissions office and say that Joe Johnson is a wonderful fellow. That's just applied now. And hopefully, you will give him a favorable review. That's all fair day. Nothing's changed. All we did is got rid of the little color word. What do you think that that work at a college or a grad school day? What do you think? I think probably someone would immediately challenge it by saying that. I think that the description here makes it clear what words have been taken out. We can read that in. But I'm not so sure that's an automatic winner because the reality is there are people of all races who have been in oppressed positions. Blacks. You know, there's been Asian slaves as we move in the western part of our country in particular. You know, so, so, so they'll be the always challenge that well, clearly it's a race. It's a black, it's a black race considered attribute. And I think the response is that not necessarily. Yeah. Now the question I had was I was that applicant that I was black and I found out that my essay had been what is it called a bridge, a bridge when the admissions process happened. You didn't treat me fairly compared to the other one.  Yeah, my, you know, my first reaction is, well, that's differential that's treating similarly situated people in a dissimilar manner, which is the basic kind of restarting definition of discrimination that there's other protected classes that are not getting treated the same way. Now, I suppose the twist on that is the Supreme Court said, yeah, but you can do that.  So I think the starting position is, yes, it sounds like discrimination based on our definition of what discrimination is. But then the counter is that Supreme Court now said you can do it. So, so I guess that's, that's the challenge. Oh my goodness.  Doesn't that get you back into the same historical problem though, which is historically, we have particularly elite colleges, but many colleges who have more academically qualified applicants than they have spots for. And so the tilting factors may very well be factors unrelated to academic merit. They may be legacy. They may be athletic prowess. They may be financial resources coming from a family of historically big donors. None of those are related to the academic qualifications. And I think most universities and colleges would point out that no, they don't want to limit their constituency to just those who are most academically qualified. They will be more around it well rounded than that. How do you shift the balance from that historical favor of privilege tipping factors over to a set of possible tipping factors that may be related to things that are non-academic, but they do balance those previous white male oriented tipping factors. I think is one is, you know, shine a light on it. I mean, I think that one thing that came out of this court, there's, I think, Richard Collenberg, I think was the name of the expert who testified in the Supreme Court case. And, you know, he went through different preferences and looked at legacy, looked at dean's list and looked at children of faculty and offspring. And I think the LDS, I think, was the LDC, I think is the acronym we put out for those three preferences and said that, you know, this accounts for 20%. In 2019, 20% of Harvard's admissions, one in five students. And he said, now keep in mind that some of these students may have been admitted anyway, but not, you know, not that many. So I think one thing you do is you shine a light on some of these other practices and bring them into questions saying that, wait a minute, you know, what are you doing here and how can you justify these things?  Right. And you made me think of something else, which I think is in the 30s or 40s that Harvard went to this thing or started to push the thing about we want a lot of geographical diversity. And there were a lot of Jews from New York who were applying, who weren't getting in, and really underneath it, it was this sort of religion neutral thing, right, geography. But the feeling was that it was really antisemitism that was driving it because they had these excellent academic candidates coming from large Jewish communities in certain places, population, right. And, and, and, you know, there was one of the arguments is what maybe that's one of the reasons Brandeis got created, you know, because of this structural gain that this elite institution was playing, you know. So, you know, even if you're going with geography, we want to have, you know, 50 states, you want 2% of the class to be from every state, right.  Well, yeah. Thinking about what some alternatives are some of the in states that have banned express use of any kind of racial preference. Texas, for instance, has gone to that top 10% to your high school class. You know, I think that is that is one way you can diversify it, because you do have schools that are segregated. And if you're taking top 10%, regardless of test scores or anything else you're taking the best students from the schools that that will provide some diversity. So that seems one way to to approach this. Then the question was, was the size of the class that you ended up having if you took top 10%. I mean, they had to get more students than they than the 10% totals were like the University of Texas or something like that. So they had to move to some other stuff, too. But yeah, I mean, you can come up with these ostensibly neutral approaches. But it's kind of artificial, all of this. Don't you feel it's kind of artificial. Yeah, goes back to Sotomayor. It's like, yeah, we all know what we're trying to do here. And we're, we're eliminating the most direct route to it. And why are we doing that? You know, if you're willing to look at this myriad of other factors to achieve a goal, then why not use the most direct route there. So an interesting possibility that some schools employ is that because they think it's valuable.  Yeah. If you use that, the one thing that has been demonstrated statistically is that Latinx students, then that population rises. It will increase enrollment because just because of the reality in homes and first generation homes, you might be speaking Spanish in the home. So that's one facially neutral criterion that actually at least for a population, probably some other ones, too, has increased enrollment. So, looking at these non-academic tipping factors that historically have been associated with a particular race and for many years a particular gender and a particular socioeconomic narrow group. How do we move the non, certainly non-academic tipping factors are going to continue to play a role? How do we move those in the direction of achieving that kind of egalitarian representative diversity that we're talking about? I was thinking that you may have to be very, very careful because the thing that I'm thinking about is outreach efforts, right? So, they're sort of the easy people who show up and know the game and apply and maybe got those $300,000 fee consultants to help them write the most immaculate essay ever, types, right? And then there's like somebody in, I don't know, not in Mesa, Arizona, but Surprise, Arizona. There's a small town for you, at least it used to be.  So, now you start a real strong program to go out to Native American reservations or go into inner cities or something like that.  And so therefore your outreach is illegal under the Harvard and UNC.  I don't know, but that's, I think that that could actually be a target of, could be placed on institutions that tried to do more. If it could even be seen to have some kind of taint of race going on there, you know? Yeah, well, I think you're right, Ben. I think admission committees are going to have to be very careful because I, you know, people are lining up to challenge schools. And we've gotten at our school, we've gotten letters saying that we're watching, you know, and if we believe that you're using any race-based preferences, we're going to sue you. So, admission committees are feeling pressure. You know, and I suppose that was one of the goals of, you know, some degree of intimidation. Right. You know, we're going to, we're going to scare you into trying to do anything creative. And so don't even give us a hint of that. And anything that even looks spatially neutral that we believe is a end around. We're going to challenge that. So I just want to suggest that when schools get those kind of letters, they should thank you for your letter and we are watching you too. Okay. And have a nice day. Okay.  You know what I mean? And I had to sit down with the dean, you know, what's going on, what are you doing? I was a private citizen. And at the end of it, I said, could you please do me a favor the next time somebody writes you? Ask them where they were. Okay, so I can contact their employer about them hassling me. You know what I mean? So, you know, Right. We're watching you too. And by the way, you did not take an entertainment deduction for that fishy part, did you? Yes. So, yeah, that's a good response. You know, another thing schools have done, and a lot of schools are doing it. They've, they've backed away from just looking at test scores and saying saying test scores are optional. Some are saying, even if you send them in, we're not going to look at them. We don't want to look at the test scores. What that means is that admission committees are going to have to work a little harder. And that's, you know, that's such a shorthand. I'm going to look at your grades, look at your test scores. That's all I need to look at. You know, that makes my job so easy.  You're going to have to find out a little bit more about your, your candidates. But that's probably a good thing. So, you know, I kind of applaud moving away from that hard and fast standardized test criteria or criterion, you know, and doing a much more broad holistic approach. You made me think of another thing, which is what about schools that interview potential students? Okay. You know, it doesn't have to be like that whole thing that happened in the symphonies where there was a problem of having women be sufficiently represented in symphonies. So they went to the thing where they had the white sheet up and the person would play behind the white sheet. And so there the committee would evaluate them and that out tended to increase. So, you know, a lot of alumni do this for schools and write up very important reports about the persons they meet. But now do they have to put a sheet up between themselves and the potential candidate so they can't say that anything about that person's race came through. You know what I mean? I mean, I don't know, but, but you could see how if you were really going to be persnicketing and being watched or like that. Oh, yes, but those alumni interviews are all focused on or have a race factor in them. So, you know what I mean? So, you know, I don't know. I just, it's something that I think about at least. So as we get down to our last minute or two, I really like the shine the light idea. Let's shine the light on those non-academic tipping factors. What those have been historically and the influence they have had, what might counteract them.  Pretty much anything that his political perspective opposes. Well, you know that the whole idea of shining light. You know, I think in 2023 has become so much more important with the kind of evolution of AI and the ability to misrepresent truth. And we are going to have to work so much harder to make sure that we can defend the truth because the fraud is going to be really hard to distinguish at first glance.  You know, unless we're prepared to rebut that and respond to that, people are going to believe it and we're going to have to be very careful. Yeah, you know, and chat GPT, you know, I can write a query to chat GPT is what my essay is, which is supposed to like, hey, show stuff that won't say my race, but will really, really kind of give the hit hit wink wink. Okay, you know, and then have chat GPT write my essay as my first draft. I'm not saying it's my last draft, but I'm saying my first draft.  You don't need your clean committee, right?  They're like, oh my God, I don't know, but I just I want to emphasize that, you know, I used to be a prelaw advisor, and I used to read the high school package that had been submitted by students who are now already in college right. Oh, just a second. And I want to tell you that personal interview thing, it where they would grade somebody on one to five or something like that, or they write these personal notes about somebody they met. Those were really interesting things that personal interaction between the that puts perspective student at some alumni somewhere. And there were things I read I remember where I knew the student now right, and I could understand why they were picked, because what they cause somebody to react to when they were in high school. I could see it in them as a junior in college, you know. And so, you know, we got to understand that that interaction that that if there were schools that do do that. Can talk about leadership skills enthusiasm, dynamism, all those kinds of factors are things that you could look at somebody you say hey that one, we want to have here. You know. We can't lose that I personally think we can't lose it because there's so many amazing people that are there. That's great David your last stops to wrap this up. Well check I just want to thank you really for hosting these programs and bring these issues to the forefront. I think what we're doing tonight is what everybody needs to be doing is thinking critically about them discussing them. You know, don't just let these things roll forward without critique and without opposition. So kudos to think take away to you and to Ben for joining us tonight. I really appreciate it. Thanks folks think tech Hawaii. Particularly to those on Maui and the big island and Maui, especially. Find your favorite charities and help out because it's worse than you can imagine. And it will be appreciated and needed. Thank you all with a lot of think tech Hawaii. Follow us on Facebook, Instagram and LinkedIn check out our website think tech Hawaii.com. Mahalo.


Good morning everybody. Or good evening or wherever time. So when you're in here in Redmond, it is morning. It's Thursday and it's time for another episode of the Visual Studio remote office hours. So since last week we had a few updates here, or I had a few updates to my home office. Look at this thing. I got like this microphone arm. It's a very cheap one. It's but it does the trick. And here's a really cool thing. So when I'm not doing my video recording or doing meetings, I'll do this. And it's gone and you can still hear me if I yell right. Very cool. This is probably the thing I've been most excited about because it makes the quality so much better of my voice. Super happy about that. Here's another thing. I forgot my pen. This is the surface pen. I forgot that in the office and I went in the other day to pick it up. I found myself doing way more screenshots now than I kind of ever had because I can no longer just show my screen to a colleague. I had to do a screenshot and send it to them often. And so I do a single click with this on the top here. And that makes my screen into the screen capture mode. And then I can just outline what I want to share. And it automatically copies it to my clipboard. So that's really cool. So the same thing happens if you don't have a pen. Windows Shift S, you can try it now on Windows 10. That will bring you into like a screen snipping mode. Anyway, so that was a few updates on the home office. This is gonna be the last time you're gonna see the basketball hoop. I'm gonna move that out. So I'm sorry for everyone who's been eager to see me try to dunk a ball in that. Not happening. All right, so that was an update on that. Today we have a fantastic show because we are getting a rare insight into how our internal customer research on the Visual Studio team takes place. We have labs, we have interviews, we have surveys, we have a bunch of things that we do. And typically it's the program managers, such as myself and others that talk to customers and learn things. And all of this goes under the umbrella term customer research, I think. And here to talk about that, we have Mr. Research himself. Carl, welcome to the show. Thank you. Thank you for being generous. Thank you for welcoming me into your garage. Yeah, I mean, I feel privileged and it's like not many people get to be part of this, so thank you. I wonder, Carl, before we get into sort of the meat of the matter, you've been at Microsoft for many years. So if you would mind introducing yourself to everybody here. Sure, okay, yeah, I've been, I'm the old timer now, I guess at this point. So I've seen sort of the changes in how we engage with customers from over the years. And so, but yeah, my title is UX researcher. And my job is to, well, a great way of thinking about it is to help us as an organization figure out what is the right thing to build and how to build it right. So the first half is about what do customers really need and trying to unveil that onion to try to figure out what is the really core problem that we need to solve. And the second part is, you know, you can think about it as the usability part of it, the utility part of it. How do we make that usable, not only from the discoverability part, but also from that day-to-day usage part, right? That sort of I'm working the editor and it has to work smoothly for me. So I'm helping the organization figure out those two halves of that equation. So anything else you wanna know about me? I was an old C programmer way back when did image processing stuff, sort of discovered human computer interaction by accident. And I had a degree in medical computer science and did this research at a cellular level for many years. And so then I just figured, hey, we're multicellular creatures. So I'll take that skill set and just translate it to us. Anything that have worked okay. So. That's very cool. I think it wasn't Bill Gates who, when he was asked the question, what he would do if he wasn't into computers, he said he would be in like biotech. And there he is. So that's the opposite maybe of you. You came from that world. Maybe, maybe, exactly, exactly. So I guess maybe those, more interesting the other side of the fence I guess. So anyway, yeah. Okay. So we're using basically your team is sort of one that facilitates the various product teams around Visual Studio. I think a lot of people maybe are not aware of this, but we have typically the way we're organized within Visual Studio is that we have a bunch of product teams, we call them, or feature crews. And they're basically usually at one PM and about five to seven engineers. And that includes the engineering lead. And they work on different things. So there's one of those teams. I used to be on one that did Visual Studio extensibility. There is one for the editors. There is one for source control, for the Git integration, and so on and so forth. And they all like, of course, work together between those teams, but also they are sort of their own unit at the same time. And they each come to you, right? And they say, Carl, I'm gonna build this feature or I've heard about this problem. Can you help me? Is that how it starts? Very often it does. There is also that we're not quite aware of the problem that we have yet, right? We think it's a problem, and but we don't understand what is really going on here. So we wanna sort of really like have people use this stuff and see this stuff and try it out for us to figure out what is really going on here. So it's some cases, the trigger is not a feature, but rather we have these students coming up to Visual Studio and they're installing it and not running Visual Studio. What is going on here? And that's where I bring in the sort of the techniques to get into that deeper level of understanding as to why and what and who and where and all that fun stuff. So that's where I come into play. So it starts at different places. It's sometimes it's an idea that somebody has, would customers want this? In other cases, it's like, what the heck is going on here? So it can come from many different directions. Okay, so you said something that grabbed my attention. You said that some people or students or whatever would install Visual Studio, but then they would never run it? Exactly. How did you figure that out? What's the reason behind that? Wow, okay, great question. So yeah, it had us flummoxed, right? I mean, when you ran the numbers and looked at, you just spent all that time installing Visual Studio. Why not running it? Why not run it?  Many of them were from the UW because it's close by and had them, we basically gave them the browser and said, hey, somebody said, Visual Studio might be an interesting product for you. Why don't you go take a look at it? And we simply watched them, right? Now, what was interesting about it is at some point, through the installation process, all us folks who have no Visual Studio know it takes a fair amount of time to install. And it's gotten a lot better over the years, but it's still, the youth today kind of think about it as a five minute. Yeah, as any college student, how long it should take to install a tool, they will say five minutes. And well, what happened was they would start installing it, and after about three minutes, they started getting twitchy, expecting it to be done soon. And then at some point, they're on the internet looking at pictures of dogs and cats because they're just waiting for it to install. And they're almost like, they've sort of forgotten about Visual Studio at that point. And so that's like one explanation for that, but it was intriguing to watch that in the lab, watching people have that experience, and not only seeing their expectations, but actually seeing when they start getting bored because you could see it in their face, you could see it in their attitude, you can see it in their physical mannerisms as they're installing Visual Studio. So what do they do? Do they reach for their phone at that point? No, they're kind of stuck in the lab, right? They're waiting for this thing to install, and they get bored and they start going to the internet and looking at other things. And that led actually to some inspiration and said, can we keep their attention while they're installing Visual Studio? Can we give them some upfront insights into how to use Visual Studio while they're waiting to not only occupy their time, but almost more importantly, that when they first start Visual Studio, they kind of already know what to do. They already have a notion of what is a solution and a project and how to pick the first template and do sort of those basic things to do Hello World, for example, right? And what we discovered is if we can get them that almost sort of a visual tutorial that they can explore before that, they were far more successful when they first opened Visual Studio. So that was one of those deeper learnings that we had along the way. That makes sense. And that's kind of funny that I didn't know that came from there because it's something we talked about for well over a year, maybe two years, right? About having something make the installation process a little bit more interesting or engaging somehow. So that all came from that research you did back then, I guess. Yeah, it was sort of, there were several deeper learnings around that too. For example, people don't want to read a whole lot of text. We kind of know that, right? But when you see it, when you see people skimming, it sort of really brings home the point that you really need to be very crisp in your communication. And the other thing that was sort of real interesting is when you want to learn how to do something, a little animated GIF, right? That shows you where you need to go instead of reading like two, three paragraphs, you know, go to this menu and hit that drop down and da, da, da, da. A little animated GIF was something that people really gravitated towards because they could see it in action. You know, and it was kind of small, but it kind of gave them the general place to start. And that just proved to be very, very effective. You do need to think about how to do those animated GIFs because they can be pretty poorly done also. If it's too small, you can't really see very well. So you have to think about it a little bit, but very effective if you get it right. Okay, so we have a question here from the Q and A asking, if I turn off the telemetry, will I then show up as having installed the product but never run it? So in other words, are these numbers real or could they be a consequence of people disabling telemetry? That I don't know. You know, you should definitely talk to Kathy about this one because she did that initial body of research and did that instrumentation work. So you would need to ask her about that. I don't know for sure. All right. I should ask her. That's a really good question, actually. It is a great question, yeah. Okay, so we have a way of bringing people into a lab. We literally have a lab in one of the buildings. Is it building 16? It is, yes, it is building 16. And it's a full-on lab where people come in and sit down and you take them through tasks and how does that process work? You want me to show you some pictures? Yeah, let's see them. All right, all right. So I'll go show screen and then pop through pictures. And so I'll just kind of bring up PowerPoint. I'll show full screen here, share the full screen and then let me know when you all can see it. Is that okay? Can you see that okay? Here we go. Yep, we see it. Yeah, it's coming through. Okay, I'll show you the cleaned up pictures when they first built it, all the shiny ones, right? Before it actually got messy and where things got busy. So basically there's a big old garage door. We can open it up, make it very inviting. And there's sort of this shared space here and you see the labs around it. And the goal was to create almost a design space, a maker space. So we can surround ourselves with, our customers experiencing these ideas, these new products, these new features and to sort of start brainstorming about solutions to some of the problems we're seeing in a space that makes it very, very collaborative, I should say. And that's why you see these screens up on the side here. I'll show you a few more pictures where what's happening is you're actually seeing inside the lab, you've seen the customer engage with the product, trying to figure it out, trying to use it for the first time or using a new version of it that we've changed and tweaked in some meaningful way and we're trying to investigate if that's a useful change or not. The individual place, this is sort of looking at it from the team's perspective on the side. This is the place, the side that we work and I'll show you a couple of pictures of us in action. And what you'll see and the users on the other side of that mirror, it's a one-way mirror actually. And the purpose of that is that we want to not bias that customer in any sort of way. So we want them to go in there and have some tasks to do and for them to go at it and for us to watch to see what happens without sort of interfering or being very deliberate when we need to jump in. So this is sort of pictures from a typical Thursday. One of the things that we do in the lab is basically these teams, these individual teams that you just spoke about earlier develop a cadence. And so very often Thursdays are a really big day and it happens on other days of the week too. But what happens is on a Monday, we figure out what we want to bring into the lab in front of customers. And it could be, like I said, an idea or it could be something that is where we're tuning. And when we just pre-schedule these customers every week. So every Thursday, we know that every team, I'm gonna have customers in front of me. And I can go explore ideas. I can do, it could be just a standard sort of usability thing or it could be a mock up of some idea that we want to find out if it has legs, if it's a meaningful change for our customers, if it can fit into their workflow and work style in a meaningful way. So that every week gives us that cadence, that constant stream where always got users around us. Some other pictures around that. This is again, the team in this design space. We're absorbing what we're hearing and what we're seeing. And I think that's an important point here too in that we're not only listening to customers, we're watching how they're doing stuff and looking at the two together, right? What people see is really useful. Sometimes what people don't see is equally important, right? Especially when you talk about discoverability of things. And so what people don't notice, they can't tell us about, but we can see them not noticing it. And last week you talked a little bit about the eye tracker and stuff like that. So we even have the sort of when appropriate, be able to put an eye tracker to see what they're not looking at, to see what part of the UI they're missing, that we need to somehow bolster up in some meaningful way. So is that, am I, I do want to stop for a moment. Yeah, I have a follow up question here. Oh, you do, all right, yes. All right, so this is amazing. This is like, this is a proper lab, right? You do actually proper testing here. This is significant. And you bring, I think I saw a number like a while ago saying that you run like 10,000 people through customer research a year or something like that. Like that's the staggering amount of people. It's a staggering, but that 10,000 also reflects other things, not just the lab, right? So what we look at, as you know, we've had, if you think about how we did our job 10 years ago, 15 years ago, right? There were specs, right? To agile, to lean, such that we all are, lab is just one space, right? It's just one place where customer engagement happens. It's convenient for many folks because it creates a cadence, it creates a healthy form for having that conversation. But we have interviews, hopefully some of the audience have participated in this, where we will remotely share with you stuff and ask for your feedback. And so, when you think about that 10,000, it really encompasses all those, what we call zero distance touch points, which is still massive when you think about 10,000 direct customer engagement, right? That's a big number and it's an amazing number. And it was so amazing that Sacha came about a year and a half ago and was quite awestruck by the whole experience and seeing how much we do. No wonder, it's 10,000 is just a gigantic number, but the obvious question then is, how do you find 10,000 people that are assumed that are not, that are different, they can't be the same people every time or maybe that's okay? Like how do you go about that? That is great. So one of the things that, and that's a great segue to when we talk about the lab is well, the world is not Redmond and Seattle. And so the lab is where people physically come in. But again, we reach out remotely and we have a number of different ways of connecting with customers. So for example, we have an organization that will recruit for us. They will reach out to folks and ask them to participate. We'll give them a profile and they will seek out, contact those folks and ask them if they can participate for a 45 minute hour. Lab sessions typically are like an hour and a half because we will do multiple different things. For instance, if we are testing APIs, and we can test APIs for usability, for example, and do it in an iterative sort of way. And so coding takes a little bit, so they're in for an hour and a half. But, and then there are services out there that actually allow us to do these little mini usability studies and little studies that are like 15 minutes long. And they will then, we create a screener and then it goes out to the world and people sign up to participate and do these little mini studies. So there's all kinds of different venues by which we do that. And then of course conferences, right? And in fact, let me show you a build from one of the conferences. Yeah, okay, build unfortunately is remote this year. Well, it's gonna be interesting, right? But anybody who's attended build will recognize this because at build, we sort of turned around the story where normally, typically it's about downloading stuff to our customers. Here it's reversing it. It's about uploading for our customers telling us, telling us their story, telling us about issues and us taking that back to the lab. So again, there is sort of many, many different venues to reach that 10,000 number. And, you know, last year we had this wall of people submitting sort of feedback to us, direct feedback in a very, very structured way. And even in some cases, adding pictures and that wall just took over. I mean, we had six, just there and that one wall, we had 660 people having given us very target information about the product, what they like, what they don't like, what they see as an improvement to it. I remember that. I thought that was such a powerful thing.  And so that was a handwritten, I think their handwriting, maybe even. And just going through and you categorize them. Some of it was about the setup experience, some was about editor experience or something like that. They were like categorized in different things. So we could, as PMs, we could go down or anyone, engineers might not, we could see and we could follow and we could learn firsthand from a direct feeling that users have engaging with the product or a frustration or something they're really happy about because I think oftentimes it's easy to just focus on things that are problems that we have to fix in the product, but many times we also have to identify, well, what are the things that work really well? Let's make sure we don't screw those up, right? That was one thing that that stood out. That was one thing that I thought that wasn't the most hilarious thing ever. That was a guy, I think, he was very unhappy, very unhappy that we changed the logo. Do you know what I'm talking about? Oh, I still have it. Can you talk about what he did? So in years past, we would have these sort of, feedback walls and people would scribble on, a sticky and maybe put their picture on it and it was sort of loosely formed and it proved to be very hard for us to get it back to the teams, right? Who does this belong to? If somebody says, oh, performance not so great. Well, who do you bring that to, right? Because very often performance is specific to an activity. It's specific to something. And what happened there was we put some structure on it and we have something called the hypothesis framework where everybody builds these hypotheses and we have sort of a mad lip style fill-in-the-blank kind of moment where we said, as this type of developer who does this kind of work, my problem is this because, and when you fill in that stuff, it gives us just a much richer venue of information for us to then get that back to the teams, to the proper people in the teams to really understand the source of that anxiety or that design idea. What happened there is here, his thing is as a long-term visual studio user who's a tattoo lover, you have unchanged the icon and he had the original logo on a visual studio on his arm and in the picture he shows the original logo. And so jokingly, he was sort of describing his pain point of us changing the visual studio logo and this imprinted tattoo on his arm is no longer current, so it was so funny. So we, so awesome. When we talk about like, hey, this has consequences if we build one feature over another or we don't fix that bug because we instead used the time to fix a bug somewhere else. Like it has consequences, but never like a consequence like that. That was just from left field. Yeah, that was, yeah, yeah. But the other thing I think it's important here to talk about is that, and especially in these times now, it's like how important a product like visual studio is a product like visual studio is to our customers, right? And how important it is for us to be very, very thoughtful and what we change, what we add into visual studio, how we think about it, right? And so clearly, it's something when you have to live in it, four, five, eight, 10 hours a day, it becomes something that's very personal to you. And that's important to consider also. And this is why these sort of venues are so important. And that's why these engagements are so important to us because we can't understand all the different contexts. There's so many different sort of perspectives that people bring into their daily lives and to create a tool that is able to be able to have affordances for those different perspectives and those different skills and those different contexts is quite challenging, so. Right, and it is something we take very seriously but I think like what you were saying, just to echo that, like now with everything going on is even more important, right? That we take this very seriously because this is now used for like very serious business around the world, right? To stop these things going on. And so it kind of adds a little bit of pressure, I'd say. That's a little, yeah. Hey, but that's good, right? It's a good pressure and we feel good about like it's being used for good things. And so there's been a really good sense, I think on the team, like team all up, Julianne, Manda and all the way down, that we're actually, we're making a difference in all the people that use Visual Studio and of course other programmers using other tools as well, like it's actually making a real difference, keeping the world running right now and coming up with algorithms and stuff that can sequence DNAs and come up with cures and whatnot. So that feels very empowering when you think about that. We as an industry, as programmers, as developers and so on that even though it doesn't seem like we're doing much, right? We're sitting at home. We're not healthcare workers or anything like that. No, but we make sure their email works that they can communicate that they, all this sort of stuff. Right, exactly, exactly it. Yep. So Carl, we have another question here. This is a good one. Oh, okay. So you mentioned that we bring new beginners into Test Visual Studio, right? We have them in to run, you know, setup and how we can improve setup by studying a new beginner Visual Studio users. But what about like power users or experts? Like, do we do any studies on them to make sure that any problems they face are also taken good care of? Oh, no, no, no. So again, you know, we're very hypothesis driven. In that particular case, we were investigating a specific problem. So in the quant data, right? We found out that these students were and so that led to that investigation. But in our lab and particularly with the IDE, we have a very broad spectrum of customers coming in. And in fact, I would say, we even had this sort of great experience where we had a developer who was blind come in and use the debugger. And it was an amazing experience because the entire dev team showed up for that to understand the accessibility of the debugger in general. And just watching that customer struggle with some things that were just kind of things that we just didn't think about or just kind of overlooked or didn't prioritize correctly. It was a remarkable experience for them and a remarkable experience for the product because I think that inspired that entire team to focus in on those issues, learned a lot and really turned around the accessibility of that product. So I have many folks, you know, 20 year veterans who will come in. I remember VB6, you know, sort of thing, right? And that actually speaks to, you know, what is really kind of a challenge here is that for any product that you use so much, you develop habits, you develop, you know, you think about just the sort of the editor, for example, you know, and how an editor works. It's a big deal. And you're in a path where, you know, you do need to make progress, you do need to improve on things, but you need to also be very cognizant of those patterns of behavior. So it becomes the challenge is trying to understand how you can augment, how can you work within that sort of learned behavior and be able to be able to sort of provide additional value in a meaningful sort of way.  And, you know, so it really, we really sort of invite as many different folks in and we love the diversity, the diversity of people using Visual Studio week to week. You know, that person who comes in, who comes at it from a very different perspective, who, you know, offers us that perspective, gives us yet another sort of way of thinking about how we should approach the problems. Yeah, and that's a really interesting thing with habits, right? Because we all do things differently. And one way that I see that very clearly with our users is we have this way of providing feedback. People can send feedback from within Visual Studio, whether it's a bug report or a suggestion ticket, right? And well, it's really, really important for one person for us to fix. Like this is an extremely important thing for them. It's the core of their workflow. But they are, you know, unique in the sense, well, there's not really any unique Visual Studio users. We are too many. Like 8 million or whatever. So usually when we see that people open a bug, saying, hey, this thing doesn't work, we think about it as being 1,000 people. Like so, one person equals 1,000 people. And that is just a number we made up. So, but the point is that there's a lot of people that have the same problem, but it could still be relatively small compared to the 8 million Visual Studio users. And so it might be something super important to one person, but not to the others. But that doesn't mean that that thing isn't important to that person because they're doing something niche or their workflow or their habits, have them use the product in a certain way. And one way I think where the UX lab and your team really comes in and shed lights on this sort of stuff is, here's a very concrete example. How do you build your solution? Well, there's probably three ways that I can think of. There might be more. I'm sure there's more. But you can either use a keyboard shortcut. A lot of people do that, that's very common. But when you take people, even people that have used Visual Studio for years through the lab, you learn that a lot of people actually go up to the build top level menu and click the build menu item itself. Or they right click their solution and say build solution from there. And so all of a sudden you learn that for the same task, there are multiple ways of doing it. And it's not like one is more correct than the others, but you might have optimized for a certain flow you thought, this is how people always do it. You never occur to you that, hey, people do it differently. And so then you end up with all these bugs and you don't know how to prioritize it necessarily. And we struggle a little bit with that. But that's where the lab really comes in, right? Because you can very clearly see that our assumptions of how people use the product probably aren't the right ones. Right, right. And then it becomes interesting. I'm gonna circle a little bit back to the student profile is that, so we sort of view running and debugging as the same thing, right? It's sort of, we've conflated the two together and that's been a long-term sort of visual studio thing. Well, when students come in and they write some code, write their Hello World code, they look for a run command. And it's not a run command. There is play and there's that little green arrow that looks familiar, but it's about debugging that doesn't quite make sense. I don't want to debug, I just want to run my code. And you will watch them look for a run command anywhere in the product. And it's sort of, again, going back to what we just talked about, it's sort of so many different people coming at it from different perspectives and trying to weave that through all that is fascinating and challenging and kind of, it's really hard, but it's hard in a good sort of way. Yeah, so this is a good segue call because now we've been talking about bringing people into the lab. This is something very, very few teams or companies can do because it's not a cheap thing. We have a whole team dedicated to running these labs. And so it's only sort of the big tech companies probably that can do that. But we do other things for this sort of qualitative information gathering. We do customer interviews where we Skype them or Microsoft Teams or call them on the phone. And we have that sort of down to a science too, right? It's like there's a certain recipe of how to do it right. There's a lot of stuff we've learned over the years. How do we, how do you conduct a good interview and how do you do a bad one? Okay, yeah. Can you talk about like a little bit? How do you even get started with doing these interviews and how do you make sure that your own biases doesn't predict the outcome before you even start? Stuff like that, some common pitfalls? Yeah, so I can, there's multiple parts to your question here. Yes. So one is like, what if I'm not a big company? Right, I'm part of a small team, right? And I don't have these resources. So that's kind of an interesting question too, right? And then is the, how do I do this successfully? How do I remove my own biases from this? And then there is sort of that, okay, there is sort of this rich space of things you can do to engage with customers. There's all these opportunities there. And they're each special in your own way and how you approach it. So this really, your question is really pretty open. Do you want to pick one of them over the other? Do you want to start with one or? Yeah, let's start with like, if you're not a big tech company, how can you do your own customer research? So one of the things that, in fact, even happened in the office when we were in the office, that I had a team who, we really didn't have resources for them. We really didn't have the ability to resource them. And so, and they were dealing with some docking windows issues. And they were trying to understand, if this is going to be intuitive and all that stuff, right? And so instead of sort of going into this large structured sort of approach, I said, follow me. And we walked to another building, another team room, and we walked into one of these team rooms and looked around for a dev.  Can we chat with you for about five minutes? And we brought our laptop with us. We had a private build of that feature and we gave them a task to do. Hey, could you do us a favor and try docking these windows? And we just watched this person try to dock these windows. And we saw, and we would say, try docking it here, try docking it there, try tap, you know, try different. And suddenly we said, ooh, that did, that worked great, that, not so much so. And we were able to take that very quickly back. And we only had to do that like two, three times.  Okay. Okay. That makes sense. So was it important then that the person that you interviewed there, I guess at that time was not on the team to build that docking feature? It was someone that had no idea, no prior bias either way. So if you were to do this in your own company or wherever you might work, let's say that your company, you have a, you sell shoes online. And so you have a bunch of developers, if you go to them and ask them, they might know the problem if it's a small team, but you can go to someone in accounting that would be a potential customer, right? Go online, but that doesn't know anything about the technical aspects that you're about to ask them about. Would that be, would that be how you do it? Absolutely.  So is somebody in accounting going to be fundamentally different than somebody in delivery around shoes? Well, maybe, right? Maybe not. And so, you know, watching somebody try to buy shoes online, if that's the sort of the goal and you're working with a new idea and seeing how they use it, it's that simple. Now the hard part here is for all of us, we talked about bias, right? We talk about bias, you know, internal versus external customer, so on and so on. There's also our own biases. And when you have a feature and somebody struggles with that feature, your first inclination is to do what? Help, right? We're all trained to help, right? We are in our industry, we're built to help, right? When we see somebody struggling, we wanna fix that problem. In this case, you have to sort of back away from that and have to let that problem happen. And in your brain, you have to go, if I can understand this problem well, okay? If I can let this problem happen so I can understand it well, then I have half a shot at figuring out how to solve it for a large population of people. That's the hard part. The hard part is like watching somebody struggle and be in pain and be confused and not wanting to jump in, rather taking the approach of letting it happen as it would. And then afterwards, you can help that person. That's perfectly okay. But it's a focus on learning at that moment. You're learning, learning, learning, learning. You're not helping, you're learning. And everything you do is about trying to mitigate that biases to maximize that learning. Does that make sense? Absolutely.  What are some problem hypothesis and what are some solution hypothesis? And we start calling people like, over 50 people we called like an hour. We had an hour with the Meach to go through a bunch of things. And this was all done on Microsoft Teams. And I remember like a lot of the things that we thought was gonna be validated. That's a slam dunk. Our assumption was that everybody here has that particular problem or I think that this is a good solution for previously established problem. And it turned out that that was wrong. And oftentimes when those people struggle with finding the solution or identifying the problem or something like that, you know, we were like, my initial reaction was I wanted to help them get there somehow. I wanted to, because they would typically say, oh, you have a problem with such, there's a problem with Visual Studio when I do such and such. And that's not why you're doing the interview, but I kind of wanted to go in and say, well, have you tried doing this and this instead? Like completely irrelevant to what we were doing, right? And then derailing the conversation and that sort of stuff. But I just find it so fascinating that we can be so wrong all the time. And then like, are we though so wrong?  And, you know, we were in a model where the most knowledgeable person was the person who we would assume to be right. If you are in a frame where you're really brainstorming, you're really engaged, you're really sort of looking for opportunities in many different places, you better be wrong. Most of the time, right? You can't be right. There's no way you can be right. The world is very, very complicated. And the real trick here is the faster you can figure out you were wrong, right? And the faster you could learn from that mistake, the better you're gonna get at coming up with a solution that the broadest set of people will have the maximum benefit from. Yeah, you wanna be able to figure these things out so you can course correct early on before you get too deep into the development cycles, right? And so you have to back out and re-architect and all that. And so a great thing to do at sort of a physical level is what you'll see in the lab is we'll be bringing these low fidelity mockups. Where what we do is we show you how that experience could feel like if you were to use it. Okay? And in some cases, it's a simple sort of what we call user step system response. Sort of model where you do this, you see this, you do this and this. Then you can do interesting things like, what would you do next? Which is almost like this mini sort of usability thing. Well, they know what to do next, if we show them the screen. And it's some cases really easy and simple to create this sort of mockup. It's literally taking a screenshot, slapping some MS Paint UI on top of it enough to give the customer a sense of what it could feel like, okay? And sometimes very often that is sufficient to find out that, well, we were smoking something that day because, you know, and rip it up and start all over again. But when we start all over again, we go, wow, that didn't work, right? Perf tips that I worked on several years ago with a team is a great example of that. The first idea that we had, this is something that came internally. Some higher up people had this notion for this window that would give performance information and it would sort of look like the windows sort of thing and it shows CPU and memory and stuff. Like a test monitor. Yeah, kind of like that. And then as you step through it, this thing would sort of, you know, change. And what we did is really simple is we took screenshot, we did sort of a really cheap variation on top of that. We showed you stepping through code and seeing that window. And without explaining anything to them, we asked them, what do you think that thing is?  And then we asked it sort of that, there's always a great question to ask, what would you do with that thing? Oh, I'd probably close it, which is kind of like the kiss of death, right? If once it's closed, it's not coming back. And so, because it was a low fidelity mockup, we did this two weeks in a row, couldn't get it to work. We made some changes, see if we could, it failed over and over and over again. Nobody got it. Nobody could articulate how they would use it in the real world. And so we basically ripped it up and started it over again. And that led to the perf tip story that we have today, which the goal was to democratize performance. And so that's the feature you see today. Nice. Are there any other sort of surprises that you have just observed over the years where we were sure about a certain outcome that then turned out to be different? Are there any fun stories from the trenches? Oh God, there's so many. There's big ones and small ones. And where do I start? Console app, that's an interesting one that, you know how the console app works, if you write your little hello world story. So like a .NET console app for instance? Exactly, something simple like that, right? So I'm gonna circle back to the student story because it's just, it's top of mind right now for me. So student, and these are CS, these are computer science students, right? From the University of Washington and some neighboring schools, so really smart people. And a lot of them were in their junior levels, already had gone through a full suite of coding classes. And they would be writing their hello world app just to get a flavor for it. And that was interesting too, because the question that we had is, hello world still relevant for new developers, for the people coming in today? Turns out, yeah, it is, but you think about it, it's a concept from the 1960s. It's from Kernagin and Richie C programming way back in the 1960s when that notion was introduced, but it seems to be still relevant today. And so they're writing this code and they write this hello world and then they run it. And then the screen will flash and go away. And they're like, what's going on? Now, this is a concept that's been in Visual Studio forever, right? The fact that you have to stop it, you have to put an input or a breakpoint, something in there in order to break that code from completing its execution. And it's been something that we've just assumed all along. And the PM who watched this from week after week watched these students hitting on the same problem. And they were left in the dark. It would disappear like, what happened? I tried to run it and nothing happened. And it was only in the situation where things are running really, really slow that you kinda saw something appear that looked like, there was something in there that could be your code. And so he took it upon himself to make that change in order to not have that window stick around. And so what was really fun is that he got an email after they got it out the door from a very senior level developer who just loved this new feature because he could take that console up and dock it, he put it onto his other screen and it stayed there. And it was able to do the work that he needed to do. And he loved it and for him it was a new feature, right? So that was a pretty interesting pivot on something that we just all had assumed after years and years and years of this is how it works, right? Oh, are you there? Sorry, I mute it because I have painters outside and they have this machine that pumps paint. So I was muting my phone, sorry. So let's see here. So what's really cool about what Carl just said, there was a test in the lab that was happening about students and how starting up Visual Studio for the first time was being perceived by these students. But what's really cool is that the people that were sitting there in that room observing, the product team sitting in the room observing all these students, they started seeing patterns that they weren't there for. It was just something emerged and they noticed something and they could brainstorm it right then and there and come to basically fix some problems that they didn't even know existed till they had those students in there for completely different reasons. And I guess that's a quite frequent thing, I would assume, right, Carl? Yeah, I'm back by the way, so I got kicked. No one noticed but me. Okay, well done. So I had that moment of panic is like, did you even hear what I said? It's just gonna stop, so could you repeat the, just frame the question for me one more time? I'm sorry. Yeah, sorry, Carl. It's that we had the findings that they had figuring out the console lab was shutting down if they didn't have a console read-line in there. Like they figured that out by just observing people in the lab, they weren't there to test out that scenario. It just happened to be a pattern that emerged and then they were able to fix something that didn't even know needed fixing. And so is that something that happens frequently? Oh yeah, and the thing is when you watch somebody, in this case, Augustine was the person who just, it drove him, watching that week to week just drove him nuts.  And it gave him that passion and that ammunition to go, we have to do something about this, right? And it happens all the time because, when you watch somebody struggle with something, we talked earlier about watching a blind developer struggle with your code, with your app and for things that you've missed, things that you've thought about in sort of not quite the right way. You didn't really understand the problem well enough. That's a source of energy. That's a source of inspiration. That's where you're willing to sort of step it up, right? And that just happens all the time because well, it's like we all wanna help. We all wanna fix things, right? It's intrinsic to our industry. It's intrinsic to who we are as tech people. And so watching and seeing and hearing this stuff is just powerful. Can't shake it. No, we're doing that in time, so. We're good, we got four minutes left. And so there's one thing we haven't talked about yet, Carl. And unfortunately, we're not gonna have much time to do, but we went through bringing people into a UX lab, which very few people can do. We talked about, hey, you can do interviews with customers like over teams or Skype or whatever is your preferred method over the phone. You can sort of interview or have like run-throughs of your features with people that are in your vicinity, in your office, you know? And we do another thing, which is we do a lot of surveys as well. And I'm sure that everyone watching this, you've seen us ask you questions, like whether it's inside Visual Studio or it's on the Visual Studio website somewhere. Even the blog asks you for your feedback. And so, you know, a common fear, of course, is that we ask too much, too many times, we give you these things. But there's a reason, and there's a method to the madness, right? How do these surveys work and why are they, why is it important that people answer them if they have the time and the capacity? So, so also, great question because there's multiple aspects to it. So, when we learn stuff in the small, like, you know, we're in the lab and we bring in, you know, five, six, 10, 15 people and we start learning what's the right way of asking the question, what is really the core problem? Then the next thing is, you know, how pervasive is this? Is everybody suffering from this or is this a meh? Sort of, you know, is there different types of people that are more affected than others, right? In some cases, it's a map for other people, it's like the world is coming to an end maybe moment. And so, the survey helps us get that breadth perspective. In some cases, we're trying to also, depending on what the hypothesis is, what we're trying to learn, it could be, you know, we're trying to say, hey, are we missing the boat here in some fundamental way and getting that input from customers? And who are these people using this in the first place? You know, we have limited visibility, we have instrumentation and stuff like that, but who you are as a person and what you're trying to accomplish is something that we're trying to make more visible and these surveys give us that window into that. And so, you know, we feel always like, are we asking too many surveys and we have some controls in place, as you know, to sort of mitigate that. And so, but yeah, it's a very real sort of issue is how do you reach out to all these people to get this feedback and you don't want to overburden people?   And so, this is just another form of collaboration. Yeah, and oftentimes we use the survey because we think there might be something that we're missing the boat on, as you were saying. And if it turns out to be true, that could lead to us bringing people in specifically to test out some of these problem areas in the lab or do some customer interviews and really get qualitative data on what is the problem here so we can get to the bottom of it. But the survey can help us to figure out if there is a there there to begin with. Exactly. It really goes both ways, right? We have sort of a, I don't have it with me, a kind of graph that shows the, you know, we talk about qualitative and quantitative. Sometimes the quantitative shows an issue that is like, what's going on here that then the qualitative helps you bring light to. Sometimes you have a qualitative observation and going, we're not really collecting quantitative information on that problem. We didn't know that we should be doing that. And then it goes back to figure out how we can get that information at that quant level. And so there is sort of this back and forth that's always happening. And that's where the interesting, that tension and that back and forth is where the learning kind of really happens. No. Awesome. Carl, we are at the end here. Thank you so much. This was super fun to learn about and see those pictures from the lab was just amazing. It's really cool stuff that's happening there. So thank you so much. You're very welcome. Thank you. That was, it was fun. Awesome. All right. All right. So before we leave, if you watch this on YouTube, remember to hit the subscribe button below. We really appreciate your viewership here and feel free to share the video. Next week is build. So we won't be doing this office hour because of build. There'll be so much more content coming to channel nine and YouTube or wherever they're gonna put it that you're gonna be so busy that you won't even have time for this anyway. So we're gonna be starting up after build. So the week after, hopefully. So thank you so much for tuning in and see you next time.


Hello is first seen giving the review of a typical mizungu where she appears to be shocked and disappointed by the test of the popular Kenyan margarine but a few moments later pal switches things up real quick with aluya accent in Swahili she goes on to blast wazungus for her looking the benefits of blue band such as big bombs and calves all right mizungu most wazungu hawa nanga mataku mataku yaw ina kongatu flat sabapu yaiyo akulangi blue band yaw wajutam yaw blue band blue band ina fanyanga mutu ananona in all the right places unaniyelewa all the right places yani heivi mbabama dumutu unona vissuri mizungu hajwangi kukula blue band yaiya nachua kukula keiki atau kisha maliza kukula kukula una letewa tisad tisad atini keiki kukis situi ni nanin fitutamutam ananona zibaya naga zibaya akilona lakini mukenya kwanza muluya akisha maliza kukula una letewa nini una letewa jai na mukati na blue band yaw njotisad yama afrika mukati na blue band na jai yaw njotisad in all the right places alafutu kona nguvu tunanona nandiyo lakini yaw kuna na yaw inu kona nguvu guys do you agree with her umzungu amesema by the way she is actually American she is a bit older there is a video over yaw umzungu mayanana gaya kwa kili alisa umzungu yaw umzungu yaw umzungu kama blue band yaw nga panika kwa mwa yaiya ni umluya mika meka tini We'll let the rest of the world know What do you use? Blue band Find girls Find girls Assets MCK Arrest my case Thank you MCK ame MCK ame zuba Because MCK ame zuba You're speechless I'm speechless Of course I'm a nano I don't understand the whole blue band thing Because we don't know blue band and we don't know what it is We don't know what it means We shouldn't know We should know How? Because we don't know gold and silver We don't know blue band We don't know what it means I don't know how to discriminate Those who don't know We don't know it Hi We don't know what it means I don't know what it means I don't know anything I don't know what it means I'm speechless I'm speechless I've been arrested No, I know We're back in comparison. Baking the next stop. Kenyan socialite, whereas Sri feet has packed head again this time for posting...


Abercrombie and Fitch Batch 46 fragrance review. Hey, what's up guys? It's Josh here. Today I want to do a little review on this Abercrombie and Fitch Batch 46. Now, I think I got this fragrance like probably in my first five Abercrombie purchases. I don't know if you guys have been following my channel, but I really like Abercrombie fragrances. They used to be so much better back in the day. When I first started this channel got into fear so I just started buying their whole line. Abercrombie at the time would come out with the new fragrance every couple months so I ended up having tons of fragrances. I would do reviews on all the fragrances and because the fragrances were just kind of so so and on top of that they weren't as popular as other designers. I just really wasn't getting the views and people didn't really care about them so there was a point a few years ago where I got rid of all the fragrances that I didn't really like that much and only kept the absolute best. Batch 46 was one of those that I actually kept. I really wish they still offered this one. It is probably one of the most masculine fragrances that I've ever smelled from ANF. The notes that they list on Fragrantica are Musk, Black Amber, and Cedar.  I do think that it is somewhat similar to Abercrombie and Fitch Woods. There was a lot of hearsay that this was a copy of Rule 925 signature fragrance and if you guys do not know about that around the late 2000s they decided to open up Rule 925 which is basically like Abercrombie and Fitch but much more expensive. There was a lot of hearsay that Rule 925 was kind of like a retouched up version of Abercrombie and Fitch Woods and I think they had something called R7 and R7 was supposed to be similar to Ezra Fitch which makes sense because I really really like Abercrombie and Fitch Woods. I really really like Ezra Fitch. For Abercrombie those were the most mature fragrances that they have. It kind of makes sense that they took away Woods and they took away Ezra Fitch and then retuned them up and basically sold them to Rule 925. Now I have not smelled the Rule 925. It is something that I've been very very interested in because I really like Abercrombie and Fitch fragrances. I've bought discontinued Abercrombie and Fitch fragrances and Rule 925 is essentially like Abercrombie and Fitch but way way better and then they have their signature fragrance so it's like man I already like Abercrombie. This is a more expensive version of Abercrombie and I like signature scents like scents that are right down the middle so I would absolutely love getting my hands on Rule 925 but I would think that it's going to be very very similar to this fragrance. I don't want to spend a few hundred dollars on it just to get the comparison but I could totally see this being Rule 925 signature scent. Some situations this is good for. I would say this is okay for the gym. It is a little bit too strong in my opinion. It's got a little bit of a musky barbershop style vibe so just for the gym it's probably not the best although it's okay. The date I would say this is okay for like a daytime date fragrance. That's about as far as it's gonna go though. I think this is a good fragrance for work not the best thing ever but I think it's overall pretty good. This is just an everyday hot weather style fragrance right down the middle. I can't really say too much things about. It doesn't really excel in any one specific area but if you just had to toss this on and head out the door I feel like it hits a lot of situations. Alright guys let's jump into the pros of this fragrance. The pros is that it is very unique. I've smelled a lot of barbershop fragrances. I've smelled a lot of A&F style fragrances and this fragrance is pretty unique. It's kind of hard to put your finger on. It has like a little bit of a tiny bit of sweetness in the background but overall you would say this is 100% not a sweet fragrance. It has like a barbershop style vibe. It's hard to describe but it's very full and masculine and kind of borders being like too strong and in your face with being somewhat likable in its own way and then I also enjoy how masculine it is. In my opinion this is about 75 to 80% A&F woods. Woods and Ezra Fitch were probably the most masculine typical gentleman style fragrances that Abercrombie had so I just really like how masculine this is. It's even more masculine than a lot of their other offerings. And then another pro is that it has a very very very good projection.  Like you could probably cut the projection of this by 10 or 20% and most people would still think it has good enough projection. If I were to go over the cons you know I really can't think of too many. It did kind of make me a little bit sad that a few years ago I was watching I think it's the guy who does stay fresh productions and he was doing like a top five fragrances that I got rid of and he ended up getting rid of Batch 46 which is pretty sad because I would think that most fragrance reviewers if they had a chance to get this at a reasonable price I think a lot of people would appreciate this. If I were to give some reasons why I think someone would get rid of this it isn't the most likable fragrance that I've ever smelled. There is a little bit of a rough in your face not as likable quality as it could be but I really like this fragrance for what it does. I like how masculine it is. I just pretty much like it all but if I had to put a con it's probably not the most likable but I really can't think of anything else besides it's probably not going to be good for winter either. I just don't really have too too many bad things to say about this one. I actually really wish they kept it in the line maybe rebranded it because for being such a legendary strong fragrance like I think that this is it's weird that it has such a simple presentation and no real character to it but if I were to give it a smell rating I'd give it a solid 8.5 mainly because I really haven't smelled this exact DNA in too many other places I'm going to give it 8.5. Likeability I'm going to give this a 7.5 it's not really not likable but at the same time it's not the most likable thing it's not something that you just put on and everyone's going to love it's not something you just put on and you know for a date things like that. I just think it's right down the middle of 7.5 and then longevity of projection I'm going to give this an 8 to an 8.5 this has a very very very good projection overall I think batch 46 is a good fragrance I think you can find on ebay for about a hundred dollars again it's supposed to copy the rule 925 signature apparently but from everything that I've looked up about rule 925 the way they describe it it's almost like they're describing this fragrance and this fragrance has enough in its DNA that I could see somebody making this like a really expensive fragrance and a really like a signature fragrance for the store it has that style vibe but anyways guys let me know if you tried batch 46 let me know if you tried rule 925 signature let me know about all that down below I love diving into the classic rule and Abercrombie lure because it was so so good back in the day but anyways guys we're on the road of 50,000 subscribers and I could not do without any of you guys so hope you guys are the best having a great day out here hopefully having a great day at home see you all in the next video peace I've smelled a lot of barbershop fragrances I've smelled a lot of a and f style fragrances and this fragrance is pretty unique


There's little doubt that the ancient civilizations that existed on earth were in fact trying to send a message through the ages. We have came as far as to understand that the geese of pyramids and the sphinx are in fact an astronomical clock that keeps track of a 26,000 year cycle. This is the length of time until the reset button, or perhaps it is something more profound. Whatever it is, the people of the earth today are safe from this occurrence, which is thought to be around 16,000 years away, but it should be up to all of us alive today to begin to share this knowledge in order to understand what the ancients were trying to reach through the ages to tell us. The word epic is an understatement when you consider and describe the sheer effort that went into building these places all across the earth. Wait till you hear this. All around the world there are places of ancient and unknown origins that appear to be connected somehow. We don't understand why this is, but this is so precise that it is impossible for it to have been an accidental phenomenon. Think about that. Cities on every corner of the earth began constructing megastructures that all aligned perfectly through thousands of miles. It's crazy. To us, it is even more crazy to suggest that these civilizations did not know of each other. Perhaps it was a united race of beings unlike our territorial individual societies of today. From Egypt to Iraq and even into China and South America, something was going on tens of thousands of years ago to alert future civilizations of something of massive importance. What code are we yet to encounter that tells us you would have to consider that these earthlings would have taken into account that only another intelligent civilization would be alerted to this eventuality. You would also have to consider that no matter how advanced this civilization may have been that they would have to use a code we would understand, much like our use of binary today. Do you understand what we are getting at here guys? The connections are there, everywhere. We need to consider how was this possible and how did we forget this knowledge of ancient geometry in particular. We know it's crazy to think that there was an advanced race of beings here before us, but there were. They were everywhere and perhaps we don't like the fact that they were more advanced than that of us today. Maybe they left the earth before a cataclysm event and raised these structures for future humans to follow. The alignment with Orion could in fact be an open invitation, who knows. In addition to the astronomical clock at Giza, which is the single most accurately aligned structure on earth by the way, and at the very center of all earth's land mass, sites like Easter Island, the city of earth, Nazca lines, etc. There is a big list, are all perfectly aligned clockwise around the globe according to the Lambert-Asmothal Equal Area Projection. Just why is this? The signs are literally everywhere and the false dating of these places of 5,000 years old is an absolute shocker in terms of what their true age actually is. The Giza complex does seem to be the central point and the most important part of the grand signal from the before time. We don't know who built these structures, and credibly it is mostly made up history and we take it as fact. We don't know the true age of the pyramids or the Sphinx at Giza. It has been suggested that 100,000 years old could even be an understatement and that Giza was built before the area was actually underwater. There used to be a tide mark halfway up the pyramids and there is even evidence there were seashells found along the pyramid stone isn't that amazing. It is thought that they were built during the last age of Leo but the evidence in that in the weathering shows them to be far older, the astronomical cycle of the Zodiac constellations could have these places dated to at least 62,000 years old. This is according to renowned researcher and lecturer John Anthony West who points out how long it would take a civilization to reach these heights of advanced knowledge.  The knowledge they have attempted to pass on to future earthlings was realized perhaps over 100,000 years ago. This is a fact you know. So just why were these places built in perfect alignment across the world? Is it all part of a gigantic warning clock? Or is it something far more important than we would ever have attempted to imagine? Only when we realize what it is we have here can the work begin. We have speculated to the point of story invention for long enough it's time for truth. Wouldn't you agree? Comments below. Thank you for watching.


A systematic review of worldwide full economic evaluations of pertussis vaccination targeting adolescents or adults published from 2000 found that booster vaccination for adolescents and adults were the most frequent, followed by cocooning and pregnant women vaccination. The performance of strategies varied considerably among different studies, with assumptions regarding underreporting correction, hurt protection, and vaccine coverage being crucial to cost-effectiveness results. Understanding the model and parameters used is essential to understand the results and identify major issues important to public health decisions. This article was authored by Ede Gatti Fernandez, Camila Cristina Martínez Rodriguez, Ana Mali Cristofam Sartori and others.