I want to introduce Brille Nikolov, who has been our course instructor for conversation design. And we're hoping that Greg Bennett is also going to be able to join us, but we haven't seen him just yet. And so let's start with Brille and not, and then if Greg manages to get in, we'll be happy to introduce him and include him too. So Brille, talk to us about your career in linguistics and beyond. Okay. And I just pinged Greg on Twitter, so we'll see if he gets in. Right. I think you mentioned he's traveling, but we'll see. My career, so in a nutshell, I'll start from the very beginning. I studied linguistics in college, along with pre-med and neuroscience. I came from a very STEM forward family, so it was always kind of assumed that the better quote-unquote better paths in life were lawyer, doctor, engineer. So I was kind of like, doctor seems fine, like, I'll go with that. But really my passion was always learning about language and linguistics, and so I started out with a minor in linguistics in college. By the time I was a senior, I had taken so many classes that I had just racked up all the credits and it had just turned into a major automatically. That's when I realized maybe I should start listening to myself a little bit more about what I love to do. I think my problem in my school was that, you know, the department wasn't really putting on any events like this, not even close. There wasn't really much conversation at all about what kinds of careers you could do beyond academia. So with that, I was graduating and kind of thinking, you know, now's the time I need to schedule my MCAT and get that ball rolling and I just found myself not doing it. Days went on. So I kind of just had to like really come to terms with what am I actually doing here and, you know, around that time Alexa, what, let me mute. Oh, she's already made it perfect. She was becoming very, very popular. That was around the time that Amazon launched Alexa and obviously Google Assistant followed shortly after. So I completely switched gears. I had a friend in design, graphic design. She had studied art in school and I was always like, that's cool but A, my parents would never approve of me doing something like that and B, what are you going to do after school? Like, how are you going to make a living? But she was doing her own thing and I was like, okay, fine. Turns out she was always the one with the internships that were paying her so much money in college and then she had a six figure salary when she graduated and I was like, teach me your ways. Like, what are you doing? How are you doing this? And she was like, well there's something called design that, you know, you can, like you could get into this too. Design for all kinds of stuff. Not just graphical interfaces, but there's now voice interfaces on Alexa and chatbot interfaces. So I was like, well, you know, I don't have a super artistic eye but I really like kind of this idea of design thinking and how designers work together and think and it, I had always been very, very, I just wanted to make things and produce stuff and create. So that's really like what design is at its core. So I just really like dove in and researched and tried to do my best to figure out what this field of Vui design is. And I'll get like weirdly specific just because I don't think like it can be hard to kind of visualize someone's actual career path and like every point at like every juncture. So, to be perfectly honest, I was like, I was in a great position when I graduated because I was a live in nanny so I didn't have to worry about making rent, or moving back home. So that put me in a spot where I was working 20 hours a week as a nanny and then had kind of 20 hours to really research apply for jobs and start like connecting with people on LinkedIn and that's kind of really like that was really the key for me is, you know, reaching out to linguists in bigger companies asking, hey what do you do all day like what is your what was your career path. I was back then, I was like four years ago but really no one was calling themselves a conversation designer just yet. And so it was really a matter of like first finding people in these companies that sort of identified as linguists and then asking them more about who they work with. And that really led me to curate a LinkedIn feed where interesting stuff was popping up. One day something popped up. It was a post from Ahmed Buzed who's kind of a thought leader in the voice space, and he was asking if anyone was interested in a voice role. Basically he framed it as like if you've never heard of that but you might be interested in designing voice interfaces you should reach out to me because again it was so early. It still is early to be honest in this field. So, yeah. And again, the reason I wanted to like mention like it was literally a post that I saw is because I was spending so much time on LinkedIn finding people connecting people and he was like a connection of a connection so I never would have seen that if I wasn't really focusing on people versus job applications if that makes sense. I was like hey, I really, I'm really glad you posted about this because I have never heard about this role before. I think it might be perfect for me and I'm just really excited that you posted it because even if we never get to work together like I feel like I have a lot of clarity now on like what I could be pursuing. And I was kind of pulled into kind of like an unpaid internship for a couple months and then kind of like a stipend type thing. After a few months and then about one year in, it turned into kind of a full time salaried position. That was my in I would say like that's really how everything started and so I was a voice designer at Whitlingo that was his company for two to three years, and then I moved on to Botmuck which is where I'm currently at. I'm not currently doing voice design like day to day but I can talk a little bit more about my role right now too if that's interesting a little later. Cool. So, I was intrigued and I want to underscore something you said which is. It was very early in conversation design. And yet, I know that there were all kinds of, you know, text to speech systems, or the reverse speech to text systems that were attempting to be as universal as we've got them today. I'm not yet doing that. So did you have any contact with those earlier in incarnations. Actually, no, like all of the veterans in the space and by veterans I mean people that were generally working on designing interactive voice response systems, like the ones where you call into calls or you call into Delta and you talk to a robot. It's really kind of the first super ubiquitous voice interfaces that people that humans were designing. And so, I'm sure like most people in that space that have been doing it for 10 1520 years, have a lot more familiarity with the, the tech behind it like the actual text to speech, and the speech to text and the inner workings. The interesting thing that I think happened just around the time I graduated was that we got to the point where we were no longer working with code if that makes sense and we were more working with you guys that your average person can use. So, to kind of like draw a parallel to just graphic design and visual interfaces on your iPhone or on like a website in the 90s when websites became, you know, ubiquitous worldwide, the only people that could create websites where people that knew how to code. And that was just kind of a function of how the technology can't like, you know, how it worked basically. Nowadays, obviously you can use Wix or you can use Squarespace, you can use all these other tools that you don't need to know any coding to be able to use and they're just a lot more user friendly. So, a very, very similar thing happened with voice, basically in like the 20 teens where instead of you know it needing to know a ton about the inner workings and what's really going on and having to know how to code to create these designs. Tools like Botmuck and others made it possible for your quote unquote non technical designer to create something just as robust as someone who does know how to code. Right. And from that early work in the 90s you may or may not be aware of the study but a guy named BJ Fogg and his students at Stanford looked at credibility of websites and one of the factors that they said that people that they got from the results of showing a lot of different websites to people this is, you know, 9293 era, probably maybe a little bit later than that maybe as late as 95. They demonstrated that websites that were more attractive in terms of graphic design were more credible than websites that might be from a trusted source like the professor who wrote the textbook on the topic. It wasn't all text. It wasn't formatted in any way. It didn't have familiar headings and footers. It didn't have some of those visual elements that may gave people confidence. This is a serious website. And so I think you're talking about similar kinds of tools that allowed non specialists to create things out of HTML when they didn't even know how to spell HTML right. Exactly. Yeah, exactly. And that's, I mean, that's a great point to bring up design is something that businesses invest heavily in because of studies like this where we know that people aren't going to trust your brand if the touch points that they experience with your mobile app or your website or your IVR or your customer service reps or anything. If it's not well designed thoughtfully designed, then you're going to lose customers and you'll you'll lose their trust so it applies to any like literally anything that your customers are touching when it comes to your journey. And that now includes voice interfaces to Right. So now we've got voices that come we say things and voices answer us, or we say things and text answer us. And it, you know, it's the Turing test every day multiple times right. Absolutely. I hope everybody here is familiar with that expression. Everybody knows Alan Turing, who was a computer scientist in the UK during World War Two, and his, he said, I'll know that computers are intelligent or whatever his claim was when I can talk to the computer, whether it's typing or speech, he didn't discriminate. And I can't tell whether it's a human or a machine at the other end, because the responses are so appropriate. And so that's sort of been a, if not spoken, then certainly unspoken goals that we should make the artificial intelligence part of this the conversational part of this credible as a human might be. And yet, I'm going to let you fill in the blank far away. No, yeah, we're still so far away.  AI is is not really like, at a point where we can trust it to perform reliably in most scenarios so we still, it's why we have designers backing these because I mean, some here may have heard of GPT three, I'll put it in the chat. Okay. GPT dash three, which is basically like an AI speech system that is now widely available to be used.  Because you can still say things that are going to cause GPT three to respond with something offensive or inappropriate or just not aligned with your brand. And so we need designers to be always like providing these guide rails guide posts guard rails. And gutters something like that. Good. Good. Okay. So, I'm going to jump around a bit because I definitely want to hear about what you're doing now. And so let's jump to the present and then we can go back and review some more things and as people are asking questions we can go ahead and do that. So you're now head of product at Bach Mach, what's product for you. What, like what is the product. Yeah, what's the product. The product is, let's see. I like to explain Bach Mach as almost like a figma or a canva or, you know, illustrator, but for conversations. Instead of instead of a visual asset that you're producing, you're producing a conversational asset, and Bach Mach just makes it easy for literally anyone, whether you're technical or not to map out different conversational paths and then be able to use that design to just shoot it over to an NLP platform that you'll be using and then deploy it so that your customers can actually interact with it. And so what kinds of functions do your bots get deployed into. Lots of different ones so if you if you talk to Delta through WhatsApp, they might use Bach Mach to design that conversation with that bot. If you talk to Alexa or Google assistant. They will use will use something like Bach Mach to kind of ideate and brainstorm in the early stages of their design phase. Go through approval processes, comment with their teammates names on okay we need, we need to change this prompt doesn't really sound right this greeting sounds a little bit off it's a bit long you know, all of this stuff that we're keeping in mind when we are designing a visual UI. So really anywhere you can have a conversation with a brand which broadly would be an IVR on a phone, or Google assistant or Alexa on a smart speaker, or a chat bot on a visual UI. That is where you may be interacting with a design that could have been designed on Bach Mach. So, I, for people who haven't been in these conversations before IVR stands for interactive voice response for when you're calling into a, you know, a toll free number, for example, right and they're trying to route you to the right place. Yes, exactly. Okay.  You're not the one who coined that phrase right. Absolutely. And that, and the reason why we feel that way is because the designs have been so bad. So what is a brain a linguist bring to this that an ordinary human does not. Yeah, great question. I'm so glad you asked. You're waiting for this one. So I find it so interesting because linguists are like, it's almost as if the industry is just now waking up to the realization that linguists, like, understand language front and backwards like that we know things on an instinctual level about how to determine patterns and how to think and analyze. For example, a prompt that might be a good thing to say to someone on the phone versus not say whether that's because, like, phonetically it isn't able to be well understood by the majority of the population or whether it's because the type of the vocabulary that you used is too formal and it doesn't really seem like it should be natural speech. I mean, that's why IVRs for so long feel so robotic. It's a lot of it I think is because it's not super well understood by non linguists that speech and text are very, very, very different things. And so when you write something down, you it, it, we're not intending for it to be spoken aloud most of the time. So, yeah, like when, when we're designing for voice, we are always thinking we're a we're always reading it aloud after we write it down because that is really the best way to be thinking about, you know, why does that sound off or does it sound too formal are we using vocabulary that is too elevated for, you know, for the target group that we're trying to speak to how do our users speak. And, you know, obvious, we have all the obvious things that we're thinking about like what language are we using with dialect but it gets so much more nuanced than that too. So, just the fact that this is our bread and butter and, and words are kind of the currency of voice design or conversation design. And I was in a great position because a we just love thinking about this stuff, which, which is going to position you, you know, from the get go to to want to just excel at it. But be we, we know things that non linguists don't know and that that can sometimes just really help solve a problem when, when something is just not working with a particular design. We're talking about design a lot and I have a feeling from, you know, knowing this group a little bit over the last few weeks that that term is so foreign to them, and they haven't yet thought of it as part of their lives. So let's let's try another attack at design. And I'll say the definition that I offered in my session on user research and design was design is rendering intent. Thank you to riff on whether you think that's a useful definition and how it plays out in voice user interfaces. Yeah, I really like that definition. And I'm glad that you called out and kind of asked us to backtrack a little bit because in my own course, I really wanted to also backtrack and go to the very, very beginning of how I wish someone would have explained design to me right when I was like graduating and had never really heard of it before. So my definition is not going to be as clear cut as yours. I, it's not my definition I borrowed from somebody else like it. And I think it gives good food for thought and good discussion. It's a, it's a great definition and it can be applied to, you know, any kind of design I mean design can be a physical object in the world like a tea kettle, it could be instructional design how you present knowledge that you're teaching. It can be obviously like a building, it can be a voice interface. So, to me, design is the perfect intersection of like beauty and function, where to me, art can only be like, if art only wants to be beauty, it can just be beauty. If science just wants to be function it can just be function, but for me design is kind of like merging those two things together and and when you're designing something you are intending to create something that does something that is functional but also does something in a way that is ideally really graceful or delightful for whoever is doing said thing. And sometimes the best design disappears so you don't even realize there is design there. Exactly, exactly. In fact, in most of the time, the best designs aren't even necessarily noticed. Right, right. So the most elegant the most beautiful I mean and then the other part is automation is the best user interface in some regards, except you've got to choose when to deploy it and when not to deploy it. Yeah, absolutely. So, speed dialing. I don't know if anybody even thinks about speed dialing anymore because we all have it all the time. Just the way we live, look up the person's name in your address book, poof, touch it and you're on speed dial. We used to have to program that stuff into our phones. And before that, then you relied on the operator to speed dial for you. Yeah, yeah. So now we have speed dial with with voice. Call Nancy, you know, exactly. I was going to say called real. Okay, good. Got the same idea. We're talking. So let's, let's go back and add some of these questions that I was proposing last night. So I understand that you have a bachelor's degree and it was joint between neuroscience and linguistics. Is that right? Correct. You never went on for masters and obviously not for a PhD. And do you feel any regrets for that at this point? No, I don't. Actually, I have a fun story about that when I was having my crisis senior year. I actually applied for the, my school had like an accelerated master's in linguistics program. It's probably fairly common. But yeah, I had realized that I had taken a lot of different grad classes. Actually as an undergrad, my department was small enough to where they were like, that's fine. I loved them so much. I just wanted to like learn everything. So, and I was also freaked out about what I was going to do after school because I was like, I don't really want to be a doctor. I guess I kind of have to, I don't know. So if I could maybe get into this master's program, a I'm buying myself some time, because I'm just like terrified of what I'm actually going to be doing after this. But B, I know that this is something that I love learning about. So I got rejected from that. And then I was like, Oh, now what am I supposed to do? Like that would have been a perfect bridge to like appease my parents, help myself kind of think a bit more, especially because now I knew a little bit about like maybe there are other paths for linguists, but yeah, I was rejected. And I think I'm like so, so, so grateful that I was rejected because if I hadn't been, I never would have probably found that post on LinkedIn, I probably wouldn't have, you know, thrown myself into networking, just like as my full time job. And, and when I did find when I did connect with Ahmed who became my boss, that, you know, he was just such a good mentor and that was really, really lucky but I kind of felt like I was in grad school when I was working for him because he, he, it was, it was interesting, like we almost were able to trade, like he was able to trade really, really good mentorship and teaching in exchange for kind of like not market level salary, if that makes sense. Right. So you took a little bump down. Yeah, you were not yet trained. But I'm sure after six months or one year you got a nice bump back up to what would be market level. Exactly. So, yeah, it was kind of a unique situation but again I want to like emphasize how, how finding people will really, for me, get you further than shooting your resume into portals. You're just graduated like two weeks ago and I'm just, I keep telling her like we're gonna, we're gonna reach out to people in LinkedIn we're not going to just submit to a bunch of different portals like you can do both but especially in this space if you find the people, the community is so small and so everybody kind of knows which companies are hiring and everybody kind of knows each other so if you, if you're able to connect with people that and make them kind of make help make them an advocate for your own career by keeping in touch with them sharing successes. And again, I want to say this too because a lot of times people will reach out to me and say like hey, you know can I have an informational interview with you and I'm always happy to do that but 99% of the time they never reach back out to me. And they've asked for the informational interview and they don't complete it or they do it. We'll have, we do have the interview afterwards. They might say thank you or something but I most of the time never hear ever again. And I know for me, when I was first asking for information interviews I was like well I don't want to bother you with my petty life like you probably don't care I'm very boring. But the truth is like I really really do care and if you and for the people who do stay in touch with me like I am now. Like we like I share, I share things with them that makes me think of questions they've asked her, you know, things that they might be interested in and they kind of like a two way street so yeah. And this is my time to then once again recommend the book by Karen wickery, which I had mentioned early early on and I don't know if you know this book and you obviously have embraced all the practices she talks about her last name is W. And somebody's going to look up that book for me. And it's about networking. Karen, and she's, she calls herself an introvert. So she says this not this is not easy for me to reach out like this but when I think of individual people who I want to learn something from or who I want to pass something to. I can do that because that's a one on one relationship and we're having that conversation. You know, asynchronously or whatever. And, and the other part is, she talks about, and I think you're saying the same thing, this business of staying in touch. You can always write back to somebody and it doesn't have to be I found my perfect job thanks so much for giving me that informational interview, all we have to write back and say, I saw this article and it made me think of our conversation. That's enough. It's a way of keeping that conversation going over the slow long term, and keeping yourself top of mind for somebody who may find out he's she's in the network she knows where all the jobs are. Thank you Marcus. Yeah, and Alex, I also like completely agree with this people do want to know how their info helped you.  But I just like, I've realized that I have no choice in life like if I want to like continue being able to do stuff that I really like, and also helping people and having people help me like, as linguists we know that communication is how things get done. And so, as an introvert I'm like, well, I'll embrace the conversation even if it, even if it's sometimes. And without quoting Margaret need let's just say small groups are very effective. Right, so you can have a small group which is two or three or four people it can make a fabulous product. Let's talk Yahoo, let's talk Google. There's something else that I want to underscore, which is you took a job at a small organization, and it gave you this huge learning opportunity. You happen to have a boss who was good at feeding you things in the right timing and scaffolding your development. But I think that in general or and I think in general, a small organization is often the place you can do that more easily. Absolutely. Like, and I also want to call this out because I've been encouraging my sister to start finding startups versus like these giant corporations that are listing jobs on LinkedIn. Because she was like, I don't want to work in a startup they kind of like scare me like it scares me to think that I could be in such an unstructured environment. I don't know like I didn't know that she felt like this maybe other people feel like this too but I want to call it out because it's not really like that like it's, it's not scary in that way it's it's more exciting, especially if you are looking for a learning like you just meant like you just mentioned Nancy and be like responsibility because responsibility is really hard to attain in the working world, and a startup is a really amazing way to, to find it like right off the bat and show that you can get things done and that you. Right, right. And so you and you recognize that there's a gap here and nobody's filling it so you're going to make an effort to do it and somebody else can clean it up. As soon as you've done the first draft, because you really don't know what the whole scope of this thing is but there is this gap. So let's put something out there that's going to fill that gap.  Right, I'm sure you've heard something like that, which I had a hard time with as a motto in my, you know, in general, I don't think of that as the way I do business right, except when you understand, you know, then you have to decide is the thing I'm trying to clean up and make beautiful or accurate. Is that really a stopper for the whole product getting out, or should we be our organization be first to market with this magical new cerebral login thing that I can just attach right back here. And it's going to get me to the website I'm thinking of immediately right or the app and or the Wikipedia page kind like that one. So, you know, if you're going to be first to market get that jacked in thing to market right away. And don't worry if the documentation has a few things that need updating because the last version. We eliminated that whole feature and you know because we replaced it with this magical thing. So, yeah, right. So that, you know, there, I think I said three things and one there but you get. Well, I think you you hit on kind of just the startup vibe like which is, we move really fast, and you have the opportunity to do a lot. And that, you know, you're, that's great in in and of itself but for when you're looking for your next role that will also give you a ton of a ton to talk about and kind of show that you already achieved and you're a self starter that's another thing that you can demonstrate through that, you know, nobody was doing X I took it on myself, I let my manager know I was going to do it and they agreed. And so, you know, we just went ahead and and obviously this turned out to be a really important thing because in the same moment as we released Salesforce came out with this thing that was sort of comparable but ours was a little simpler because we didn't have so many people working on it. You can iterate faster and we can iterate faster right right. Now, at the same time as we're talking about startups and startup culture and all that. I also want to put, put out the reminder that I mentioned which is for people who are not us citizens are green card holders, it may be tougher to get a job at a startup because they are not going to know how to sponsor you for work visa. And they don't have the bandwidth to hire somebody who knows how to do that. So that's a disadvantage of startups for some of our audience. Absolutely. Yeah, good call. Okay. What kind of a career ladder. Do you see in Vue for people. So, I would say an entry level Vue designer. The salary for that is around 60 to 75 K. From there you would go to kind of like a mid level designer. And then senior designers can make upwards of 120 K. And, and so that's just like kind of the straightforward, just Vue water. But for me, like what happened to me was my first boss came from a product role at Amazon. He was, he's first and foremost a product person, not a designer. And so product people and designers work together to sort of realize the whole vision. I like to think of the product person as what are we designing and then the designer as how is it going to be experienced. So you might have Amazon's mobile app, and you decide okay on this app you're going to be able to buy stuff and return stuff but we're not going to let you track packages yet that's out of scope we're not going to have that feature included. But the designer is who is the person that actually makes those features happen and and makes it so that either it is really really easy to be able to buy stuff or it's like kind of a pain to be able to buy stuff. And so people work hand in hand to take that analogy to like a Vue setting of Vue product manager would be someone who says okay, we're going to let people ask Alexa to turn the lights on and to set a timer. But for now we're not going to actually let them. What set an alarm. Yeah, yeah, play music that's a better one. But the Vue designer is the one who decides okay how many different how many turns conversational turns is it going to take for somebody to actually achieve this playing music feature. It's a very like close relationship and for me I started as a designer and realize that product is actually something I love to so that's why I'm doing product now. And when you kind of get when you kind of break into the design world. There are so many different paths that you can take from there you can go into product, you can go into UX research you can go into development if that's what you like there. There's adjacent roles on any given product team that are all super interesting too. So the latter is not just like a one lane thing it you can go like so many different ways. Good. Good. Okay, so this is we're not talking about a ladder we're talking about kind of a jungle gym or something like that. Exactly. Okay, good. That made me think of two things when you were talking. Let me see if I can draw up either one of them. I'm going to go back to my list and I'm going to come back to that then. So, how do you feel the covert is affected your work life or the other organizations that you're in touch with your clients. Yeah, that's a good question. I would say, looking back. Companies are investing more invoice now hope it did not slow that down it might have sped it up. Really like I would say for any kind of conversation design role, you covered didn't impact it at all and if it did it was in a positive way because ultimately, businesses are really investing in a lot of like alternative ways of communicating with their customers and that includes voice and chatbots. So, the conversational industry at large is growing very, very, very quickly and it's not going to slow down. Okay, I'm going to throw in two questions here one is okay I got I've retrieved some of those things that I was thinking about you also have taken on a big role in women and voice. So, how has that contributed to or enhanced your career invoice and what's what's it given you and what's what kind of things do you have to give it. So, I would say that getting involved with any organization like that, whether it's kind of like a local chapter of a voice UX meetup, or maybe it's women and voice or maybe it's, you know, any other like a Google, Google servers, experts program like all of those are just great ways to connect with more people to be perfectly honest and like, that's, that's really like why I'm still a part of it. Along with just the learnings and kind of the events that go on. Yeah, I mean, that's how I think of my involvement with Bay Kai is very much a continuing education program that for a long time I was the recipient of and now I'm offering person for this part of the time. Yeah, and that's what's so cool about voice is that a the community is really small yet. So it's just like really just get in now and like find people and make friends. Most people want to be your friend. Most people are very supportive. And the other thing is that okay, I just adore this about tech in general, but especially with voice. There are no right answers right now and I struggled my whole career, my whole educational career with like testing and math and chemistry and it was always just like this upward battle of uphill battle of like not being able to get those A's on those tests because I my brain didn't really think like that but I was so good at so many other things but the way that the school system is set up you don't really like get to shine in those other ways as much as you would if you were just naturally good at like chemistry or math. So, in tech, it's just like total opposite side of the spectrum it's whoever is has really good ideas and wants to put them out in the world and wants to contribute. And wants to become a part of women in voice or these other groups and just put themselves out there. No one has the right answers because tech is literally like 25 years old. And like, so everybody knows that we're all just kind of like bullshitting and like figuring out as we go and if you have cool ideas people will listen to you. And you don't need a master's degree or PhD to be listened to if you have those. That's great to because you kind of have like a really deep perspective on something really specific that you can then, you know, apply to whatever your role is there. But if you don't, it's just about like what you're doing, and it's not about what you've done. So that's something that I just like personally, I don't think I would want to leave tech because I know that, for example, my roommates who work on the hill for Congress men and stuff. It's very very much like the opposite it's more about like what are your credentials and tech is just not like that at all. I want to pull up the comment from the chat with just got answered and see if you have additional answers for that. And Wendy asked his women in voice open to members who don't currently have a job in vui. Oh, absolutely. Yeah, anyone who's interested in really, you can join. Same is true for all these other affiliative groups that we keep talking about. Nobody's going to ask for your license at the door. No, okay. No, expressing an interest and and asking what Vui stands for is still fine as a way of getting in. Then people get to tell you all about what they've been doing. And you get to ask more questions all the better. Yes. Yeah, I mean, even at these larger tech companies like I have a bunch of friends at fame companies like Facebook, Amazon, Google and the stores that I hear them tell it's it's really just like nobody knows what's going on it's more just about like what can we test out like tech is just such a crazy industry for that reason it's all about. It's not like you're, there's nothing really at stake right like if you're if you are a doctor like there's a lot of stake you need to have those credentials in place but with tech we're like, we're launching a thing that can talk to people like let's just try random stuff out and see what right and then there is other kinds of tech which where you better have somebody checking your work a lot, you know, because it's controlling medical devices or the power plants or whatever it is and so you want to have non fatal errors happening less, you know, less and less. They're definitely serious size of design to where you don't want to be, you know, you do want to be ethical and mindful of various different things but in general it's kind of an open box so we're still like defining those rules and we're figuring out what is ethical. So there's just so much room to contribute there, which I think is really cool. Good, good. There was a question here about whether a product manager, and that's probably different from product designer but you can take both both points of view. Does that require a business degree. Absolutely not. Absolutely not but I could absolutely see why you would think that. The product manager, the product manager role is also just kind of new it's, it really became mainstream, maybe 10 years ago. So, so product managers, like, I was talking to someone the other day who used to be like director of product at Chase, I think it was, and she has also worked kind of in. Really, really like high level roles in various product roles at all kinds of different companies. She was telling me that when she, she's been talking to a few like recent grads recently, recently that told her like I did a product manager focus in their degree. Like, how, like, how is it even possible to teach product because it really is kind of a, you really learn on the job. And there are a lot of different things that you can do to set yourself up to become hire a product manager but there's literally no way to train to become a product manager except to be one. And it's kind of like trying to train for being a CEO like nothing on earth is going to teach you how to be a good CEO other than being a good, other than being a CEO so You mean reading that tall list of business books on my bedside is going to make me a great CEO. Unfortunately not. I'm sorry. I wish it could but but yeah product is another really cool career path that I would be happy to talk to anyone about. And I think product in because when we talk about it often it has both the it's in the same way that at the university there's a dean, and there's a provost right so one of them looks inward and one of them looks outward. And the other product there are people like that too and so you're not doing all of it yourself if you're the product manager or the product, even the designer, you're probably relying on your UX team to give you some designs and get asked to get some research for you. Marketing people have other research that they're going to give you the forecasting people are looking at other industries around to see how that's going to affect their buying habits are going to affect your product success. And the finance people have other kinds of pressures to put on you. So there's a lot of input that you're getting both from internal to the company, the marketplace in the whole world. How your brand and product are perceived, and whether the market's ready for you to jump into voice. And at this point everybody's ready for everything to jump into voice. So let me ask a hard question, see if you have any sprouts to that and then I have another question. Well, I did also want to say as linguists, I think we're also really like we're cut out to be a great product manager to because product management is all about influencing authority. And the way that you do that well is through very particular types of communication and advocating and being able to rally people so knowing what we know about how language can be used as a tool can be very, very useful in the role of a product manager who is really kind of this conductor, or like, they're kind of orchestrating an entire thing they're bringing a lot of different people together and they're kind of the core. And then they're also like influencing things and making things happen, which is fun. Right on, I'm going to change the topic again, and that but it's still related to all this stuff. Talk to me about accessibility and voice. So, to me, voice is one of the last pieces of the accessibility puzzle if that kind of makes sense and I would love to know what your thoughts on that to our Nancy but I mean if you think about it. If you want to communicate with computers and talk to computers has always been like seeing something and, you know, typing some input or maybe tapping it. And then now we have you know you can listen to audio on on your screen, but you still can't really talk to your computer to make stuff happen. There's all these different modes of like input and output when you're talking to any computer or trying to get information. And with voice in and audio out as kind of being much more mainstream and probably also on the verge of being used in just your typical computer experience, like you can kind of picture instead of on the Google homepage well actually this already exists if you go to Google. There's like a little speaker where you can speak into it. So like, I don't know I like to think of it as this spectrum of ways that you can give information and get information. Depending on who your audience is, you want to be leveraging a variety of different ways to give info or get info, and that might be voice or it may be. It may be typing or it may be reading or it may be listening and and when we kind of think holistically about all these different modes of input and output then we are making things as maximum, maximally accessible as we possibly can. Yeah, I mean the first interface for the computer was the card deck, right, which wasn't or the tape, the tape role, neither of those are particularly accessible to humans. No. And not very efficient or and we didn't have a lot of memory to tax either so you know there wasn't much space for her storing anything, but I agree with you that I think that voice opens up a lot of opportunities for people at various levels of literacy. So in some ways voice is a great opener. And on the other hand, everybody with speech differences, whether intelligible to you and me or not, and it may be as easy as a foreign accent or it may be something like deaf speech or an arthritic speech or something like that, which is not necessarily as intelligible. And so then, can the machine understand better than the person. That that's an interesting case too. And I've been watching our captioning and noticing how well, I mean I'm surprised how well it handles accented English, and we've got plenty of examples in our group of strongly accented English and captions have been not perfect but you know, pretty darn good given the history of bad captioning. And then of course so if you if you can speak to the machine, but if you can't hear the response from the machine or understand the response, then that's tricky too. This is, it's a wonderful set of dilemmas to have with an almost robust set of technologies. Yeah, more and more robust. Yeah. Absolutely. Yeah. And there's a there's an interesting model that I've been using a lot more lately which is called the social model of disability. Okay, which breaks it breaks disability into three distinct categories so first is permanent disabilities. For example, Alzheimer's, or like motor impairments, or how about trifocals. I mean, I'm not disabled, but I do wear a trifocals right. No, that's a great one. And then there's. Okay, permanent. And then there's temporary, then the temporary like broken arm. There's temporary, like broken arm or having laryngitis, right. Can't really speak to Alexa.  Exactly. So, so yeah, I think, again, like thinking about all these different modes of input and output. Also, I think that situational or the social motor disability can also be really helpful when you're thinking about like when would something be best leveraged. Yeah, it's an interesting dilemma and it's worth us thinking about each time with each new product and each instance of a product, because who are you excluding just by the modality.  How do you alert the engineering staff. This has got to be built in the following way. Yeah, because we are disabling people right disability is the, the somebody's life circumstances whatever it is permanent situational and the built environment, having incompatibilities where there's no way to get them together. Right. And even just for, for, like, the, the general population, we all have kind of biological constraints where we can, for example, we can read and scan something way way faster than we can listen to something being spoken to us. So, you know, that's kind of an accessibility consideration as well but it applies to all of us just in the, in the general population. Same thing goes for speaking, we can speak much faster than we can type so, again, it's kind of like this puzzle of like how do we want to arrange the input and output. So what kinds of testing do you get involved in with the products that you're designing, or what kinds of testing happens whether it's you that does it or somebody else. I would say, on a team that. So, as our product leader. I'm also kind of like our only researcher, our only tester. So, like, make do with what we have which is really customer interviews are like crucial like, for example if, if someone from my team has an idea like okay let's add this into the product maybe this is a good idea as product manager. My number one job is to say like, okay, cool idea. Should we a should we even do this be should we prioritize this if the answer is yes we should do this. And it's so tempting as humans to kind of put ourselves into the shoes of our users and say like, absolutely that's a great idea I would love to have this in that product so let's do it. But, but it's like, for me it was just so important to break that habit and always say like, I could be completely wrong in every scenario so I'm going to go ahead and ask, at least three people that are, you know, our customers. It's kind of a great sample size but when you're very limited on time and resources to do any testing it's at least something to where we can kind of get a read on the situation and that will you know if three different customers are telling us, oh yeah like we've wanted that for a while. It gives us more to go off of them. And just being like, oh, I think this is a great idea. So, interviews are crucial and then we're kind of building out a more robust testing process for when we're actually like beta testing and a new feature. Yeah, like the teams that use our product also do a lot of testing and that, like you really cannot launch a conversational system without doing testing, because fundamentally, the system isn't going to know what to do unless it has data already kind of being fed into it. So what kind of testing do they do? I understand you do these semi-structured interviews or you show them something and so on, and I'm going to put the Nielsen-Norman article about how many users, because that's actually the statistical justification for five users. And there is an experimental design that uses chi-squared as its statistics and therefore you can have some level of confidence if all five of your users come from some sameness, right? And you don't want to have one user who's a end user and one user who's a beta C user and, you know, from different kinds of marketplaces, but if they're all generally from the same group, then five users is enough to test a feature or a small number of people. That's interesting, okay. Yeah. And that's why I've been reading this website, the Nielsen-Norman group website, before it was Nielsen-Norman, it was just Jacob Nielsen in the beginning. And he put out something called alert box every week or two weeks, and he was the first avoider of graphics, and he talked about why it was an all-text thing. He's since been gone on and, you know, this is 20 years later or more, 30 years I think he's been doing this. So you can understand why 30 years ago he was not real fond of graphics. They were heavyweight, hard load, and mostly being used for decorative things. Now graphics have lots of shared meanings among the users and so on. Anyway, you're fine with only three because you know your product and getting three voices is pretty darn good. So when your other teams you said are doing their testing as well, what kinds of testing are they doing and how is it different from what you're doing when you're interviewing three customers? Yeah, so the teams that are using our tool, like our customers, they are building conversations and they will actually use kind of our built-in testing feature to send a link out to someone. So if I was creating a voice experience and I needed to know if it was intuitive or if it was confusing people in certain areas, I could send a link to you Nancy and then you could actually like interact with it, whether you type or you speak to it. However, they wanted to set it up and then they like that team then back in their own portal, they would be able to see, okay Nancy kind of went on this little detour, it looks like she got confused here, but she got back to the happy path and then she actually achieved the goal. So this is like kind of how I'm seeing teams actually test their own conversational experiences along with, you know, gathering as much data about how Nancy is going to ask to track her package. There are so many different ways people can ask that and your model is going to break if you don't have much variation of what to account for, what to expect when people ask your model that. So with conversation, it's way, way more important to do a lot of testing because fundamentally the system can't really function without that. That's a great example I hadn't thought about it quite that hard and that linguists are so good at paraphrase and so this is the place where that kind of ability is definitely going to be needed. Oh yeah, absolutely. And as linguists we know how to come up with a lot of different ways someone could possibly ask for something. But again, it's always important to break out of that and acknowledge like we can't we can't possibly think of every way someone could ask for something so we're going to, we're going to rely on the testing data. Yeah, good, good. Moira has a question really quick. Good. Okay. Best way to get internship with a startup. Is to find the startup first and then email someone. So, let's see. I would recommend like actually discovered a trick recently. Okay, there's, there's a website called crunch base. I don't know if anyone's familiar with that, but it is basically, it's for like investors and founders, mostly. It's like an investment website but the reason it's important is because basically every company under the sun that's ever received funding from an investor is on there. And when a company and it will list based on like okay it got seed funding which is like its first round of funding from investors or it got series a funding which is the second one so on. And then an idea of how big the company is. And if they just received funding, they're definitely hiring. You can kind of backtrack by using crunch base to find companies that interest you, and then using kind of like what you know about what their funding level is. So, I would recommend it to say like okay maybe this is a good place to work at maybe they're hiring and then always make sure to first try to find a real person like whether it's on LinkedIn. It could be a recruiter or maybe it's just the founder if it's a really small company. And then like try to find an email if at all possible if not find them on Twitter or LinkedIn. That's where most people will be living. If that doesn't work submit your application but also just be persistent because things just get lost in people's inboxes, and it's not, it's not annoying when people follow up it's actually really helpful. And then if that doesn't work just find a different person at the company. And if that doesn't work, then give up because there are a lot of other companies out there that you could work for. I love that advice somebody asked Janice wants to know what we can learn in school to prepare to work in vui. That's a really good question. If your school offers any classes in UX or design. That would be kind of like the first thing I'd recommend, because ultimately vui design is a subfield of UX design. And then otherwise a conversation analysis class would be really good. Discourse analysis drama classes, because when you are, you know, when you're acting your, your performing dialogue and dialogue is what we are creating and designing when we are conversation designers. What else creative writing would be really good. I took this class called you become as in like ubiquitous computing. That really kind of opened my eyes to the world of tech, but yeah, I don't know. You become was the term found a form. Oh God, and my poor spell check doesn't know it. By the term was found. Sorry, term was created by a guy named Mark wiser who was at Xerox park for many years, and he used to say the difference between virtual reality and you become is you become is appropriately compatible with reality. That was a funny thing to say. That's awesome. Yeah, you become is a fun field. It's also fun community if you get involved with the conference and stuff. Yeah, fun people and they're good imaginations. Yeah. Yeah. So we're hearing two things from you. One is, yes, there's lots of things you can learn in school that will be helpful to you in developing a vui career. And to, there are lots of things outside of school that you can learn from peers in the field right now and peers in adjacent fields. And so don't count on school being the only place you're going to learn this stuff. Yes, don't. And you can do all this yourself like there. All the tools are out there to create your own portfolio to design a project.  I made up some data about them. And then we really just designed a voice experience from the ground up for them. And my students today and on Thursday are going to take everything we've learned and kind of package it up into a portfolio. So none of this was like part of school like I really just thought of all of this with my imagination and you can do the same you can also you. Things that you did like creating the company, creating customer base and creating some data about it really were a way to put constraints around what kind of a product you could create for that audience and that product space yeah. So that you don't have to think about everything in the whole world and that's very confusing and too much to. This is a filter that you've given them to be able to constrain some of the parameters. Yeah, exactly. And Miriam I think you and I were at the same you become conference because I was also in Hawaii. So that's cool. There's something called a wait, were you talking man. I just said fabulous because I learned a ton about the kind of work that people in that area. Do I learned about the research I learned about the cool things that people were designing. And I was at a resort hotel in Hawaii. So what's what's bad there nothing is beautiful. I even still have a couple of the lunch bags are used all the time. It's swag. Good swag. We got lunch in the sustainable bags so. Right. Yeah, that was awesome. I'm getting on the topic of jobs and internships and and what you can do outside of school. I learned recently about something called a value validation project. I'm going to link to right here, including article that really taught walks you exactly through how to do a value validation project and the whole like idea behind it is that, you know, any hiring manager any team gets so many applications to a role than they could ever hire into it. So if you so obviously like everyone always says you need to stand out but how do you do that with a resume and with a portfolio. A value validation project is like, I think the most cut and dry way that you can do this. It really is just about like showing what you know and how how you're willing to kind of showcase and put in effort and time to show them that versus just, you know, the fact that you maybe submitted the same cover letter to 15 different companies. Good. Thank you for that hint and we'll definitely look that up. That looks like fun. Okay. I think we can say thank you very much. And I really appreciated this. Let me. One thing I was thinking about while we were talking was if Greg were here. I think he might have chimed in with. There are things that you know in linguistics that people in engineering have never thought about and are useful for you to communicate with them. For example, if you look at his LinkedIn and go back a year or two, he gives a talk where there are 10 things about conversation that you need to know to be a good conversation designer things that are so obvious to us. You don't think you have to tell people but you do. Yeah, she has an opening, a conversation has a closing. People greet one another and say their names or identify themselves. So your bot should say I'm a bot. You can call me Harry or whatever but I'm a bot. And then this business of bots have now gotten smarter so that you don't actually, if I tell you something in theory you should remember it you bought should remember it and be able to follow on from that and not ask that same question a different way. And so Greg had these like 10 little items that seem to a linguist like this is totally pro forma. Everybody knows this. It's true everybody knows it, but not at the level of being able to say it. And of course a nice number for a listicle is 10. So I'm sure he did a little bit of squeezing to make them all fit into that 10 model, but I thought that was a very clever thing to do and it's certainly got him a lot of attention, both inside the and outside the company to be able to take something like that what we think of as common knowledge, and it is sort of common knowledge, but giving some very specific guides to this is what you know, this is what people assume. This is what people don't want to see all combined into that little list was very helpful. Really, this is great and congratulations on the course, and thank you so much for spending this hour plus with us. And good. Thanks, Nancy. Happy to spend time. All of these amazing sessions. My pleasure.