quickly move on to the next session and it's time for some crystal gazing. Our next session is with Sarah Salter, Head of Innovation, WaveMaker, who will tell us what the future of voice in marketing is. Over to you, Sarah. Thank you so much. I'm just gonna hopefully manage to pull up my screen one second. So firstly, thank you very much for having me. Super excited to be here today. I am very much gonna talk to kind of the future of voice and share with you how we're seeing the future of voice and how we're very much kind of seeing its future in marketing and advertising. So I think in the past, marketers are very much kind of given voice technology a respectable nod, but this year we very much expect to see a shift where voice and conversational AI will play a much more central role in those marketing plans. And this is very much kind of driven by a few factors. Essentially we're entering a new era in voice with two key drivers converging to create this big moment in time. The rise of intelligent applications and also AI that creates a more personalized web experience and then meeting this paradigm shift that is voice as the next stage of user interface. I'm sure many of you have seen this happen in your own households, but the pandemic has very much accelerated the adoption of new technology at home by potentially three to four years. Many researchers stating that families will become much more reliant on voice enabled devices over the next five years. And voice growth is very much coming from kind of what was talked about earlier, this intuitive nature and the rapid adoption of technology amongst younger audiences, many of whom are growing up with assistants being spoken to multiple times around their house. And a lot of research has shown that parents are now putting voice assistants in baby's bedrooms from birth. We're also seeing as we touched on earlier, seven year olds essentially using voice to select what they're gonna watch, search for information or even ask what time dinner will be ready. I have a six year old son and he already chooses voice over dials, controls and buttons. And he picks voice for its speed, its convenience and because he genuinely believes that Alexa is part of our family. His conversations with voice assistants are very natural and intuitive. He'll even argue with Alexa and he'll keep reframing his question until he gets an answer. Whereas my conversations are as a digital native, they're very search focused and they're very demand driven. I might ask Alexa to play a song, whereas my son would ask Alexa to have a dance with him and pick a song that they both like. Just kind of showing how he very much speaks to this technology in a completely different voice native way. A lot of people also, a lot of children also believe that Alexa is a real person. A lot of research has shown that people think there's actually a real person sitting inside that voice device. And what to expect? So by 2030, we believe that every brand will have a personalized voice and that conversational marketing will truly be the future of customer experience. Perfect, thank you. So as I mentioned, studies have anticipated voice assistants becoming much more conversational, joining in with family conversations more naturally, but also with personalized content, being able to make dinner recommendations, being able to pick brand preferences for people. And if you go on to the next slide, what's gonna be really interesting is that voice will essentially play a role in every aspect of our lives. We'll see voice powered interaction expand beyond the car and smartphone and smart home to new areas of mobility and other unexpected areas. Voice will be in retail, in transit, in travel and the public will become much more accustomed to choosing voice as one of several ways to interact with a product. So within the same product, people will discover that sometimes it's easier to speak while other times it's easier to touch or type. They'll realize that voice and the option to use voice is not just a novelty and the results will be that there will be even more products that will be voice enabled and will allow people to interact in a manner that suits them. Next, essentially the net effect is an emerging channel that is already mass. I'm based in the UK and in the UK alone, we have 86% of the UK public having used a voice assistant in the last six months and we're seeing huge stats globally from this increase. So on the next slide, I'm going to just kind of touch on essentially five of the biggest trends and emerging technology in voice innovation. That's the ubiquity of voice, touching on the growth of voice social. It's application in retail and voice commerce, multi-sensory experiences and going into some kind of deep learning and synthetics. I'm very much going to kind of touch top line on these areas because obviously they're vast and hopefully give you some good examples. So next, essentially voice will be available everywhere. Every home, every product, it will become standard on appliances and what we're seeing from voice manufacturers is brands like Amazon have already brought Alexa to property managers. They're aiming to have entire homes that are smart home enabled and make it easier for property managers to set up and maintain their smart homes. But perhaps the biggest opportunity is voice on the go. And the mobility industry is very much on the cusp of revolution, electrification, connected and autonomous driving and shared mobility. And what we're seeing is these automakers investing in their own voice controlled assistance. However, research is showing that car owners are increasingly preferring familiar voice assistants like the ones they have in their home over those specifically designed by automakers. And we're seeing just this huge increase in AI powered in vehicle voice assistants. And the possibilities to utilize that time whilst driving and connecting on the go are endless. And then finally, we're also seeing voice integrated into applications, integrated into websites. This example is Snapchat where you can now search for your lens via voice. And essentially it will make those complicated website mobile navigations much simpler. It will kind of go from being very much nice to have which brands are looking at it right now to an expectation. And it will also allow marketers to focus much more on context. So how and when they communicate. And on the next slide, I'm just gonna play you an example. And this is essentially a product that we have developed with nationwide bank. And it's a hyper local voice skill. And this voice skill was very much designed with auto in mind. And it's the idea essentially that whilst you're on the go that's the time when you're keeping your eye out for new properties that you might want to buy. If you don't mind pressing play, I'll let the video do the talking. Alexa, open nationwide local voices. Welcome to nationwide local voices from nationwide building society. Find out what life is really like in 20 of Bristol's most diverse and vibrant neighborhoods. You can hear from people who live in Bristol about the culture, community, and the places in their neighborhoods. Hi, I'm Ben and I live in Montreal. Bathville had a lot of families. One of the best things about Kings Island is there's so many places to go out in the evening and meet people. So we moved to Brisbane to about 17 years ago A lot of the traders at this market, they're local people that are setting up their own businesses. It's the first outdoor market in the Northwest has ever had. Thanks for listening to nationwide local voices from nationwide building society. Goodbye. Just an example of how we're really kind of thinking context first and looking at the future of audio in car and voicing car. And now moving on to kind of one of the other big trend areas. And this is the future of social media. And we truly believe that the future of social media is audio and voice. I've evidence this kind of on the left hand side by the rise in voice only platforms and also the recent purchase of locker room by Spotify. Voice content essentially is perceived as being more personal and more informal than text. People can create more content at a much faster rate. So the average person speaks three times as fast as they can type. So it's a great way to be able to put more content out into the world. So why is this use case taken off? I think humans stuck at home during quarantine readily available smartphones and apps, the lights of Clubhouse, for example, and the desire for human connection beyond text. And probably the fatigue of just too many Zoom calls as well. Essentially, there's a huge opportunity in this space and an opportunity for brands to be first to market through emerging ad opportunities. So sponsored audio influences, live voice, commerce, promoted rooms and content that could be extended from social platforms into branded podcasts, potentially. The opportunities are really endless in social. And it's just a huge moment to be able to kind of experiment. The next area I want to just touch on is retail. So going on to the next slide, voice essentially will replace touch in many situations due to the health concerns post a pandemic. The challenges of the pandemic are unleashing completely new use cases for voice first systems from the obvious kind of touchless functions to much more friendly food ordering. Voice will just continue to rise throughout this year. It will make its way into retail stores, restaurants and any facility where voice can reduce friction and promote that contactless transaction. The other area. Next slide, please, in retail is essentially shopping and voice will essentially change the way that we shop. And a great example of this is Alexa and Whole Foods. So essentially what you do is you would put in your your buds or through your phone and talk to Alexa. And this is an example where essentially Alexa will guide you through a Whole Foods market. It will tell you where to find the products on your shopping list or it will help inspire you and tell you where to find the ingredients on the shelf for a specific dish. And I think this is a really great example for marketers to be really thoughtful about how they can kind of build those in retail voice experiences by building those helpful inspiring audio nudges in that path to purchase and combining those physical and digital retail experiences. And on the next slide, I just wanted to touch on how also we can open up new shopping environments. And I touched on the car use case before and I think it's a very early example, but it really is only just getting started. Being able to shop for groceries or to take out from the car creates completely new environments where people will be able to shop. And as I mentioned, almost most major car manufacturers are adding in car voice assistance. And the Amazon Alexa auto, which you can see on this picture is equipping older vehicles with voice assistance.  And you can do all of that without taking your eyes off the road and also filling that dead time, you know, where you're just driving often and looking for something to interact with and do. It is a huge, huge opportunity for voice retail. And next, please. And then kind of going into the area of multisensory sound is well known for either entertaining or providing helpful factual information. But also sound is feeling. It plays a really important role in adding depth and dimension to the digital world, and it can also help enhance our real world experiences. And up until now, the digital world is predominantly through visual and there's a great article that Microsoft have written about where this sensory imbalance of just being visual can cause us to feel overwhelmed, distracted, depressed. And this is actually an opportunity to tap into the unique way that voice sound can evoke a feeling, a memory and emotion and help capture and hold our attention in much more interesting ways. Next slide, please, is a great example, essentially, of capturing that full attention is an example from Darkfield Radio. And it's an experience that's been created just primarily using voice and audio. And essentially, this experience, I watched a YouTube video with somebody describe the experience and they said through sound alone, they felt like they could feel the breath of someone on the back of their neck. Super creepy, but very, very powerful. And I think very much like the early days of other media, we need to be exploring, experimenting and inventing new combinations of sensory experience and finding new ways to connect the digital world with our physical and emotional worlds. We need to develop a creative process and methodology to help us design across the senses and across all devices. And then touching on my last area on the next slide, please. And this is very much around deep learning and synthetics. We are already seeing a huge number of brands explore synthetic technology from personalized messaging to personalized assistance. And this is an example from Nestle, where they essentially created their own cookie coach to solve the problem of customers getting different advice on how to bake the perfect cookie from their customer assistance. And this example on the right hand side from Lays shows how future customer experiences need to feel bespoke and personal. They essentially created a chat interface that would allow you to have a dialogue between the customer and Lionel Messi, the footballer. And these synthetics and these kind of personalized voice experiences are only going to grow. And if you could go to the next slide and just play a very short clip of this. Really, I think where the opportunity potentially lies is in the incredible, incredible advancement of synthetic voices. And this video, essentially, next slide, please. Shows this huge advancement in synthetic voice and the emotion and the realism that can be built into it. If you just maybe play the first 20 seconds just so you can kind of get a feeling for it, that would be great. Is it not allowing you to play? No, it's not allowing you to play. Let me see if it will allow. Let me see if it will allow me to play one second. OK, if you guys share your screen again, you should just be able to hear my sound for the video, which might work, given it's audio. If I'm lucky, I hear her. It's like she's right here with us. Her warmth, her love. That is her. I just don't understand. So just an example, hopefully you all heard that. OK, let me see if I can come out of there now. The computer is not playing ball. OK, great. If you don't mind the deck back up. So essentially, there are five opportunities to achieve an unfair growth advantage in voice. Firstly, we need to focus on the context. So very much like we've done with nationwide, what are those moments where people are going to be needing your brand and where can you be there for them? We need to utilise, test and learn through emerging social opportunities. Social voice is huge and nobody is yet taking full advantage of it. Building helpful or inspiring voice nudges into that path to purchase, defining how voice should make the audience feel and not just think and owning a persona with a branded voice assistant and thinking about bringing in synthetic voices. And finally, you know, essentially, who better to kind of quote in our summary if you don't mind going to the last slide than Vanessa Go at Spotify, who really kind of states why audio has and always will be such a powerful way to connect with audiences. Essentially, you can close your eyes, but you cannot close your ears. Thank you very much and sorry for the slight technical hitches. They always happen, right? Thank you. Thank you so much, Sarah. And what a wonderful session that was. So this small glitch in the digital world is what the transition is all about. And we can't really do anything about that. But wonderful session nonetheless. Thank you.


Well, with your Suggestion of what's wrong with the concepts of self individual and personality. I'm not totally surprised and I'll tell you why I I'll say it again. What's wrong with the concepts of self individual and personality. I come From a place where this professor went around from university to another university holding lecture lectures where he Pronounced that there was no I He said it in Hebrew, of course and in Hebrew. It's even more elegant. He said ain Ani ain is Aleph you'd noon. I mean is Aleph noon Yud He said there is no I and he said to this lady in the front row you know you and I have much more in common than Me today and I did Samach when he was seven years old so Absolutely agree Regrettably I did Samach took this from David Hume David Hume Wrote exactly this text more or less the redhead the the Scottish redhead. He wrote it in 1739 So in 1739 you wrote there is no He wrote actually self is fiction there is no self and I have very so very little in common with with myself in my youth Okay, he said he said actually and I fully agree with that it's Emma and David Hume and so on so forth I think men is a river not a lake Men flows panter a heraclitus Panter a you cannot enter the same river twice, right and you cannot meet the same person twice twice So you can't meet yourself twice Of course you cannot meet yourself twice Consequently psychology will never ever be a science. It's a pseudoscience Because the subject matter is mutable Constantly changes and it's transformed in substantial and meaningful ways over periods of time So but that's beside that's that's another issue another topic all together There are many current current day scholars for example James Giles of Canada And he says that there is not such thing as self. There's not such thing as personal identity Philip Bromberg who came up with the self states theory Young himself had the concept of complexes which were divorced from the rest of the self So in effect he described multiplicity of selves in object relations theory We have what we when it was called ego nuclei and this ego nuclear nuclei Ultimately merged and integrated, but you spend a lot of time as a child having multiple egos Some personalities is another theory with some subpersonalities. There's a theory With egos to ego states ego state therapy even and there is something called internal family system Which also involves a multiplicity? So what I'm trying to say multiplicity of states what I'm trying to say is that This is not a new idea at all the the Opposition to the rigid concept of a core identity that is immutable not changing That is not amenable to to modification via the environment It's very reminiscent of Darwinian evolution where all the change in Darwin's work all the change is internal through mutations The environment has no impact on genetics and today we know it's not true today We are more lamar kiss than we used to be and we know that the environment does modify genetic expression And by inconsequence modifies heredity. It's the same in psychology. There was this perception of a Core core unitary rigid self that is formed Latest age six and you are like that for the lifespan. Can we call it suchness? It's a each person each person who would have a certain suchness. There will be yes There will be such he is there will be a hand illness and a vacanness. Yes, and it would be for life It's not if you consider for a minute Following fireman you should always ask yourself in which cultural social context Was a theory developed in my home of What of this of the self being being valid being being a unitary core thing? This was developed by Germans and Austrians at the turn of the 19th century in the beginning of the 20th century These people lived under Kaiser Wilhelm and in the Austro-Hungarian Empire Habsburg the Habsburgs These people lived in a middle class with extremely rigid values Victorian in effect. So these people who had invented the concept of self Their their mindset was hierarchical fixed rigid disciplinarium Etc. So of course they created and a picture of psychology, which was I don't understand What what happened before that? I mean didn't people know that they I'm I'm I and you are you no There was no concept of self. There was I of course there was I but there was no concept of self Concept of self was up single-handedly invented by Freud and young later. You you mean the the the term So not only the term the perception that there is a unitary court that never ever changes and so on on the very contrary in the Middle Ages The church offered you a way to not be you You were sinful and the church offered you numerous ways to be absolvations indulgence Indulgence in changing your faith a change the concept of change personal change Yes, right was the core ideology in the Middle Ages. You're right. You're right Yes, and you can you can change and you're you're substantially change. You're Asked to change Invited to change but not change in minor ways substantially change from a sinner to a saint and so And so but because in the in medieval in in the medieval perception of of humanity There were two states either you were a sinner or you were not a sinner or even saint and so these were the the organizing principles of mankind's soul or psyche and You could transition from being a sinner to being a saint while in in the psychology of Freud and Jung You were if you were a sinner by age six you were a sinner for life Oh, really? I mean if you if you developed there was a fix self There was no hope for redemption not after not after not after six years old After six years old which is called the formative years You developed what you called a constellated self and it was with you for life. That's it It's interesting Constellation in in he was mazal and mazal is the mazal is a constellation. There was your luck luck your fortune fortune So and I'm asking myself. Why did they create such a rigid? theorem such a rigid, you know way of looking at the human beings because they were living in a society that Regarded human beings like that Society which was hierarchical dictatorial rigid Victorian that was also the time when people had to receive Family names, so they would be recognized. Yes, it was a regiment of society bureaucracy was first created by Bismarck and then you had Weber who was a few reticent of bureaucracy Human beings became Atoms they became cogs and wheels and wheels in a big machine Identity identity numbers are very important. Yes, for instance, if you want to know whether I am I ask me What my identity number is and then the Benny handle of yesterday will definitely be the same as the one of tomorrow because Tomorrow, I'll have the same identity number as the day before yesterday. Yes, so This unitary core is counterfactual. It's not true People are not like that People do change and so the my perceptions that man is a river not a lake and so I came up with a new Theory in essence in psychology that is now gaining gaining foothold in the mainstream Possibly will replace the paradigm of self and individual and my my work is I Called it pseudo identity. So here's pseudo identity. Yeah, I will explain it in a few sentences First of all, I I I say that human beings have an operating system The operating system is internal And the operating system makes decisions about Which Benny handle will interact in any specific with any specific environment? So I'm saying that there are many many Benny handles. I call them self states after philly problem There are self states. There are many of them They are many of them simultaneously It's not that they all exist Simultaneously their core extent But there is an operating system that says Benny handle number seven should come to the four Right because of the so how does the operating? How does the operating system decide? Which Benny will come to the four which Benny will be it it takes in queues information from the environment And information from inside internal queues It puts them together and it says Benny handle seven is the best suited To deal with this with this set of circumstances With these constraints With these boundary conditions with these specific people who happen to be here, etc So Benny seven is best suited Benny seven come here Go out and take care of things. I have an example Benny handle used to be a radio announcer And Benny handle three was now we're going to play a symphony by Tchaikovsky Benny handle seven would be excuse me. Do you know where Chernichovsky street is which is a totally different Benny handle? Yeah, the Benny handles the self states They of course have a lot in common But they have specific sets of skills and traits And behaviors and even facial expressions And so on that are Sufficiently differentiated them from each other and there is an operating system that decides based on on the reality Internal reality and external reality because sometimes another Benny handle will appear Because internally he became depressed So there will be another Benny handle who will tackle the depression in order to guarantee survival So what what is the principle? How does the operating system decide which Benny handle would take charge? At a given moment at a given moment reactive to a specific environment and to specific internal processes What is the decision-making procedure of that operating system? It's something called self efficacy Self efficacy is a concept in psychology that says That people try to extract the best outcomes From any given environment. This is called self efficacy. You're efficient. You're efficacious The operating system is asking the question and only one question Which Benny handle self state is best suited to the environment Taking into account his internal state Best suited to the environment in the sense That he will likely obtain the best outcomes if he takes over So then the operating system suppresses Benny handle three who is not best suited And bring forth Benny seven who is likely to get better outcomes from the environment The next thing is what happens if Benny seven and Benny three Are equally qualified to tackle this specific environment with these specific people The internal states of depression or anxiety or love Or fear or what what happens if you have two self states which can tackle the situation equally equally It's a stalemate Okay, and one of my major contributions to To this approach is to suggest that it is possible to have binary systems Because until now everyone Philip Bromberg and Giles and all the names are mentioned They suggested that there's only one self state at any given time And i'm saying it's not true It's possible to have two self states competing arguing Conflicting and being co-present Yes, and because they're equally qualified and they both want to manifest and to express and so they begin to have a fight An internal fight And and this this I call it the binary system And the binary system Gives rise to mental health disorders I actually succeeded to to to show direct linkage between binary systems And narcissistic personality disorders mood disorders anxiety disorders Stuttering Possibly yeah, it's called somatization if the stuttering is you know why I thought of that because In in Hebrew we say the gum game which means gum You have the same syllable again gum gum and gum is is also and gum is also so you have two Forces competing in your in your psyche And and you don't know which to bring out so for a long time I've been thinking that Stuttering may stem from the fact that you don't know how to say what you want to say So you say it in in in doubles or in triples two two self states are trying to obtain monopoly of you And because they're equally qualified to deal with the environment They create a stalemate So this stalemate may somatize may manifest in physical physically Stuttering paralysis Freud describes states of paralysis Uh, your hand can be paralyzed your face. I mean The body can take over and we have today Scholars like van der keel can so on they they clearly say the trauma can create somatic Bodily manifestation And so the binary system explains almost all mental illness It's when there is no consensus as to which self state will take over In narcissism for example, there is a false self and a true self The false self is the facade the compensatory facade Compensatory facade that the narcissist is trying to show the world. I'm great. I'm perfect. I'm brilliant. I'm decent that Internally, he feels inferior. He feels so he's compensating these two self states Are equally efficacious and so this creates a lot of dissonance in the narcissists most of the time But if he is in certain environments, the false self is much more qualified And then the narcissist is at peace so when the narcissist for example is Is an artist And he has an audience And he's playing the the violin and the audience is appreciative and they clap The false self is much better qualified to deal with this situation. The false self. That's the facade the facade that i'm brilliant Violinist It's much more qualified. Right at that point the narcissist is at peace He feels wonderful He feels elated. There's no conflict. No dissonance. No argument because because the the the real self is put to sleep Yes, because it's not relevant. It's not right efficacious with it. I see that but Binary the possibility of a binary system gives rise to mental illness one of the main problems in previous theories of self states They couldn't account for mental illness because think for a minute if the operating system Operates on the basis of efficacy the basis of efficiency.  Something's not no much. Yeah, something's not working You're not efficient So if you have only One state situation where the operating system chooses benny 3 and benny 7 benny would be perfect forever There will be no problem What's but what happens if benny if benny develops a mental illness how to how to explain this Did the operating system fail? How could it fail? It knows which benny suits which so my explanation is they're actually competing bennies multiple bennies who are who are competing And the operating system is unable to decide between them Because they're all equally efficacious So this is I won't tell you why but I think I I know what you're talking about Yes, everyone knows what i'm talking about because every my work is based on much much closer to reality than the concept of self and because These are obstructions the concept of self is an obstruction anyone will tell you That they had situations where they didn't recognize themselves Anyone will tell you I can't believe I did this it's it's so not me Yes, true. It wasn't you. It was somebody else who is also you. Yes, it's a self state and this leads to This leads to the question of switching How the how is the switch taking place? So I came up with I I enlarge upon concepts of collapse and modification Collapse collapse and more more ratification Mortification collapse is when the self state no longer yields self efficacious outcomes fails the self state fails The sets of fails this creates Mortification mortification means enormous fear enormous apprehension anxiety dysfunction paralysis Etc etc etc angst Exactly and yes angst is much better word and then the the operating system switches It it Disables deactivates the failing the collapsed state And introduces another state which will be more Selfifications and then There's a question of what happens if you have seven Benny handles of 700 Benny handles. We don't know How many Benny handles there are How do you still feel that you're Benny handle? So of course all these Benny handles. Let's talk about Walter Mitty. Okay, Walter Mitty How do you still know that you are what they are? Gets close to home How does Walter Mitty knows that he's Walter Mitty? So Walter Mitty Knows that I mean has this feeling that he is one has this feeling of one unity of one continuity Because he shares memories And because he shares resources All the self states for example can access the imagination of Walter Mitty All the self states can access his intelligence at his recent his memories his memories. Does he also have hopes? Hopes these are effects. Yes Aspirations, which is which is an award for hopes He can they can they all access the same databases They they share the same databases and this creates a feeling of togetherness. Yes, right a feeling of teamwork kind of yes, and this is the continuity that we are feeling and But This the self states don't share all the memories The self states don't share all the memories. So we there is something called dissociative partition Only some of the memories are shared a big portion 80 percent 90 percent 60 percent But some memories some memories are unique to each self state And they are not they are not shared Now finally Why Why do people why do people fracture? Why do they fragment into self states? Why isn't Jung right?  Maybe he's right. Maybe I'm wrong. I mean why Why do I think that it's much more likely that people are fragmented? Then that they are they start out fragmented Everyone agrees that they start out Basically fragmented More or less because of the different influences they have From the father from the mother from the because they for example at some point They realize that mother is not the same as they are So there's already one object there a break. There is a break a schism and so on so More or less everyone agrees that in the initial few months Of life we are all fragmented somehow. There's a debate how the debate how but This but then the prevailing theories Multiple is that there is a process called synthesis or integration And all these fragments form finally a unitary whole. Okay, and I'm saying No, they don't form a unitary. They form an assemblage A group a troop like a theater troop, you know, yeah of self states ensembler ensemble of self states for life And then the question arises and that's a final question. I mean I can't go into all the my work The question arises. Why? Why there's no integration? Why don't I don't I believe in integration? I don't believe in integration because Of dissociation and because of trauma Life is full of traumas from a very early age Traumas Dissociations where your need when your needs are not met When you're frustrated When you are humiliated when you're shamed Etc etc They create A failure of synthesis and integration They create something which which is called realization deficit so This forces The child to actually maintain self states That each of these self states can cope with an unmet need Or a problem or a frustration or a trauma or The amazing thing what I find amazing is that everyone agrees that The first few years of life are exceedingly traumatic Just separating from mother At age six months to two years It's a process called it was two years in my case. It's it's This process is exceedingly traumatic just this So Everyone agrees that there is trauma massive trauma in early childhood and so on But then they're going to say, okay, there is a normal trauma, but it doesn't matter Which is shocking. Of course it matters It it does not allow A synthesis it does not allow constellation It leaves us broken for life And then we we take these fragments and we use them to cope with reality as best we can So we can say that we are actually as we said earlier. We are an ensemble. We are um a kaleidoscope Of fragments Working together. Yes I that's that's precisely what I believe. I believe we have a series of narratives I believe self states are essentially narratives We have a series of narratives Because in early childhood we learned that we are going to be frustrated. We're going to be traumatized. We're going to suffer pain Many of our needs are not going to be met We've learned this So we said, okay, we're going to create solutions For each of these situations multiple solutions These multiple solutions include what we call ego functions the ability to cope with reality The ability to regulate your sense of self-worth in the face of adversity in the face of conflict The the the ability to test your environment And to get cues from your environment. So There are these solutions these solutions and narrative solutions. This is a story It's a story you're telling yourself about who you are You know And then as the environment fluctuates and changes and flows You use your resources as you as as is self efficacious I think it's defies belief logic and observation To say that people are the same from age six to age 90. This Is mind-boggling. I find this mind-boggling and yet this is what we teach in university and So this might work


I'm not to use talking on the microphone. So if I'm not audible or if I'm too loud, just let, just wave your arms and we'll do something about it. All right? Okay, so let's get started. So what I'm gonna talk about today is something to do with the data culture, right? And in order to say this at all, without data culture, agile is just a process, right? Before we jump in, I'm just gonna spend 30 seconds talking about who I am and what I do, right? Just to set context. I work at Hike Messenger in case you haven't heard about Hike. Hike is a social app where you can connect with people who matter most to you and have fun with them on the app. And also you can consume content and do tons of other stuff, right? So that's Hike. I am a data scientist and I work as part of the analytics and data org within Hike. My team is responsible for a few different things. On number one, it's responsible for making sure that the BI and reporting system is up and running and data is available all the time. And on the other side, we are responsible for making sure that we mine user insight out of all the data that we have about our users and about our product. And then we use that user insight to personalize the experience for our users, right? So that's pretty much what I do. That's my day job, right? Okay, so what are we gonna talk about today? What are we gonna talk about? Building a data culture. First and foremost, what is a data culture? What do we even mean by a data culture, right? And then once we have some common understanding of what a data culture is, what do we do? How do we structurally change an organization to get this data culture thing happening? And what I'm gonna talk about now for the next 40 minutes or so are things that I have learned over the past year, year and a half, working at Hike. This is what I do at Hike, right? This is sort of my primary role at Hike, right? And again, what we're gonna talk about today is not something I read in a book last week or something, right? This is something that I have sort of worked on over the last year and a half, and we've made a lot of mistakes and we've learned a lot of things, right? So hopefully what I'm gonna do today is share at least some of those learnings with you folks, right? I'm gonna try my best at the very least, all right? So what is a data culture, right? So in order to answer that question, what is a data culture, right? Took the easy way out. What I did was just Google for organizational culture, right? And that's what sort of the first result that shows up is this, right? It says organizational culture is a system of shared assumption, values and beliefs, which governs how people behave in organizations, right? These shared values have a strong influence on the people in the organization and dictate how they dress, act and perform their jobs. And then I did the obvious thing. I just changed a few words around and just like control F, just changed a few words around and here's what I call a data culture. Data culture is a system of shared assumptions, values and beliefs, sound familiar, right? Which governs how people interact with and make use of data in organizations, right? These shared values have a strong influence on the people in the organization and dictate how they communicate, take decisions and perform every aspect of their job, right? So this is what in my mind is a data culture, right? And what hopefully we are gonna do over the next few minutes is talk about how can we get this going within a company, right? How can we make all this happen within the company? And Dipeshu just spoke a few minutes before me. He said very, you know, he said it very well. You can't build culture, you can't sort of make culture happen with force, right? You have to foster it and over time, it's a process, over time, you will hopefully see the right signs, the right things happen, right? And that is how you build a culture. So three major points from the previous slide and I wanna elaborate on that just a little bit more. Shared assumptions, right? When we say shared, everybody in the company has to have a common understanding of what we're talking about, right? Shared assumptions. Data should mean the same things to everyone in the company. That's different from saying everybody should draw the same conclusions from the data. It's, everybody should have the same common understanding of what data is, right? This is about creating a level playing field within the company, right? Usually there'll be a few folks in the company who are, you know, very data savvy. And, you know, they'll be able to do a lot more with data. And there are some other folks who are not that data savvy. And maybe their job even doesn't require them to be, but then they will not use data as much as the other folks do, right? Shared assumptions mean that regardless of who you are and what your job role is, you're all playing on a level playing field. You have the same access to data. You understand data the same way and you're able to use data just as freely, regardless of who you are, right? Shared values, everybody in the company places the same amount of importance, right? We're talking about a value system. We're talking about everybody in the company sharing a common understanding of what they should do with data, how they should use data, how they should communicate with data, and so on. Shared beliefs, right? Everybody has this common understanding or common belief in the power of data. And this sort of only comes if you have experienced it yourself over and over and over again, right? You've used data, you've seen the power, and you've done it many times over and over again. That's the only way you'll start believing that this thing that we call data is really powerful, right? Sure, yeah, sure. So I can give an example of what do we mean by shared beliefs. If you go to someone in the company and say we have a lot of data, why don't you start taking this data and making some use of it? There'll be resistance, right? People don't want to try new things. Let's say for like two or three times, you make them do it, right? Somehow or the other. If when they use the data for those two or three times, suddenly magically, they start seeing results. They start seeing that data is making their jobs easier or making them much more effective at what they do. That's what I'm talking about, right? That is a belief, right? Then they will start believing that what they do, they'll be able to do much better if they start using data, right? So here's how we start, you know, a process which will hopefully end up with creating a data culture within your company or building a data culture within a company, right? I strongly believe that there are two fundamental pillars that we need to build within the company and we're gonna talk more about what those pillars are and how we build them and so on. We have to then, once you've built them, focus relentlessly on adoption and when we do both over a period of time, we get vibrant data culture within the company, right? Build those two pillars and we're gonna talk about how we build those pillars and create levers so that you can take hold of those levers, use them like a crank, right? Turn them around over and over again, many times, hundreds of times and if you do these two things, right? You know, six months or a few, I don't know, weeks, months down the line, one day you'll go into your company, into your organization and you'll realize that something magical is happening, right? And that would be a data culture. So what are those two pillars? The first pillar I'm gonna talk about is called data democratization, right? Making sure everybody in the company, one has access to data and two is able to use the data in the right way, at the right times, to draw the right conclusions in the whole decision making process, right? This is at a very high level what data democratization means and the second pillar, experimentation at scale, right? What does experimentation do? Each step forward in the company or in the organization's evolution, the way you take it is by taking, sort of by making small experiments on what the way forward should be and these experiments should ideally help you move forward in the right direction. It should keep you oriented in the right direction all the time, right? And just sort of a quick thought on how do these things sort of connect with agile, right? Like how does this all add up? I believe that these two pillars are actually fundamental for a company to go truly agile, right? Both are needed to de-risk what you do to be able to develop something iteratively for different parts of the company to move independently without sort of coming in each other's way, but at the same time, move forward in the same direction and be in sync, right? So these are the two pillars and we're gonna talk about them in much more detail now. Before we move forward, I'm gonna set up a small sort of 4K study, right? Let's assume that all of us work for this company. We're in a widget sync, right? We are an e-commerce company. We sell widgets online. We have an app and we have a website, right? And it turns out that, you know, many people use widgets and they're vastly popular and they come in different shapes, sizes, colors, whatever, what have you, right? And so very competitive market, we're taking on all the e-commerce giants in the world, right? So that's who we are. Why are we doing this? Many of the concepts that we're gonna talk about in the next few minutes are very hard to relate to in the abstract, right? Having sort of an example to use when we talk about those concepts is hopefully gonna help us make sense of this sort of much faster, right? So we are all part of this company, right? Now, as it turns out, widget sync is an internet-age company, right? So they have like tons of data. They collect a ton of data about everything that you can imagine, right? And we are talking about two kinds of data, data about what happened in the past historical data and also what's happening right now in real time, right? So you have things like inventory data, you have orders data, delivery data, product details data, you name it, we have it, right? And, you know, as it turns out, different components of these data sit in different sort of data stores, right? Some of them sit in, you know, SQL databases, some in some distributed storage, somewhere in the cloud, some of it sort of like, you know, app or web logs data, the volume is so huge it just goes straight into archived storage. Some of this may be marketing data, for example. Might be sitting around in a spreadsheet on someone's computer, right? So all of this data is available, is there within the company. But the problem is it doesn't talk to each other, right? It's sort of different elements of data sitting around within the company. None of it works with each other, right? Now getting data from one source, from one of these elements, might be possible if you know the right person, right? But getting access to a data from across these elements is next to impossible, right? Because the way we talked about just now, we just don't talk to each other at all. These different systems, right? And why don't they, these systems talk to each other? Couple of problems, one is technological, right? You have these sort of different kinds of data stores and you know, they just don't talk to each other. And the other could be organizational, right? What I mean by organizational problems is that as companies grow larger, they grow in silos, right? You have sort of departments or, I don't know, or fiefdoms, if I may call them that, who are very, very protective about what they have, data being one of those, one of those things, right? So if you go to your supply chain management department and say, hey, I want inventory data for the last six months, the first question they're gonna ask you is why? Why do you need it? We can't give that data out to everyone. It's very sensitive, right? Our competitors might find out how we stockpile certain inventors, right? You'll hear a lot of reasons. Basic fact being, organizations are siloed, right? And I saw a couple of smiles around the room, so I think people know what I'm talking about, right? So in the boardroom, right? In the company's leadership, everybody's talking about how we have tons of data and our people, our employees should be making a lot more use of the data, right? That's the conversation always happening within our leadership team. But you know what? If people can't get their hands on the data, you can't really blame them for not being able to use it, right? So this is a key problem. Nobody in our company, Widgetsync, is able to get their hands on data and use it effectively, right? So that's problem number one and we need to solve it somehow. There are deeper problems. Let's look at a couple of facts about our users, about our customers and see what happens, right? Customers who come through from clicking a Facebook ad prefer orange-colored widgets. Showing a discount coupon as soon as a product is added to cart leads to better conversions in tier one cities, right? Now these things, assume that these are true about our customers. How would we, as employees of the company, realize that this is true, right? These are correlations, right? These, you know, people who come in through clicking on a Facebook ad are strongly correlated with people who order orange widgets, right? That's correlation, but how do we figure that out? This is how we figure it out, right? We look at data across different elements, across different departments, right? And if we are able to get access to data from across silos, we have some chance of being able to figure these things out, right? Mind some insight out of it. Here's the second problem. Insights almost always come from overlaying different kinds of data on top of each other. Very, very rarely will you find insights only within your inventory database or within your marketing database. Sure, there's some stuff there, but more often than not, your insights, the most powerful ones are gonna come from sharing data and looking at data across silos, right? So this is problem number two. So what do we do? How do we solve these two problems, right? The answer is building a single unified data platform, right? Now I'm not gonna spend too much time into talking about how we build this or what should be the right architecture for this because this is just too specific to whatever organization you're talking about, right? But just a couple of points. Number one, this is an area of major engineering investment, right? This is not a project that two engineers are gonna sit over a weekend and finish, right? This is something that's gonna take once, maybe one year, right? So the leadership has to really get behind this and as leaders, we have to get behind this and say that we really need to build a single unified data platform. The good news is that even though the whole process is gonna take a year or I don't know how long, we can start small, right? We can build iteratively. We can start with a very, very simple version and scale up over time, right? That's the good news. Now I'm saying this, this is something that we're gonna have to significantly invest in. I've not yet talked about what value we're gonna get out of it, right? We're gonna talk about that in a few minutes. But are we done yet, right? Assume that we build a single unified data platform. You know, you build a tool where someone can go in, maybe a fire up a web browser and access any data within the organization, right? It's possible, right? Let's say we did that. What next? Are we done yet? Not really. What would you have to do to access data from sort of these different data stores, right? This is an example SQL query. I don't even know if it's correct or if it works. The objective of showing this here is that to get data from diverse data sources, you're gonna have to do something like this. And regardless of what pool you use, you might use SQL, you might use something else, right? There might be a few data sort of savvy folks sitting in this room who might say, okay, big deal, we'll do it. But let me promise you, 90% of the people within the organization are going to faint if you tell them that this is what you need to do to get to data, right? Not gonna happen. They're gonna say, please try the next person. I'm not gonna do it. Is it because SQL is hard? No. I can promise you again that if we had to, we could train everybody within the organization to write stuff like this within a couple of weeks, a month at most. It's more to do with fear of the unknown, right? All of us as human beings are resistant to change where we are fearful of what the unknown holds for us. So the first reaction when you see something like this is I don't want to do anything to do. I don't want to have anything to do with this, right? So what next? We build this single unified data platform and data is now available freely to everyone, but nobody can access it, right? What if we build tooling which made this look like this? And again, folks who know SQL will tell you that this is very simple, right? It's not hard to do at all. The point being, we, the architects of the data platform, have to build these ourselves. We have to anticipate what are the different things that people in the company are going to need from the data angle. And we have to pre-build all those tools so that actually fetching the data the last mile, the user who fires up that tool and tries to get data, his or her job is absolutely simple, right? Zero friction. You want them to get to your tool and get at that data within seconds. They should not have to think about how to get the data at all. That should be taken out of the equation, right? So that's the other key learning here. While we're talking about tools, what about dashboards, right? So when you build that single unified data platform, you could also have, you know, dashboards sitting on top of that, right? Which are showing you fancy graphs of how your metrics are doing. Many a time, dashboards work only as reporting tools, they tell you what is happening, right? In this case, you know, you see a drop right at the very end, if you ask the question why has that drop happened, the dashboard doesn't really help you, right? It doesn't tell you why exactly that drop happened, right? So this is a dashboard as a reporting tool. What we should ideally be gunning for is a dashboard as an analytics tool, right? Not only should you be able to see what's happened, you should be able to click on it, interact with it, ask questions and get answers, right? Drill downs, filters, pivots, right? These are the different kinds of functionality that you, you know, would wish to have in a dashboard. So when we're talking about building all this tooling, may it be dashboards, may it be SQL, may it be any other tool? The idea is to take away all friction possible, right? Someone who wants to get at the data should be able to do it very, very quickly, very, very easy, right? What about metrics, right? Most organizations will have sort of a host of different metrics with which they track performance. What about them? If everyone talked about the same metrics differently, they become that much more harder to understand, right? These are just dummy graphs. I just put some dummy data in Excel and I just made different kinds of stuff, right? It's the same data. Imagine you're sitting in a room with, I don't know, your senior leadership and someone shows you a graph and then you have to make sense out of it, right? The first, I don't know, 45 seconds are sort of a mad rush to trying to make sense of what that data is even telling you, right? What is that blue line? What is that red line? What is the metric? What is moving up? What is moving down? So that's a lot of effort, right? And again, it's intimidating, right?  How do you make that happen? The way to make that happen is to standardize one, all metrics, all the important metrics for our organization. We have sort of standard definitions for them. And number two, you standardize the visual representations of them, how they look on the chart. The blue bar always talks about category one. The orange bar always talks about category two. Now, I mean, I'm guessing this sort of sounds like police state, right?  Well, here's the deal, right? If you're talking about everybody in the company having a common understanding of the data, this is vitally important, right? Again, the idea is people who are data savvy or who are comfortable with data should ideally not have an edge over other people in the company when it comes to understanding data. What they do with the data, of course, right? So that's the key point here. Yeah, so build tooling that will make it easy for everyone in the company to access data, standardize everything, standardize all the metrics, and this is what will build shared assumptions, right?  Everyone has equal access to data, understand it in the same way, right? Nobody's at sort of a disadvantage when you're talking about data. Now, you've built all this tooling, right?  How about adoption? Do you think suddenly the ones you've built it, people are gonna start using it? Absolutely not, right? You've taken the first step, which is, you know, you've taken away people's excuse for not using it, but that doesn't mean people are gonna start using it suddenly, right? So imagine at Widgets Inc, we have this design review meeting going on. We're gonna release a brand new app next week and we're gonna review this, right? This is how your meeting has to start. Every day, our app sees 250,000 unique visitors, 48% of them buy Widgets on average, 350 rupees each. Our hypothesis is that our new app design will increase the conversion ratio by 5%. This is a preamble, then you review whatever the design, the new design is. And this is how you end. We feel that the checkout flow optimization and blah, blah, blah will contribute to giving us a 5% increase in conversion rates. So what have we done? We've taken this meeting, which was, you know, gonna show you a design for a new app and you have converted it, you have sort of laid a frame of data around it, right? You have made sure that everybody in the company, if you're able to do this, if you're making sure that everybody in the company is actually leading their conversations with data and ending their conversations with data, right? Every meeting, every document, every presentation should ideally go in this way. I talked about levers, right? I talked about you build a pillar, then you take a lever and you crank it around over and over and over again. This is that lever, right? This is how you take that lever and crank it around again. Every conversation within the company has to lead with data and end with data, right? Now, I talked about we build this whole data platform, the tooling and all that, what's the value, right? We haven't really showed what the true value of this tooling is, right? Now, this is a question which most companies would probably agree is very important for them, right? Who are our most valuable customers, right? And you can sort of change that question to anything else, right? Who are our most reliable suppliers? What are our top selling products, right? You can frame any question that you want to answer in this way, right? There are many ways of actually trying to figure this out. For example, everybody who transacts more than 5,000 rupees a month is your valuable customer. There are other methodologies as well. The problems with these is that these are able to identify your valuable customers after the fact, right? Only after you finish the month are you gonna say, okay, last month, this is the list of my top thousand customers, right? What if we were able to do this before the fact? What if we were able to predict who are going to be our most valuable customers next month? Now, this is a very, very, very simple, basic machine learning model, which will do just that. And the intent of showing this or talking about this today is not to show you that we can do machine learning. Of course, you can. The idea is this is how your tooling is gonna get used, right? Imagine this, I don't know, data scientist, an analyst who's sitting with your company who's gonna build this model, he or she needs access to all this data. Earlier, it used to take, I don't know, three weeks, a month to get their hands on this data. Now, it's gonna take 15 minutes, right? What are we doing here? We're freeing up our team or our people to focus on business problems, not on data problems, on business problems. They don't have to worry about how to get the data. Data is available, freely available. Now, they can get started on the business problems, right? We talked about shared values, right? Everybody in the company shares a common understanding of the importance of data. This is what building shared values means, right? So this was the first pillar, data democratization, right? And I hope I've been able to convey at least some of, you know, at least in some small way, how powerful this pillar is, right? This is about getting everybody on the same page as far as data goes. Now, we're gonna talk about the second pillar, right? Which is experimentation. Experimentation also known as AB testing has been around for a long time, right? I'm guessing people would have at least seen it or heard about it before, if not actually used, right? One of the key philosophies for experimentation and also agile is build, measure, learn, right? You build something, you test it out, you learn something from how you test it out. And if it works, you sort of pursue it, you iterate on it, you improve it and you put it into your product. If it doesn't work, you throw it away, right? Now, the question is how do you make this a repeatable process so that, you know, we talked about that lever and that crank, how can you turn this into a lever and a crank so that you can do this over and over again, right? This actually is sort of an expanded version of build, measure, learn. And again, there's a lot of literature on this online. I'm not talking about something that I've invented or something, this is just about how to operationalize this, right? The main criticism of build, measure, learn is deciding what do you build, right? What should be that next idea that you should try out, right? And since you are anyways gonna run a very, very, very quick and dirty experiment, many a time it turns out that people think it's okay to try out random stuff, right? It doesn't really have to, you know, you don't have to spend too much time researching, you know, what to build, what not to build. Anyways, you're gonna run an experiment for a week and throw it away if it doesn't work. So, you know, go ahead and do whatever you like. This is very sort of dangerous as far as thinking goes. So how do you change that, right? Here's the thing about ideas, right? Everybody has a gut, every gut has a feeling and everybody with a gut feeling thinks they're right, right? And again, there is no way for us to distinguish my gut feeling against yours, right? This is just no way. Is the COO's gut always right? We don't know that, right? For example, widgets in could sell twice as many widgets if they would advertise in colleges. Repeat buying would go through the roof if you send flowers with every delivery. You can think of whatever you want here, right? How do we decide what are the right ideas to even test out, even to try out? Can we build some process around that? Here's one way to do that, right? And I, again, there are a lot of different ways of doing this, this is just one way which I find works very well. You can frame any question you like or any important question for your organization into this, into this format, into this template, right? What are the actions that most widely separate valuable users from others? Refraising that for a different problem, what separates orders that get delivered on time from those that don't, right? What separates teams that retain people from teams that lose people, right? You can sort of take any domain, any problem and sort, and frame it within this template, right? What you do is figure out what are the set of actions, right? This circle here is some actions and the other circle is who are the valuable customers? And again, what, if you change the question, the labels will change, that's all. You find those actions where this intersection, this middle area is the largest as a proportion of the entire set, right? And I'll give you some examples. You take this list of actions which comes out of this analysis and again, this is a database, completely scientific way of doing this, right? You take each action at a time, figure out what that intersection is. You take the ones with the strongest correlations to rank them, right? And then your experimentation framework is just gonna take one idea at a time, try it out, turn the crank, and figure it out if it works or not. If it does, great. Pursue it, iterate on it.  Now, what we're talking about here is taking ideas and taking discretion out of it, right? We are making it almost a formulaic thing, right? Many people would argue that you're taking creativity out of the system, right? You're saying that I can't even be creative about what ideas we try out. That's not actually what we're doing. All we're trying to do is focus creativity on the right parts of the problem, right? Creativity should ideally be focused on the solutions, right? This piece, the ideas from data piece is just trying to identify what are the right problems to solve, right? Solutions are never gonna come from data. Solutions are always going to come from creativity, passion, energy, right? Data is only gonna tell you if your solutions have worked or not, right? So ideas as a process is effectively reorienting your creativity or focusing it on the right part of the problem, which is coming up with solutions. And here's an example, right? Let's say, did not see an out of stock in their first week, sort of came out to be the highest correlated idea, right? Correlating with valuable customers. How do we make this happen? This is where your, this is where the creativity should start flowing, right? This is where you should start thinking about how to make this happen, right? Maybe show false in stock, even when things are out of stock. Don't show the item at all if it's not in stock. I don't know. We could probably brainstorm and think of 20 more things, right? But this is where the creativity comes in. Then what do you do? You sort of figure out what to build. The idea here is you build one thing. The idea again is to test out whether whether this way of solving the problem works, right? Whether this idea holds any potential for the future, what we're not trying to do is find the best possible solution. We can do that in the future, right? We can optimize what we've built as we go along. The first step is even to validate that idea, right? So build only one thing, test it out. I'm just following that ideas, build, code, measure, data, learn, sort of framework. What you do here is you build a very, very simple MVP so that you can test the idea out. And again, this is very specific to water building, so we're not gonna spend too much time here. Now we've come to the major phase. This is sort of the most interesting phase when we talk about experimentation, right? It's very important to note that success of that idea or that hypothesis is very different from success of the experiment, right? The idea is successful if it actually works in practice. If not, you know, if customers don't see out of stock in their first week, do they spend more on your app? Is that true or not? That's the success of the idea. Whether your experiment is actually able to discover what reality is, that's the success of the experiment. When we have made a process around what ideas we even prioritize, we have given ourselves the ability to be dispassionate about whether the idea works or not. It's not come from my gut or yours, right? So we don't care if it doesn't work. We can always, we have a long list, right? We can go to the next one. There is no pressure that, you know, the CEO really believes in this idea. So let's make sure that the experiment tells us that, right? No pressure at all. You just take all that out of the equation, right? So again, success of the idea and success of the experiment are two clearly distinct things. And this is something that we have to be very cautious about when we're running experiment. What metrics to measure? How to figure out whether your experiment proves that the idea works or not? Without going to too much detail, I would recommend that we have at least one or two people in the company who understand the statistics theory behind this and who are able to coach other people about it. Who are able to sort of lay the foundations of understanding this, right? What you do ideally when you're running experiments is you frame a very, very, very tight hypothesis. And then you launch the experiment. You want to prove or disprove this hypothesis, right? Ensuring 100% availability during a new user's first week causes percent of new users placing at least three orders in their first two weeks to go up from 20 to 23%. Now this reads like, you know, I don't know what all's happening here. We can break this down. This is the metric that we're gonna measure. This is the metric that's gonna tell us whether our idea is successful or not. This is the effect that our idea is going to have on that metric. And if you're able to achieve this, we say that the idea is successful, right? Sample size, again, leaving aside the stats theory, leaving aside what exactly, you know, how do you go about calculating sample sizes? There are online free calculators which you can use, right? It's not hard to do. What's the intuition, right? Across the top is that hypothesis actually true in practice, in reality, right? We don't know that. There's no way for us to know that, right? That's why we're running the experiment to find out whether this thing actually works. On the left, what does the experiment tell us? Is the hypothesis true? Yes or no? Ideally, you would want to land on one of those green check marks, right? If you do this, your experiment was successful. You either discovered that the idea works or it doesn't work, and you found out what was actually true in reality, right? If you fall on one of those red crosses, it means that you've made a mistake. Your experiment has made a mistake. It either says the idea is great when the idea is actually a dud, or it says that the idea is a dud when actually your idea is great, right? At a very, very sort of intuitive level, sample size is the lever, which is going to give you more confidence that you land on one of those green check marks. The higher your sample size, the more confident you will be of landing on that green check mark, right? Now, obviously, that doesn't mean we can use a huge sample. There are other constraints on how big a sample you can use. You ideally don't want to expose a new feature or a feature under trial, or only an MVP product to a lot of people, many a time, right? So there is a natural tension there. Your experiment craves more users, more higher sample size, whereas there are other product-related sort of constraints which are pushing down the sample size, right? Now, how do you run the experiment, right? It comes back to tooling. And when we talked about data democratization, we talked about making it super simple. Take away the excuse that I can't use this. I can't do this. It's too hard for me, right? Your tooling should ideally make it as simple as possible. You can either build this tooling in-house. You can buy it. There are options out there. But the idea is that you should have a good performance tooling in place so that running experiments is sort of a breeze, right? Yeah? Also, the aspect of duration. Sure. How long would you run an experiment? If you cut it through short, you could land in the direction of the top. Absolutely. So an example of that would be that your metric says that sales would go up by 5%. Right? And you would start to take experiment today and you know the metric. Sorry, I'm sorry. So the question was, what duration should you run an experiment for? It might just happen that if you'll run it for longer, you might actually end up landing on one of those red crosses, right? And the example I was giving was that, for example, the metric could be sales, right? Your sales will go up by 5% or something. You would start the experiment today, you check the metric tomorrow, and it's up by 8% or something. You check back after a week, maybe it's back down to 3%, right? And then in that case, you have made a mistake, right? How long should you run an experiment? No right answer. Right? It is something that you have to take a call on based on what the metric is and based on how fast changing is your environment. Right?  If you're talking about an established product, a small change in say sales or revenues, you know, it will stick. It will change over time. It's not volatile, right? But if you're talking about a volatile metric or a volatile product, it could change. What you see today may not be true tomorrow, might not be true the next week, right? So it's a little bit of an art rather than a science. But so long as you pick the right metric and you have enough knowledge about how that metric moves based on your experience in the past, you can make some judgment about it. While you don't have that much experience about the metric, it's always a good idea to run an experiment for as long as possible. Obviously, you can't say I'm running an experiment for six months, right? You can't. So start with at least a week or two, right? Until you get comfortable with what metrics you're using, start with a week or two and then maybe you can run them much tighter than that, right? Does that answer your question? Right? So we talked about tooling. You can either build it or buy it, but it's important to have the right tooling. And again, once the experiment is done, this is sort of mechanical, right? You pull out data, you compare the metric across your control and test sets. You do check for statistical significance. And in sort of a one-liner, what the statistical significance means, it's how confident are you that the change in metrics that you're seeing across these two sets is actually because of that product or feature that you've built and not just due to random chance, right? And then as sort of the last step of your experimentation process, you do what you would do at the end of a sprint, right? You do a retro. You figure out how the experiment went, what you learned from it, not what you learned from it, only from that feature or idea perspective, but in terms of the whole process of experimentation, right? When you run that next experiment, you want to do better, right? And again, we talked about that lever, about that crank which you take and you sort of keep turning it over and over again. So key takeaway is make it super simple, to run experiments, make the idea stage completely process-driven, right? Take away discretion from this stage. And again, just a tip, if you can, align experiment cycles to sprint cycles. Ideally, if you have, I don't know, if you have a two-exprint, you run an experiment in the two-exprint, you are analyzing the experiment that you ran in the previous sprint and you're planning for the experiment right? If you're able to align these two cycles, I guess that would make it much more effective, right? Now key takeaway here was that lever, right? Experimentation is just those five or six steps, right? The lever, you hang on to it, you turn the crank over and over again so that people start seeing value, right? If people, the first time around, nobody will want to run an experiment. So the first three, four, five times, right? Adoption, more often than not, comes top-down, right? You have to tell them, you have to use it. Trust me, you have to use it. You probably have to do that a few times. Over time, they start, they're gonna start seeing value, right? You will reach a stage where people will have experienced the magic, right? How experiments run, how the results pop back out and tell you something about reality. They'll experience that magic often enough that they'll start believing in it, right? And that's how you build shared belief. In conclusion, these are two pillars, data democratization, making sure everybody's on a level playing field as far as data is concerned and experimentation, right? Showing people, everybody in the organization, the power of data, right? And doing it over and over and over again. If you do this, right? Over time, over maybe six months, maybe a year, right? We will find that we've built up our data culture. That's it. So I have, left my email idea at the end. So if anybody has feedback or any ideas around what we talked about, feel free to hit me up. Any questions before we end? Sure. So I have not... Obviously, that's not practically possible. If some parts of your data need to be guarded, you can put access control on that. But as far as possible, as far as practicable, the more access people have to data, the more trust you build in the system and the better you are able to build these three layers, the shared assumptions, values and views. If your people get the idea that you're hiding something from me, they start trusting you less, right? I mean, that's just human nature. So yes, there are times when you have to sort of keep certain sensitive data out of the public domain, out of everybody else's eyes. But I would advise against doing that too much. Conclusion? Sure. Have I seen this in experience that data showed us something and actually, that was not true. That turned out to be false many a time. Many a time. Almost every other time, absolutely. And that's how you learn and get better. That's how you learn... So I talked about... Which is experience, exactly. This is like blood, sweat, tears. You make a mistake, you learn from it. So that's what experience teaches you how to read. So how to deal with it simple and so stop them from doing it. But you can't. Yeah. So, I mean, are there any external influences on the data which are actually causing your experiment reads or your data reads to be wrong? Is it possible? Of course it's possible. How do you guard against it? You have to build different guard rails to make sure that doesn't happen. Your experimentation or data processes are not going to tell you that. Unless there is some huge anomaly that you're seeing, maybe once in a while. More often than not, your security is something that you have to build out completely separately. Possibly. Anything else? There are a few. I mean, I've evaluated a couple and maybe one-on-one we can talk about what those are. So one sort of quick tip there is building something in-house is always better than whatever tool you get. But obviously not always can you do that. You might have other constraints on why you want to buy something. Sorry, just one last question. Alright, we'll just talk offline. Thank you folks.