Mae'r reliant dda, mae'r twel, a wrth gwrs gweithio pwysgwyr, oherwydd mae'n kyfil yn bryd ystod yn gyfyrdd, ac mae'r chyfnod, hwnnw, o wjangall pan â gefnogaeth hwnnw wedi'i bwysig o'r twel, byddai'n ffordd ychydig ar y cyfry o gyflym, wrth gwrs gawr o bwrth gwrs gwrsglunodd a'r egwycoff iawn, mae'n bwysig o'r twel, ac mae wir yn ei mwy yw cerddion mewn. Mae'r twel eich gweithio, mae'n bwysig o'r twel i'r twel. But it's unlikely. My name is Caroline. It's possible, but I'm actually Patrick Dunn. Caroline, unfortunately, is ill, so I've stepped in at the last moment to take a presentation. I know the presentation quite well. I've gone through it. And interestingly, I wonder if Caroline, I wonder if she's watching now, I actually disagree with one small part of it. So, you know, a lot of learning is about disagreement and contradiction. So that's probably a good thing. So just about me, I've been doing learning technology projects for about 40 years, actually, as of next month. I did my first one in 1983, and I do a lot of academic work as well. I've got three master's degrees as well. So this is what we're talking about. Just briefly about UFI. We support the development of digital technologies that help us obtain vocational skills. We're about vocational skills. OK, and as a charity, we have a particular focus on this kind of learner, those who haven't done particularly well or been served particularly well by mainstream education. Now, of course, defining mainstream education is a pretty important thing here. And I kind of struggled with it this morning. So I just came up with this phrase, didn't do particularly well at school. That kind of thing or didn't go to school. And if you look at the stats, they're pretty shocking, actually. I was looking at the literacy stats this morning. There's about seven million people in the UK classed as, I think, it's illiterate or pure or with difficulty with literacy age. I think eight or nine reading age, something like that. So you think about wandering around your average shopping centre, the number of people that walk in pasture that can't actually read the signs on the shops or the packaging, there's a lot, actually. So they're the folks that have not been particularly served well by the traditional mainstream. OK, so the next bit is a bit of audience interaction. Let me just see if I've got it right. Yeah. And as we go, we've got a select audience here today, ladies and gentlemen. And so what I was going to ask was, do you or do any of you have a sense that that there was a particular aspect of the mainstream education system that didn't serve you well? Do you know anyone? I need just to give you an example. My my partner, do you know what I promised? I wouldn't mention her specifically if I did. I'm going to go somewhere else. OK, now I'm going to go somewhere else completely as well. A friend of my son's has got a real hang up about maths. He'll never get maths GCSE. It's it. He just he just can't do it. And he has a sense, his parents keep telling him that it's going to be a hindrance in his life. And I think this is the wrong, you know, it's a wrong bit of parenting. But also it's one of these things that you do come across. And a number of adults have a hang up about a particular subject. It's often maths. And it can be all sorts of things. You know, the I'm just going to get out the way briefly so that we can all see.   I was very lucky. Didn't come from a privileged background, but I was fine all the way through. But I know loads of people who just got one of those, you know. And if you know someone, just reflect on it, if folks online the same. And there can be all sorts of reasons for those categories. I've got here are situational, institutional or dispositional. I really like those. So it could be something to do with the situation you're in. It could be simply who you are, your disposition, or it could be the institutional situation that you're in. OK, so there's all sorts of reasons. But give me, I do have to go through these notes because this is a great presentation, but Caroline wrote the script. So let's have a look at the human element, because what we're saying here and essentially what we're talking about here is design, designing for those. Whom the educational, the mainstream education system has not done a fantastic job and how we can design for learning technologies to help them out. And I really like this diagram as a a series of zones, a series of concentric circles that puts the learner right at the centre. Now, the point here is because of course you always supposed to put the learner at the centre, aren't you? That's what you're supposed to do when you're designing learning experiences. You're supposed to say we put the learner at the centre. But when you have learners, students who are from for whatever reason, not fitting in, not well served, there are some special conditions here. So, for example, the learner may be particularly they may not be very good at expressing their needs, articulating what they need and what they want and so on, their direction and where they want to go. It's a different perspective that's required. Now, all these other folks, due to support peers, experts, and they'll be much more articulate, much more articulate and much more able to to manage the situation. But putting the learner at the core here can be tough, right? And so it requires a particularly astute and occasionally forensic approach to design, which I'll talk about to make sure that they are well served. So having had a look at who the sort of the human, the human bit, now let's look at the design process. It's worth saying that my my sort of passion in life is is designed. So I'm really pleased that Caroline wrote this bit in. I worked in a couple of design agencies. I was so obsessed by it. And what we've got here is a fairly well known cartoon that says and this isn't particularly about learning design. This is about design in general. It says, OK, let's say we're designing something from a customer for a customer. The customer wanted a tyre on the piece of rope hanging on the tree. Now, the client didn't describe that right. So they sort of got the hanging bit and the rope bit right. But actually, from their point of view, it looked like a swing. The engineer thought, ah, I'm going to have two bits of rope on a vertical part of the trunk. And then the manufacturer did what the engineer said. And so you got this chain of misunderstanding and a chain of misunderstanding is extremely dangerous and damaging where you've got a learner who hasn't been served well anyway because their experience is not going to be great. So this is what it kind of looks like. The learner says, how do I know what I need? You know, I know what I don't like. I don't like what I've usually been told and how I've usually been educated. The well-meaning professional wanders around shows and conferences like this and says, do you know what? I heard this cool thing, the sales guy said, but why don't you use Blidyblad technology or design solution? The project manager says, OK, great, we'll do that. I've joined up a budget and the IT person says, do you know what? VR is great. No one told me not to use VR. I'm doing a session partly based around VR this afternoon. So we have a chain of problems here. Now, the key thing here is we must talk about impact. And this is something that we at UFI are very driven by. And I quite like this phrase. It says, until your learners learn something, I don't like the spelling I've learnt, though. You have no evidence that your design and technology are effective or have impact, OK? Until your learner has changed. Again, I talk a lot about that this afternoon. Until your learner has changed, you haven't got any evidence that you've had any impact. All right, so that's what we have to talk about. Our problem at the moment is that technology is advancing so rapidly and so unpredictably that it's hard to track where we are. And we also, we live in an unstable society, politically, economically. And I have a slight hang up with the word transition. We're not in transition because we're changing forever now, right? We're not going from one state to another. We're changing forever. And in a situation like this, technology is a subset of the second point, isn't it? The unstable society, the cultural change is a function of technology, but it's also a subset. OK, so how do we? Actually, this is the bit I slightly disagree with. Caroline's got this thing here that says, right. Her perspective is that humans don't like constant change, right? So our problem is that if we're designing for people who are badly served, what we'd like to do is to say, this is what we're familiar with. This is how we plan for the future. And here's the structure whereby we can design for them. OK, now I actually think human beings on a spectrum here of change. I think we've got another audience person coming in. Do you come in? Hi there. Yeah, no worries at all. I'll just do a quick recap. I will do a mega quick recap. So what we're talking about is that we're talking about those with. It wouldn't be particularly well served by the mainstream, putting them at the centre of the design process. We talked about a rather flawed design process in which assumptions drive poor design and the fact that those badly served by the mainstream education system need a particularly astute and almost forensic approach design. And now I've just been talking about an unstable world in which everything changes. And what I was just saying was that actually, I think in terms of human response to change, you've got a spectrum. And at one end, you've got folks that absolutely dread it. And at the other, you've got people like me who love it. Right. And so we have to. And the thing about innovation and change is you need a mix of people, right? If you've got loads of people like me, we mess things up. If you've got loads of people who only like stable states, you don't innovate. So you need a mix of people to design a process. OK. So we go on to design thinking. Now you may or may not be familiar with these. There's the left hand one is called the double diamond. Interestingly, it's got its 20th birthday this year. It was developed 20 years ago by the design council. It's a very simple process. I think, to be honest, it's a process from a slightly different time, but as a way of thinking, it's useful. And what it says is there was an academic in the 90s, and I've completely forgotten his name, who who came up with this concept that design is about the journey out in the journey in the journey out in the journey in. So what you do is you diverge, you diverge from the point of a challenge. And then you and then you examine the nature of that challenge and then you converge towards an understanding of what you're dealing with. OK. You then say, right, what are we going to deliver? And so you go out and in again. And I think this is this is great as a principle. I think it's it's it's symptomatic of a time where things would be more stable. You had to see what I mean because you diverge from a point by the time you got to the convergence again, everything's changed, right, which is why I like the cyclical one. Yeah. So the idea is you whiz round the circle, you go, right, what do we understand? I empathise with my problem, I define it, I iterate. I'm deliberately speaking very quickly because often this cycle is very quick. I ideate, I prototype, I test, I implement, all right. And then you spin round again and you spin round again. Yeah, until you get to the point where you've tested, you prototype, you've messed about, frankly, this can be very messy, which is why you need people who are not entirely stable state people until you got to a point where your understanding is developed and you've experimented and at that point you can implement. OK. So the key thing, though, is design thinking. You've got to be aware of how you're thinking about design, particularly, and forgive me if I'm repeating myself, particularly where you're dealing with people who have not been well served and for whom the design process has to be astute, almost to the point of being forensic. So approaches, you start by identifying the problem you're trying to fix. And I want to come back to that in a second, particularly for the context of this conference. How do you involve the learner in the design process and then how do you remain flexible and responsive? And I've got a sort of question 3A that popped into my head this morning, which is how do you keep an open mind? Yeah, it's not just about being flexible and responsive because you can be respectful, flexible and responsive, but not have a particularly open mind. But that's a little bit contradictory. But anyway, I want to go back to the first one, though. What's the problem you're trying to fix? So. Who in this room would describe themselves as involved in H.E.? Ah, that's interesting. So do you differ all three? OK, brilliant. So hands up online audience. How many of you are involved in H.E.? Oh, OK. So the point here is, do you identify within higher education the concept of a problem? Do you see what I mean?  Well, of course, students are often a problem, but not in the way we're talking about, OK? So in the vocational sphere where we work, we often start with a very, you know, a profound problem. Don't we? You know, we have people. No, I'm going to disappear into detail. I'll come back to that later. In H.E., you don't tend to identify, don't to start with a problem, OK? So we may have to have another bit of a vocabulary, another term for that, which is potentially a starting state. It's not a problem, but it's a starting point, isn't it? So if you're trying to and I'm deliberately start I'm deliberately going to use a subject which is not vocational and I hope I'm not stumbling or offending anyone here philosophy. Yeah, do you start a course on philosophy by thinking, oh, my goodness, me, we got a problem here? No, you don't. You say, here's a starting point. I have a bunch of folks that are just starting a qualification and at the end they will be different. So in this case, it's not problem solution. It's start point, end point. I just thought I'd just sort of talk a little bit about that. So what's the problem you're trying to fix? Or what's your starting point? So the thing here is that the point at which you start. Has to be defined by a number of stakeholders. We've got the same map again there. You must make sure that your understanding of and you're listening to the voice of the learner is responsive to the various stakeholders. And it can be quite tricky to get to the voice of the learner. There's no point in building solution if you're trying to fix the wrong thing, if you're not actually listening to the learner astutually enough. You will often find that some of the other stakeholders have very different agendas. So, for example, they might start by saying, well, this is all interesting, but it's too expensive. Right, that's a very important thing to include in the in the in the debate, in the investigation. But the comparison of viewpoints, the second bullet down is extremely important. The checking of assumptions is extremely important. The reaching of consensus can be profoundly difficult given the diversity of viewpoints. And so to involve the learner in the design process and remain flexible. You have I really like this diagram. I wish I'd come up with it. You have this concept of this mindset and a toolkit. And I think they're both really important. The mindset involves things like checking assumptions. I talk about that a lot. I think again, I'm going to put my foot in it here. I think those of us who've done who are inverted commas educated, I think we make a lot of assumptions about things, don't we? You know, it's much easier to be fixed in your assumptions if you've studied a lot. You really thought about things. I have put my foot in it, haven't I? Anyway, mindset, building trust, empathising. Find the motivation. Find the motivation of the learner. What do they need? OK, so there's various aspects of mindset. And again, I really like the tools on the right. Personas, we're sure personas are the characterisations of people. Categorisations. We did a really interesting piece of research with the RSA in which we looked at why adults don't kind of fill their lives full of learning. Why don't adults keep learning? OK, and we identified. I think it was six or was it five personas Myron, I can't remember. I think it's six personas and we looked at the characteristic of each of these types of individual and what stops them from learning. And they range from, you know, frankly, your Amazon driver who doesn't have time and doesn't believe they have the slightest chance of learning anything. And I think we I think we call that the functional learner. And then there was the sort of I'm not going to recall this very well, but there are learners around who simply they would like to, but they don't believe that they're a learning type, you know, they don't believe that the kind of person and then the whole bunch of things. So personas is a great tool. Learner journey mapping, I like that one. Using agile approaches familiar with those. Again, it's very similar to the design cycle, the design thinking cycle that we had a couple of slides ago. And so iterative agile processes excuse me. So just a few examples of our work. The the key the key thing here is that each of these. Deal with I'm trying to avoid the word underprivileged with with these learners who haven't done well in inverted commas, right? And they take really very different approaches. The one I know best is First Step Trust.  And the reason it does that is because the people they deal with with their really dreadful starting points totally believe they can't learn. They can't learn anything, right? And it gives them a process whereby they develop self-esteem and they say to themselves, do you know what? If I can learn to stick a tyre on a car, actually, that's quite complicated and it's a bit risky and it's effortful. Maybe I can learn something else, right? So it's a it's partly a virtual reality project. But one of my real sort of height, one of my one of my insights, one of the light bulbs switched on for me, was in a way it's not just about the virtual reality. It's about that point where someone who not long ago I'm not going to go into details with, you know, was living a very dangerous and impoverished life. Someone puts a headset on them and the world they're seeing is damn slightly better than the one they live in, right? And they can do something in it. They have a functional activity they can actually achieve. And they go through a process and they think, let me look if I can do this and someone cares for me enough to do this, then I can learn other stuff as well, right? And that is profoundly putting the learner at the centre of the design process, thinking about what their needs are. They did a huge amount of research and conversation with these people. They got to know them really well and it and it works, you know? Now, the other projects I know less well. I think that the neuro care one is interesting. The whole development process was it's for it's for care workers. The whole development process was built around conversations, focus groups, discussions with with the care workers and the solution they came up with was very human centred videos, just examples of how you should do the job. So it's not the most high-tech of solutions, but it's very much focused and designed around what they discovered about the learners who had not been at all well treated. Maureen, by the way, if you do want to chip in and, you know, just sort of build on my understanding of of these projects, that's that that'd be a good idea. We have maths kitchen. Talked about maths at the start with, and I didn't mention my partner. So. Teachers make a lot of assumptions about levels of numeracy and. There's a kind of, I don't know, there's sometimes a tacit assumption an inner voice with a number of people that just said, I don't like maths.  And so what the what the designers here did was they kind of unpicked this whole this whole assumption, this self-perception that maths wasn't wasn't for them and developed some kind of assessment, adaptive assessment, which allowed people to succeed little by little to the point where the person taking the assessment thought, you know what, I can do some of this. And as they were doing this assessment, it actually measured where they were so that when they started the sort of the learning bit, they knew where to start and they were confident they could start. So that is. The end of the main bit of the presentation. And I've been slightly surprised by an extra slide there. Do excuse me. Ah, here we go. So. Learning technologies can help. And they've got more chance of creating impact when the design process is problem and learner focus. I kept saying that. I really like the mindset and toolbox slide. I will obsess endlessly about agility and iteration. You know, when I talk about people that need a stable state, I frankly don't think they're very good at being agile and iterative. All right. This is why you need a mix in a design team. You need a mix of stable state and agitators to start to you really do. And I really like the honesty, open and collaborative approach. You remember that the circle diagram with the heart in the middle. There are lots of stakeholders you've got to be honest. You've got to be open. You've got to be clear about what your position is and you've got to collaborate. Any questions or comments or thoughts. And thank you again for Caroline, who I hope is watching for a fantastic presentation. And it's been a real privilege to be her deputy, as it were. One thing I was thinking about, I love that. Can we go back to that diagram? You know the one I mean. It's that perfect, that's the one. Can it such a teeny word, but it can be so key to the process? Would you like to talk about trust and gaining learner's trust to actually take part in that process? Cos it's not an easy thing. Is it getting honest feedback from learners about what they want? When sometimes they can't visualise it, or in my experience, they're just like, oh heavens, why are they asking me? On the back foot, how do we go about building that trusting relationship? What do you think? OK, so the question was, how do we gain trust? How do we build trust with these people? I think often you've got, the teacher is a great gatekeeper.  And then you immediately clam up or give the answers that they think person wants to hear. So I think you've actually maybe got student counsellors, yeah, yeah, certainly having a familiar person, but maybe even smaller focus groups. I don't know, but it certainly can be difficult. Because I've certainly got a bit of a pushback. Why are they asking me and teacher as expert? And does this mean you don't know what you're doing if you're asking me what you should do instead? Do you know it's interesting when you said a trusted person, what occurred to me? Because of course, the reason I batted it back to you was because I needed time to think. So my experience with First Step Trust is a remarkable organisation. What they have done to gain trust is they found one or two people who represent them. So you find one person who's had a dreadful life and you gain their trust. You just find one and you work with them and it doesn't matter how long you take, you work with them. That person can then gain the trust of everyone else. Do you see what I mean? You don't want somebody like me to go in and say, hey, man, what's your problem, right? What you want, cheers, thanks for coming. What you want is to cultivate an ambassador, don't you? And that works. That works, yeah. And something that Ronnie Wilson remarked about. Ronnie Wilson at MBE, First Step Trust, amazing bloke. Something he's done is he has worked with particular individuals sometimes, I think it might be years to gain their trust and you get this thing, people like me, all of a sudden, and they advocate and he's done that brilliantly. So I think that's one way of doing it. I do like the idea of focus groups, but I wouldn't even call them focus groups. I just say group chats, as in face-to-face group chats. That works. Otherwise, any other thoughts on gaining trust? One of these that we mentioned, all of these things take time and I think people don't allocate that time that they're going to be able to move and shut down. Yeah, Linja said parachute in. You don't want to parachute in. That's for sure. That's for sure. I said burgers, possibly in poppy. Burgers in poppy. Yeah. Is it again? Pizza parties. Pizza parties, yeah, yeah, yeah, yeah. OK. Yeah. If you're a student at any level, you're already going to have a... If you feel that you've been failed and you're struggling with learning, you're already going to have a bit of a distrust like this might not be for me.  So it does require like educators, teachers, whoever is involved in that process to be willing to dedicate the time and showcase no, I'm here to help. Yeah, absolutely, yeah. I think that's a very big one. Yeah, yeah. Absolutely, thank you. Yeah, and I'll just chuck this in. Dr. Seena Donovan is now at Edgewood University. Her doctoral thesis and her continuing work is on the bond of trust between the educators and students. Oh, it's interesting. She's a fantastic person to read on now. OK. I trust that we'll be able to assess it. Now, I'm just going to... Thank you so much. There's a new one person plus us here to the board. Thank you, thank you very much. And I'm also, we are also promoting the... What's this called, Maureen? It's... I know it's... Oh, it's the Voktec Future Skills Award 2023, the Voktec Future Skills Award 2023. If you could make one change to the skill system to get more adults learning, what would it be? What would it be? There is 5,000 quid in it for you, yeah? It's worth thinking about, isn't it? And it's a good question, a great question. Yeah, what would you do to get more adults learning? Yeah. And what I like about it, it's a very open question, isn't it? Learning what? Learning when? Learning how? How are we going to get more people involved? So, thank you very much. Thank you very much, folks online. I think that is probably it. You can get a copy of these slides. Do sign up to our newsletter. And I think that's it. Thank you very much for coming. Thank you very much for viewing. Thank you.  No, shall we do a compressed version? If you could just start from the beginning, just do that. I thought so. Each session has a little half hour of work, but then this one is on its own on a half hour. So it was a little bit confusing. Yes, it needed work. You're not first people to say this, so when you do it, what will feed it back to the creative input? I'm going to do a five-minute version, because you could do the first five or 10 minutes, couldn't you? OK, here we go again. This is mainstream education version 2.0. So, I'm not going to do the who we are with the UFI Object Trust. OK, I'll say a little bit. We do three things. We issue grants. We advocate through partnerships. We're a partner of Alt. We invest money in ventures. What I talked about originally, first time through, was what do we mean by having not done very well through mainstream education? If you just want a sort of paraphrase, you didn't do very well at school. The mainstream didn't serve you very well. It could be for many, many reasons. It could be any of these. So, in terms of putting the human at the heart, what do we mean? Well, it's all very well saying, look, we're going to put the human at the heart. At UFI, we talk endlessly about user-focused thinking, about user-focused design. The problem, the challenge with those who've not been served well, is that they're probably not able to articulate what they want or need. They're probably really bad at that. So you have to have a particularly astute, insightful, sensitive, almost forensic approach to putting them there at the centre. And not just listening to tutor support experts, peers and so on, but trying to get to them and gain their trust, which is why Lin, I think, introduced that. We talked about the design process and this kind of funny, but rather sad, bitterly sad representation of poor design. What the customer wants. I want one of them. The client describes it wrongly. So the engineer designs it, you know, misinterpets what they say and they design that. And then, of course, the manufacturer makes that, right? So the gap between what the, in this case, the learner, the student wants and what's delivered, the gap can be enormous. Over coffee or whatever, I can tell you some. I worked in the design business for quite a while. Some absolutely hilarious multi-million pound failures because we did that. I was involved in the 15 million pound failure. And this is what, you know, the learner wasn't me, by the way. The learner says, how do I know what I need? The professional says, well, I saw this at a trade fair. The project manager says, fantastic. I'm glad you saw that trade fair. I've got a budget and a plan. And the IT person says, oh, I love VR. Nobody told me that I shouldn't do VR. We talked about the sort of chaotic advance of life generally and technology as both as a subset and a subset of that. We then talked about design thinking principles. And we talked about the very well established design council model called the double diamond, which is basically based on an approach called journey at journey in. In other words, you diverge from a challenge or a problem. You converge to an understanding of it and then you make something. This one is a more iterative, more cyclical view of design thinking. Of course, the reason I'm talking about design thinking is because the approach to design thinking, when you're dealing with people who have been badly served, has to be particularly sensitive. So we talked about this kind of iteration model, understanding exploration, materialized understanding and the fact that it is very iterative and you might have to go around a few times. Design thinking approaches, you need to identify the problem you're trying to fix. How do you involve the learner? And how do you remain flexible and responsive? Best slide of the day in any of the presentations in this conference. I didn't do this. I didn't construct this, by the way. I'm sorry for that. I'm actually standing in for someone else. So I can say this is brilliant without being arrogant. So we talked about mindset and these kind of things in the mindset that are very important. I'm particularly a light on building trust, thank you, Lyn. And not making assumptions. And then we talked a bit about personas, impact measures, agile approaches. So the idea of tools and ways of thinking. And then I gave some examples and I endlessly talked about the wonderful first step trust who deal with very disadvantaged, deprived individuals and given them the chance to start. That's it. They know their people and they give them a very first simple experience to get them started and encourage them how to learn. And that's it.


Hi, I am Subashish and it is a pleasure to be here for my talk, which really to me is a brief field guide to this very vast discipline that causality and causal inferences. So, you see causal questions have fascinated and puzzled us humans in equal measures for maybe over two millennias and beyond. Right from initial work done by Grand Rondmann Aristotle to in some 300 plus BC in ancient Greece to work happening at the frontiers of statistics and machine learning as we speak.  By the way, in case you are wondering why do we have this you know curious cat here, I promise to get to that in the next couple of slides. Quick question to get started with the session, hopefully not too obtrusive for the audience here. How many here are really fond of chocolates, a quick raise of hands? All right, we are all in good company in that case. So, it will be for as very interesting for you, it was very interesting for me at least to note that chocolate consumption has this very strong interlinkages with well, Nobel Prize winning chances beat that right and this is basis a study that came out in the New England Journal of Medicine in 2012. What they really do is they plot the chocolate consumption to the x axis and this is by 22 countries and they plot Nobel Prize winners are 10 million to the y and you note that it almost seems that as chocolate consumption increases, Nobel Prize winners also increase it is almost a linear pattern with a correlation coefficient of 0.8 jokes apart that is pretty strong as a linear correlation right. Now, how many in this room really strongly believe that chocolate consumption can incrementally influence your chances of Nobel Prize winning? That few okay, we have a bunch full of skeptics here good for science, but why not? Maybe chocolate consumption gets my creative juices flowing gets me intellectually stimulated, but will not not convincing enough okay, it seems a skeptical audience again. So, if there is a fallacy lurking behind the scenes right, what do you think is happening? Could anybody want to take a crack on that? Shea, please, please sir, okay, okay, very nice, very nice. All right, we have one shot here and go on. Okay, so what is perhaps really happening is there is a bunch of factors that could affect both of these variables. One obvious one is the economic state of the nation concern. So, more well off countries, citizens there might tend to partake in consumption of more luxury goods like chocolates. They also tend to have better infrastructure investment into education, investment to scientific R&D which may produce more Nobel laureates and you know hence this lurking factor influencing both gives this very interesting strange results, but sorry to disappoint the chocoholics consuming more chocolate won't get you a Nobel Prize. It's probably correlated, it's not causation here. Now this in fact is one major reason, a lurking third factor that affects two variables that leads to such cases of what we call as spewier's correlation and I'm sure you remember your stats 101 professor telling you that correlation is not always equal to causation and you see that is one of the reasons why you get to see such iffy results. Well, if correlation is not equal to causation then what is causation and how do we get to causal effects. Now I did not want to come up with a stiff academic definition of sort, but rather wanted to motivate that using a couple of common causal questions and I'll focus on one here. Now notice what it's trying to get to, it's trying to answer what is the effect of an intervention, the introduction of a mobile app on a metric of interest which is churn for an e-tailer and this is the general anatomy of any causal question. You have an intervention or a treatment as we would call it in causal analysis and you're trying to find out the effect of that on an end metric of interest. If you notice the questions all along that's the generic framework that they always follow. The other thing and that's slightly interded so forgive me for playing the devil's advocate here. You'd notice that these questions come from various walks of life, you have questions from public policy, from marketing, medical sciences, epidemiology, economics and so on and that is because causal questions are really ubiquitous and I'm joking if you scrutinize carefully causal questions are lurking in nook and crannies of your life and then burst out at you. So wrapping our heads around causal thinking is effectively beyond, it will help you beyond this analytics in your daily life as well but in terms of analytics why are we really concerned about causality and causal inference, right? This is something that many of you may be familiar with, this is standard analytics maturity model that Gardner came out with and it shows the various complexities of different types of analysis and the value that they can unlock. So when you start over they do something very basic like descriptive analytics and so on and working solely on correlation actually will do just fine even up to the point where you work with predictive analytics. You see standard machine learning algorithms do not differentiate between causation and correlation yet you're getting fantastic results, right? The point being that when you're moving to the final, final frontier of prescriptive analytics where you're really trying to prescribe what would be, what can be prescribed to move the business needle, what should be your prescription, recommendation. That's where there's a strong feedback that should come in from diagnostic analytics which really is the causal component of it. It really tells you why is something happening and what is the extent to what it's happening. Now to concretize, assume there is this causal inferencing framework and you're running a marketing campaign and you kind of understand through that that email notifications kind lead to lift and sales, right? So once you have this wisdom in place, this causal wisdom in place, you can actually action on it. You can recommend to lever email notifications to drive up your sales and that's the power of causality. Well, I'm hoping that we have given a broad brush introduction in a nutshell of causality and why do we need causality and so on and so forth. Now causal inference is really what gets you to the deduction of this causal effects. There are more than one analytical frameworks that are possible to do causal effects. We'll be focusing on just one today called the potential outcome model or the PO model. Like any statistical framework, it is based on a bunch of assumptions and we'll be focusing on one particular one today. More such assumptions are met, the better your chances of getting to more robust results. Now it's always good to put faces to name and this gentleman that you see here, forgive the pixelation and so are the pioneers of potential outcome model starting from Jersey Neumann 1920s, that's when this entire thing began. It died down a bit, it was reinvigorated in the 70s by Donald Rubin and in fact this model is alternately called Rubin's causal model as well in honor of Mr. Rubin and then again his associates Paul Rosenbaum and Paul Holland kind of took up the mantle. Well, to illustrate the basic essence of the potential outcome model, let's take a relatable example, nothing esoteric, nothing fancy. We have this person who's suffering from headache and he's contemplating on popping an aspirin to get relief. So in this instance, his treatment has really two options, either he pops in the aspirin that's what he gets treated or he does not pop in the aspirin, he does not go to the treatment. Corresponding to each such treatment options, there are hypothetical potential outcomes possible. So what really happens if he takes an aspirin, either he could get a relief or not get a relief. Likewise for not popping an aspirin, realized outcomes can only be either you get a relief that is your headaches gone within the next two hours maybe or not or essentially not get a relief, your headaches persist beyond the next two hours. He pops in the pill, what we notice is this, that the headaches gone, would our conclusion forget causality logical this right, would our conclusion be that aspirin has caused this remission? How many say yes, raise of hands? How many say no, divided house, all right. So proper causal thinking would probably tell you that we really can't say. And this is where the count of actual thinking comes into place, the count of actual in this case is what would have happened had he not popped in the aspirin. Now had he not popped in the aspirin and he had not experienced relief, vis-a-vis he had popped in the aspirin and experienced relief, then there is a difference in outcomes right, and that's when you can possibly say there is a causal effect of taking an aspirin. But herein lies the classic fundamental challenge of causality. You cannot have a person in two states simultaneously in the same time. This is not Mr. Schrodinger's cad, you cannot be dead in life. Likewise you cannot have a person pop and not pop the pill at the same time. And this is neatly illustrated in this case, I don't know if it's visible but what we are really trying to show is we are decomposing the observed outcome. And the treatment option is either 0 or 1, right, we spoke of it. So if you just plug in either of them, you'd realize there's one outcome. The other outcome will be unobserved, it's latent. You cannot observe it, you can just get one outcome, right. So we are stuck. Seems that getting into causal reads at an individual level at least is an impossible business, right. What do we do? We do what we always do in statistics. We expand the net and we think about a sample, right. So we move from the individual effects to a sample. So we look at over a sample and maybe try to get to an average read over the sample. And what would that require us to do? Rather than complicated maths, I want to do this graphically. So what we need is this sample of folks who are suffering from headache. Have them pop in an aspirin, have them also not pop in an aspirin at the same time, and observe the percentage experience of relief in each of the scenario. Take the difference of that and that will be a causal effect. But wow, that is again impossible. Remember the fundamental challenge of causality. This cannot happen simultaneously at the same time. What happens in practical, real, realistic world is something like this.   You get to the rates of relief in each and get to the differences. But the problem is you cannot really call that as causal. And why is that so? Because these two sets that you notice are not composing of the same set of people. They are different, heterogeneous. They could differ in terms of age, in gender, medical conditions and so. So despite the fact that if you observe any difference in the rates of transmission, who knows if it is due to administering aspirin or the pre-existing difference in effects. The one thing I wanted to drive home here is that when you're trying to get to any kind of causal effects, it'll always kind of be a comparison between, in this case, the treated and the non-treated set. They'll go by different, this goes by different nomenclature. It could say exposed, non-exposed, treated, non-treated, test control and so on. But it will basically be that. And keep this in mind because this will be effective in the next couple of slides. So we're back to square one. Seems that we really can't get to causal effects. Well, not so fast. There is something called randomized control trial or something that we call popular in the industry as A-B testing. What A-B testing again does is it takes the set of folks and randomly splits them into two parts. Now when it does this random splitting or randomization, what it really ensures is the set of folks in this two subsets, they're roughly similar. They're pretty comparable between themselves in terms of the overall characteristics. Which means that now, if you take the percentage experiences relief and kind of come up with a difference, you can sort of come close to what would be an ideal state. And A-B testing in the truest sense are really the goal standards when it comes to getting to causal leads. But, but, there's a big but. A-B testing is not always possible. Many reasons why? First of all due to practical considerations. At that time, you know, you as a poor analyst will be tasked with analyzing something for which A-B testing wasn't done. It's a retrospective study. You cannot turn back time, you don't have a time machine. And you have to work with non-experimental data. At times, there'll be multiple A-B tests running over the population, which will make it very difficult to tease out the effect of a particular A-B test. At times, cohorting can be non-random. So, in this case here, we have assumed that the cohorting is perfectly random. It randomizes, so there's a homogeneity between these two sets. Things can go extremely oriented. There have been cases where things have been. More importantly though, A-B testing may be undesirable at times. You cannot randomly assign people to smoke and not smoke to treat out whether they can get cancer or not. That's plain unethical. In practical terms, I was reading this recent article by the head of data science of Coursera, I think, Alina Glasberg-Sanz. And she made this very cogent point. So, Coursera has this certificate that I mean, essentially, if you need to get certificates from them, you need to purchase them, right? You need to pay for that kind of. So, Coursera as a policy does not do any kind of A-B test for this pricing. They actually rely on quasi-experimental methods or observation methods. You'll get to cover some of them later and try to understand the effect of price differences and what could be the optimal prices to set. Long story short, at this point, at many instance, you, my dear analyst, will be left with just observational data. And observational data has its own set of pesky challenges. The one prominent challenge is called confounding. Now, before getting into any technical artifacts, what does the word confounding conjure up in your mind? Just plain English. Anybody wants to take a stab at that? Right? Is it again? Confusing, okay, okay. All right, all right. Yes, confusing, befattling, and offer skating, and so on. Something that makes things fuzzy. And confounding here is exactly that. So, let me take an example. Assume that we are trying to find out the effect of exposure on an online ad for a brand on purchasing the brand online. Now, there's this theory and it's pretty proved that the more a person is online, right, more active he is online, the more likely he is to get exposed to that, and also buy the product online, right? So again, you see there's this one variable influencing both the treatment and the outcome. So even if you want to just tease out or kind of find out what is the effect, sole effect of digital ad exposure on buying online, you really can't. There's an offer skating happening. By the way, we have covered confounders before. In fact, the one that we spoke of in terms of Nobel Prize winning, the economic state of the country was bit of a confounder here. There's this other example in terms of medical sciences. Try to, it's the same thing, it reinforces the same thing really, where let's say that you come out with an experimental treatment. What really may happen is folks who tried the treatment also are the most sicker ones. So the state of the sickness may affect both trying out the treatment, unfortunately also the mortality rate. So again, it's difficult to tease out the effect of trying out the treatment on mortality. There's a confounder here in terms of sickness. So synopsizing what we have thus far, EB testing are the gold standard. Though EB testing is not always possible, randomized control trial is not always possible. It will be left with observation data at times. Observational data has this principal problem of confounders, the confounding, which really means there's some pre-existing difference between the treated and the non-treated side. I think you get the drift now. Our principal task in terms of causal inferencing, and that's one crucial thing that we always have to do, is to level the field. What we have to do is take off this pre-existing difference, mitigate them as much as possible, so as to restore a sense of balance between the treated and the non-treated group, and thus making them extremely comparable. That's what, you know, there's a main funder behind causal inference. Now, this is what is encapsulated by the major causal assumptions. I just have time to cover one, perhaps DD major one, which is called exchangeability, and I'll hug back to the previous example that we talked about. So assume that via research, you get to know that it is, you know, users who serve beyond 10 hours a week, they are more susceptible to viewing the digital ad. So what you can do, now, again, getting back, if you need to get to any kind of causal effects, you need to compare between the treated, in this case the exposed customers and the non-exposed customers, right? Now, what you can really do is keep only two customers who have been online more than 10 hours in a week. So effectively what you're doing is, you're leveling the field, you're controlling for the differences, you're taking away that source of pre-treatment bias, right? And that is basically what exchangeability kind of halves on. It tells you that provided you know what the confounder is, in this case, assume we know this is the confounder, you condition it, and you condition it in such a manner that the treated and the non-treated set, they're exchangeable. There's no difference, pre-existing difference between both the sets. And that's why we call this as exchangeability, right? One quick thing is that here we have assumed that this is the only confounder, may not be the case. Thing is it could be many things, it could be gender, geographical location, propensity for the brand, and so on and so forth. Now, the more exhaustive and the more covered your list of confounders are, the more your chances that you will possibly be able to satisfy such assumptions and get to more robust causal reads. Now, such causal principles are kind of brought to life by different techniques to tease out causal effects. Whole bunch of them, much even beyond what you see here in this slide, I will be focusing though on two of the slightly connected techniques called COVID-based matching and propensity-based matching. And rather than talking more about in theory, I'll try to cover them through a simulated case study. Hopefully that makes it slightly more clear. Try to define the problem first. So there's this retailer who has come out with the same-day delivery plan, which is, you order something, food, grocery, et cetera, it'll be delivered to your doorsteps in the same day. Now, the business hypothesis is that customers who subscribe to this program because of the superior customer experience they get, they will tend to become more engaged, more loyal to the retailer. And then the aftermath of subscription will tend to make more purchases. That's the manifestation of the loyalty that way, right? Big question is, does this play out and then what is the magnitude of this effect? Now, assume that this program has been launched at month M and we are at month M plus six for the moment. So it was launched in Feb, we are in August. And what we start off with investigating, of course, is an M will be a kind of point of reference always through this discussion, the month of joining. So what we start off with, who are the customers who subscribed to this program in that given month? There are 5,000 or so customers, in fact, who have subscribed to the program in the given month. Now, what we also obviously want to see is the key question, remember, is are they more engaged in the aftermath? And when we say more engaged, it's always a relative question. Are they more engaged with respect to customers who did not subscribe? And engagement here is something we define in terms of certain key retail behavior metrics. There are three that we'll use here. One is total number of transactions made in this six month phase. The second is spend per transaction. The third is number of items bought per transaction. So we have these three metrics in the post six month phase for the converted customer set. Now, to get to any kind of causal effects, we know by this time that we need to compare between the converters and the non-converters, which means that we have a set of customers who did not convert 100,000 of them, a random set from the overall population of non-converters, and similar metrics for them in the post six months period. We also expect that it may be very likely that there are pre-extinct differences between the customers who convert and not convert. That is why we also have the similar three variables, transactions, spends, and items in the previous six months before the, you know, this program was launched. Overall then, we have a data set with about 5,000 converters and 100,000 non-converters and over this eight variables, right? What we notice when we drill a bit deeper and trying to understand the difference between the set of converters and non-converters is interesting. So, this is what we kind of see. This is, so converters are those who subscribe to this program in a given month. Yeah, okay. All right, so this is the density plot, the fancy histogram of sort if you may, and the one that you see here is for total number of transactions. The graph to green is for the subscribers or the converters. The graph to pink issue is for non-converters. You'd note that on average, the converters make more transactions than the non-converters. In the last six months, I mean, this is pre-six month phase before this program was launched entirely. They made about 35 transactions, whereas customers who did not subscribe to the program may have made about 30 transactions. The same story plays out for spend per transactions. Customers who convert it tend to spend more than customers who have not converted, which means there already is a pre-existing bias of sort, right? Subscribers typically are among the, already are among the better customers. We're saying hello to confounding again, right? You know, the more engaged customers are more likely to subscribe to same day shopping. They're also more likely to buy more in the aftermath, right? So again, it will become very difficult to disentangle the effect of this program on incremental purchases. If we do not control for this differences, the pre-existing differences that we spoke above, and this pre-existing differences in, you know, engagement is manifested by the levels of the three variables that we spoke above, right? Transaction, spend, and items per transaction. Can I take that data after this, with a bit of difference curve? Okay, so this is, the crux of it is that, how really do we kind of get to the effects of this program on incremental behavior, incremental purchase behavior, right? This is what kind of COVID mismatching can help you do. So COVID rates are a statistical term for what we nowadays call as features in machine learning. And the way it works is something like this. So assume we have this one customer who converted or subscribed in a given month. These are his salient, you know, behavioral characteristics, the three variables that we have spoken up of in the previous six month phase. And assume for now we don't have 100,000 non-converters, but three non-converters, again they're salient behavioral characteristics in the previous six months. Now, if we had to control for engagedness in the pre-six month phase, what we need to do is find out a customer who is very similar to this person in terms of this three metrics, right? That we would have controlled for the difference that already exists. One way to do this is simple. You calculate the distances between this customer and the other customer. Here we have taken a very simple Euclidean distance in this instance. And what you do is follow a greedy procedure and find out the closest clone of this customer, the nearest neighbor of this customer, right? So what happens then, you see these customers, one from a converter set, one from the non-converter set, they're very close now in terms of these three metrics. They're almost exchangeable. We have actually satisfied the exchangeability. One can swap for the other. We have also controlled for this pre-existing differences in effects. Now, some quick pointers on the fundamental nuances before we get in the depths of this. One, I think you realize that this is based a lot on the measure of distance. We talked about calculating Euclidean. Not a great thing to do always. There could be instances where your variables could be different measures. It could be in dollars. It could one could be in inches. It could be height. What could be something else? Best to use something that kind of normalizes for that. What's used in literature and in typical industry is Mohan-Lawish distance, which really scales the distance by the variance covariance matrix. In fact, better, what's also used at times is the robust variant of M distance. It's called robust Mohan-Lawish, which is a bit more resistant to outliers. The other thing is calipers. This actually is caliper. We talked about finding out distances between a customer in the converted set and the non-converted set. But what if the distance is too wide? That's not a good thing, right? Would you really want to consider such a customer in your analysis because the matching isn't good? Calipers essentially is a distance. What it really says is if the distance between this matched pair, if it's beyond a certain limit, do not include it for analysis. Now, whether you want to include it or not is a very subjective question. Come to think about it. If you have to leave that out, you'll probably have more accurate results, right? So you have less of bias that way. But if you leave that out, you will have a lesser sample. So it's a classic bias variance, kind of a trade-off that you need to take into consideration. The other thing is we spoke off of one-to-one matching. There's nothing stopping you from doing one-to-many. In fact, if you do one-to-many, you'll have more samples. Your variance is reduced, but it's a flip now. If you move from the nearest neighbor to the second to the third, your accuracy of the match is slightly reduced. So again, a classic, you know, bias variance kind of a trade-off. Okay, so we spoke of the fact, and we have kept on speaking about the fact that the fundamental notion is there is a treated and a non-treated or an exposed non-exp... Let's call in this case a converter, a non-converter set, and there are pre-existing differences. So there is a fundamental disharmony. There's fundamentally an imbalance between both the sets, and we need to mitigate that. Then the first step for us is really to assess what is the extent of this imbalance. So let's take an example. Assume that we are trying to find out what is the extent of imbalance between transactions for subscribers and transactions for the non-subscribers. And a classic way to do that would be to do hypothesis testing that we all familiar with basic inference theory, right? You test for the equality of means, and so on. The only thing is that, typically, the large sample sizes that such studies involve, courtesy the large control group, typically the large group of non-subscribers, would often mean that even small differences in means can get very amplified. And you may end up rejecting the non-hypothesis, even if that is the truth. So a type one kind of error can happen very rampantly. So what's really done in this case is we resort to something of a more effective, but also a great measure, which is called standardized mean difference. Standard mean difference does something quite simple. It looks at these two groups, again, transactions for subscribers, transactions for non-subscribers. You calculate the difference between both, and you divide there by the square root of the variance. So you're standardizing. The other thing is you'd notice that this is bereft of any artifact of the sample size, which typically hypothesis test would involve the test statistic there. So it doesn't suffer from that defect. General practice, there's the absolute value of this is taking, the modulus of this. So obviously if this is zero, that means it's kind of a perfect balance. There's some rough rules of thumb from field studies and experimental results. So typically if it's less than 0.1, you're good, you're cool, don't worry about it. If it's more than 0.2, this metric, it's then that you start worrying. Things are not in a very good shape. And in fact, in our instance, if you look at those three metrics that we are concerned about, the SMDs actually are quite high. They are always higher than 0.6 and go up to one for transaction. So it's a bad state. So now we have assessed the fact that there is a severe amount of imbalance in these three metrics of interest. There is a preexisting difference. Now matching can be used to mitigate that. Assume that we have implemented matching and then what really happens? So then what really happens is interesting. The SMDs actually fall to between 0.02 to 0.03 for each of these variables. This is a drastic fall from where it was at in a pre-matching state to where it's at now. This is actually the density plot for transactions and you note now there's a good amount of overlap between the treated and the non-treated set. So it's fair to note now that we have brought in a semblance of harmony. We've almost kind of taken away that element of pre-treatment differences. And what we have now is a clone customer for each customer in the subscribe set. So we have, that way, 5,000 pairs of customers. We have, like I told you, we have 5,000 subscribers and we have one clone from the non-subscribers set for each of them. So we have a pair of 5,000 that way. Assume that what we're interested in is finding out does transaction get affected? Do we notice any tangible change in engagement in terms of number of transactions in the post six months of subscription? Because that's the thing that we're concerned about. What happens in the aftermath of that subscription program? We can do something simple. You can just tabulate the transactions for each customer in the set and calculate the differences. The easiest thing to do would be to kind of take an average of that, which is something we have done. The average turns out to be about bit over two transactions. So two transactions, incremental two transactions for the set which subscribed over the set which did not subscribe over six months, which means about four transactions over here. And if this expanded over a wide customer base, this has quite a bit of an effect. It's a quite a bit number of transactions. You could also do significance testing. Just take that with a pinch of salt. P values have taken a lot of flak off late and so on, legitimately so. And it turns out to be quite significant if you do a simple P testing on the differences. We also do the same thing for spend per transaction. The interesting thing that we notice there's a very small delta for spend per transaction. So what's probably happening is customers are more engaged. As in they're making more transactions but they're not spending more money per transaction. But eventually that still meets more engagement. Overall, your overall spends are increasing given the fact that your transactions are increasing and so on. Now all that is fine. What creates a problem for COVID is matching is when your feature space explodes, right? When you're dealing with too many features, first of all, calculating or getting to such nearest numbers can be computationally complex. The second thing is even measures of distances are very fuzzy in such high dimensional spaces, the curse of dimensionality, right? So the thought is that what if we could rather, for now, what we had done was we were taking these vectors, these three dimensional vectors and calculate the distances between them. What if we could rather condense this into a scalar, just numbers and calculate simple differences. Then we wouldn't suffer from such effects, right? And this is the gist of something next that we're going into called propensity score matching. What propensity score matching does is really it predicts the chances for a customer to be part of either of this subscriber or non-subscriber, I mean essentially part of the subscribers group, right? So it produces a score for that. And how does it do that? So we have data for this, the customers that joined in the given month, right? So we have data whether customers have joined or not. So it's a one, zero variable of sort. You also have predictors, the behavioral variables of interest, right? What you can readily do when you see it lent itself very well to a classification kind of a modeling framework, you can use these predictors to actually predict the chances of a customer subscribing to the program. And you get a kind of a score, which is a scalar, which you can now use to find out customers who are kind of nearest to each other, right? We are avoiding calculating distances entirely. And this is what we kind of do. What I have used here is something very simple. I just used generalized linear models or binary logistic regression model. This is the same old data that you are by now kind of familiar with. As an output from the model, it spits out a score between zero and one for customers to subscribe. And what you can do is again, calculate just the simple differences, right? Between customers in the subscriber set and the non-subscriber set, again, get a close clone and the same process repeats. Practical tip, given that the probability scores are confined between zero and one, at times it becomes very difficult to find out close clones. The variance is very less, it's very closely clustered. So generally what happens in practices, you actually transform this score from a probability space to a logout space. Because the variance is more, it ranges from minus infinity to positive infinity and it's quite easy that way to find out close clones. It's cleaner that way. So we follow the same process. We do this exercise and in the aftermath of this, again, we notice that there's a sense of balance that's there in the data now. The SMDs are in the range of 0.02, 0.03, something similar to what we had noticed for covariate based matching. We also do the same analysis for understanding if there are treatment effects, that there's a higher engagement for this customer who's subscribed. And we've shown this here only for transactions. And you notice again, the delta is about two point something. So it kind of hugs back to what we had noticed for covariate based matching. Now, this is not the only instance where you can use matching. Matching has various use cases. It's very often used in advertising industry, especially for understanding the ROI for online campaigns. It's often used in multi-touch attribution models to understand the ROI for different digital investments. My favorite instance, though, is where Heineken had kind of used it to understand the effect of cooler replacements on stores which sell beers, to understand whether it leads to any kind of sales uplift for them. By the way, all references for these are given here. So these two feel free to go through and share. In terms of tooling, so the techniques that we spoke of primarily come from a statistical school of thought and no surprises that they're slightly better covered in R. Python has all that you need to kind of build up on it, but for now it's better covered. In R itself, I have primarily used match and match it, but there are more options that are available for you to do in R. I'd like to speak about one of them, which is called sensitive analysis, which I could not get to cover because of the time limit here, but what really happens is that once you have done with matching, so see, what we have assumed here is we have assumed that there's this set of confounders. There's three variables that we spoke of, but there could always be unknown unknowns, confounders that we aren't aware of, so unknown sources of bias, and what may happen as a result is, things could change drastically if you do not account for them. Your results, your magnitude of results, your direction of results can change. So what may be needed at times is to stress test your model, to understand how resistant your models are to such specification biases. That's what sensitive analysis is done. It's very important to kind of do it perhaps at the end of the analysis, so definitely to go through the materials when you can. By the way, like I said, I think I alluded to it. What we've covered in matching is definitely not all. There is much, much more to matching than just this, and there's this companion GitHub repository I've created with more materials, more documents, the source codes, et cetera, and data set for this. Share that with you, so please do go through. Hopefully that'll help you get started on your journey in matching. Now, quick thing. Does this mean that matching is the silver bullet to get to all our causal effects? No, absolutely not. And this is this very interesting case study that Facebook came out with. By the way, this is a quote by Cochran, who is one of this founding fathers of modern statistics. Now, this is very interesting study that Facebook did. What happened is they had this, about 14 online campaigns that they had run for which they actually had to use randomized control trials, which means they're using a gold standard. They have reads on the efficiency of the campaign using RCTs. What they wanted to do is use observational method and see how closely can this gold standard numbers be approximated. Now, Facebook had an unique and enormous wealth of data in terms of users. I think no surprise, you guys all know about that. And that's one key thing. The second key thing is, if you go through the paper, again, reference is given. They say that each of the campaigns could range users between 2 million to 14 million, which means that you have an enormous sample size, which gives you a tremendous power to analysis, right? And they used fair amount of sophisticated observational methods even beyond what we covered in the scope of the analysis. They used both the ones that we did, but beyond that as well. And despite that, what they saw is that in about 50% of the instances, observational methods overstate the efficiency by a factor of 3x. And why is that so? It's basically because of what we spoke of. There could be unknown unknowns. There were unobserved sources of bias that weren't accounted for in such large scale experiments. So what this is, is a sobering tale or what this tells us is that you need to be very sure about your data generating process. What is influencing? What, how is it influencing? And be sure about the structures. Well, that is a cautionary tale, but I really wanted to end up this session with something that I find incredibly hopeful, which is work happening at the confluence of causal inference and machine learning. And there has been a lot of attention of late on why to push the frontiers of deep learning and artificial intelligence. We need to teach machines to differentiate between cause and correlation and effect of it. Much of it has come due to, you know, there's a hug back to action by Judea Pearl, who is a big founder figure of the probabilistic school of thought in causal reasoning. This very famous book called Booker by, very interesting, you know, definitely worth going through. There already was work happening in bits and pockets on machine learning. So this is a group that Professor Vandalan leads in UC Berkeley that works on something called targeted learning, which is almost the machine learning we are getting into causal inferencing. Of late, if you have been following the major AI conferences, I Clearance, so you'd notice that along with ethics, governance, security, causality is one of the major themes that have emerged, right? In fact, in the recent IKEA conference that happened in New Orleans in May, something this year, there was this very interesting presentation by this very famous researcher, Leo Batu, who works in Facebook. And he proposed coming out of the system that kind of tells you what is the invariant property of a kind of overall system, which means that he's trying to get to measure that kind of calls out previous correlation and gets to causal effects. Beyond the scope of this discussion, but I've given the link to it, very interesting the way that they're doing it. And in general, what I think is we are headed to very interesting times in terms of causal inferencing and very exciting times ahead overall. Thank you for your time. Thanks, sir. We only have time for one question. All right, yes, sir. You can connect with Swash's offline. Hi, really nice talk. So in the example you quoted, what would have been the gold standard? I'm not clear because we were still doing A-B testing. It's just that A-B testing was in terms of time and not like, so what would have been the gold standard in that thing? A-B testing perhaps would have been that you actually allow some customers to subscribe and essentially randomize between customers and let such customers, send the customers through some subscriptions and keep off some other customers from subscriptions that way. So even in that, we would have to do the matching rate. Even if you're doing that, we have to do the matching. So what I got is that the flow was that gold standard is this. If you don't follow the gold standard, if you just have observation data, you would follow matching, which I don't think is entirely true. Even with the gold standard or whatever, you would have to do the matching to ensure that the groups are divergent, same, yeah, yeah. Is that correct? That's correct. If you can do every testing, you don't need to do matching at all. Did I get that question correct? So I think I was saying that even if you are- Can we take it offline because it can't be here because of the movement? We can take it offline. I can explain that to you, yeah. All right, I think, thank you.