All right, let's find a new slide deck now. All right, it is 11.35 in my watch. So we'll get started. We've given five minutes buffer for people to get in and let them come in. So this title of this talk is inverting the pyramid. And we can introduce the word testing in it because I'm going to eliminate the word testing from it eventually. That's essentially the idea of this talk. This is an experience report of me having done this for two specific organizations where we have inverted the testing pyramid. And I'll talk about what that means and why the title. So everyone's familiar with, you know, back in the Stone Age, the kind of development we did. And there's the surprise. Yes? So what do we see in stuff like this, right? When we do development in this fashion where we plan, design, distribute work, work in isolation, and then come back and try to integrate stuff, right? I've not even put the testing part yet, right? I'm just talking about this. Even this stage, we see a big surprise. So what we see generally is a lot of last minute surprises. Bad things are visible too late. And we find that as a real problem, all right? Actually, let me pause for a second and ask how many people got a chance to look the slides before they came in, OK? So for some people, what I'm going to talk is kind of sounding familiar. But for the rest of them, we'll kind of go a little slower. My idea is to brush through first initial section and spend a lot of time getting into more interactive session where I want to talk through some basics and then get people's experience and kind of take it from there on an evolutionary path, all right? So bad things are visible too late. We get last minute surprises. This was a typical problem. So to address this problem, what did we do? From a distribute work and then get back and integrate everything together and then test it to address that issue, what was the practice that teams started practicing? Continuous integration, right? Any other practice? Early feedback. How do we get early feedback? What specific practice? Test-driven development. Fantastic. What else did we do? Continuous delivery. So we went further all the way and started looking at continuous delivery, right? So let's talk about birth of continuous integration. Anyone knows which was the first continuous integration server? Fantastic. So I was on the team where we actually built the first continuous integration server. The first version of continuous integration server was something we wrote in flat 15 minutes. What was it? It was a cron job which would wake up and run the build and essentially would go through and say, OK, everything integrates and the automated test that exists seems to work fine, all right? No fancy reports, no nothing. Just a cron job running on the server that we had, right? That was kind of the birth of CI. I'm not going to go into details of CI unless you guys feel I should. I'm assuming at this stage everyone's familiar with what continuous integration is in a nutshell. So that's kind of to address the issue of late integration, the practice of continuous integration was put in place. As Martin Fowler, who essentially popularized this practice, Martin Fowler should get the credit for bringing continuous integration to the masses, he says that if it hurts, do it often and do it early, right? So that's the kind of principle behind continuous integration. If integrating late hurts you, then do it often and do it early. How does CI help? Increasing collaboration, decreasing delivery time, increasing feedback or early feedback, decreasing wastage because of all the rework and things like that, and inevitably it essentially leads to a higher quality product that we are building. Is that been kind of experience you guys have faced? I want feedback and I want early feedback and I want a continuous integration, right? So we did that. That was early 2000, 2001, right? What next? It's been 30 years. We've got to come up with something better than that, right? So you might have heard about the whole lean startup community. The Eric Rice wrote the lean startup book, but there have been a lot of interesting things the lean startup community has been doing. And they talk about what is referred to as continuous deployment, right? So taking continuous integration one step further and doing continuous deployment. I'm not going to continuous delivery yet. I'm specifically talking about continuous deployment, right? Everyone's familiar with what continuous deployment is, right? Like we do the continuous integration, we want to take it one step further and we also want to push it to our customers, right? As and when we do the continuous integration cycle, right? Sounds like a very small change, right? Sounds like a very small change. We were only deploying it on a test server. Now we're actually going to push it to a production server. Obviously, we're not going to push it to all our servers. We're going to do it in an incremental fashion. We're going to roll it out in a way such that we can roll back if bad things happen. So we will do some of those things. But it sounds like a very small change. And when I first heard about this and I saw companies doing it because I was at a few companies which were actually doing continuous delivery way before the word was coined or way before anyone actually was talking about it, right? If you use Gmail, Gmail has been doing continuous delivery since 2006, right? Way before the word continuous delivery was coined. But that small change actually has a profound impact on how you work. But the key thing I want to focus is I don't want to go into continuous deployment, but there's one element inside continuous deployment which is testing, right? What happens to testing? Now inside, if we want to do continuous deployment. Obviously, continuous deployment is like the vision for every company at this point. Everyone wants to be able to deploy stuff continuously. I've worked with companies which are doing hardware development. They actually have tractors running in things. And they're talking about doing continuous delivery, right? They want to continuously deploy software on the tractors running across the world. There are companies doing all kinds of medical devices, and they're talking about continuously pushing in the latest greatest stuff, right? So that seems like the gold dream for every company. But what is the problem to get there? Any guesses? Testing. Because we all have what we love, the ice cream cone, right? Or in other words, the inverted testing pyramid, and which is kind of what I would want to focus the rest of this talk, talking about, this is the problem we have today. And if you want to go to continuous delivery, how do we make the transition? And I'm going to talk about what specific things we have done to make this transition. We're also going to set some context in terms of what this means and how this goes. Yeah, of course. This is based on my observation of what percentage of the tests in a given company, how they were distributed, right? So there were only about 1% to 5% of tests that were unit tests, right? There were about 5% to 10% of the tests in terms of integration. So when I say percentage, I'm talking about coverage, right? So only about 5% to 10% of the coverage of the overall code was coming from these kind of tests. And then end-to-end tests had a 10% to 20% coverage. Majority of the coverage came from manual testing. How many people here have this problem today? So I want to talk about how, with two specific companies, we have inverted the testing pyramid. We've fixed the testing pyramid. I'm going to talk about how we fixed it and what are the specific things we did. But that's kind of freely the focus of this talk. So this is a good time to leave if this is not what you expect from the talk. Commercial break, like before starting any good thing, we have to have a commercial break in between. So my name is Nareesh Jain. I live in Mumbai. I started the bunch of communities. I'm an entrepreneur. I have two startup companies. One helps kids learn mental arithmetic. And another one is a marketplace for conferences. None of these are launched. We are not doing continuous delivery on these projects yet because we're still even trying to figure out how to solve the problem. We've spent about a year now building something for kids on the iPad, which is a game. And it took us a year to even get to this stage to figure out. We've obviously done regular tests or almost continuous tests with kids all the time over the year. But there's still a long way to go. Bunch of companies I've worked with as an employee or as a consultant just to give you some background in terms of the different experiences that I've had. So that's enough about me. Now let's get back to the actual topic. Has anyone heard about the distinction between testing and checking? I don't expect you to read that. Anyone knows who this guy is? Michael Bolton? Good guesses. So Michael Bolton was the first person actually at one of the conferences he came up and he started talking about the distinction between testing and checking. And everyone was like, what do you mean? Doesn't make any sense. Of course, we do testing.  You do checking. So what is the distinction between testing and checking? I already know something. And I want to validate if what I know actually meets my criteria. And he says that is checking. That is not testing. What is testing then? Testing is where I don't know exactly what I want. I'm exploring. I'm trying to figure out what would happen if I did this to the system. If I poke the system in this way, what would happen? So this is profound because what I want to highlight here is you can automate 100% everything that is checking centric. Everything that we think of checking can be automated 100%. But everything that is testing centric, which is exploratory centric, where I don't know what to assert, where I don't know what is the expected outcome. I cannot automate. So when we talk about continuous delivery, are teams doing checking or are teams doing testing? They can do checking. But interestingly, I'll also talk about how they also do testing and how they incorporate testing into this thing. But the point I'm trying to make is vast majority of companies today are doing checking and thinking they are doing testing. And they're still stuck in the manual thing because they think it's exploratory in nature. I was recently at a company and they were saying, you cannot automate 100%. But agile says you need to automate 100%. That is not true. These guys don't know what they're talking about. So I said, hang on. Let me draw a distinction, checking and testing. Now can we say what can be automated, what cannot be automated? And they're like, ah, now we understand why they're saying you can actually do 100% automation, which is essentially checking and not testing. So coming back to our inverted testing pyramid, what we want to do is this is the end state we want to be in. I'm going to pause, let you read through this, and then I'm going to talk through this. You might have seen another triangle, which is a much smaller triangle, which actually I'll talk a little later in terms of that triangle works well, but if you're actually doing anything real world substantially big, this is my experience. This is after doing it for about 10 years, this is where we ended up with. And this is what I think as of today, the current state of what the inverted testing pyramid should look like. It might change tomorrow. So disclaimer right there, right? Everyone's familiar with unit tests. So we're saying we need about 70% of coverage of your overall functionality from unit tests, right? At least 70%. Is that easy to achieve? Experience shows that we can actually do it fairly well on large number of projects. We've done this fairly well. So these numbers are not research numbers, right? These numbers are personal experience numbers. So take it with a pinch of salt because this is my experience. This is not an industry standard. Business logic acceptance tests, what does that mean? Business logic acceptance tests is on large projects we have large number of teams trying to build a whole product. So to get one end to end workflow of what a user would experience is fairly large, right? If I'm in a bank and I'm trying to get one trade go from one end to another end, that might actually take about five or six days for it to complete. As in not development time but actual physical time for the trade to actually go through, right? If I am in a medical company, right? If I'm in a hospital, it might take six days for you to actually do a blood assessment to get the report to figure out what next treatment needs to be done and that's a flow when a patient comes in, right? That might take a significant amount of effort. It might need deep experience. So you take each portion. So, you know, given a blood test report what are the results for it? That itself is a good enough logic which could be covered through this acceptance test. But will that meet an end to end requirement for a user? Not really, right? So I'm trying to distinguish between an end to end acceptance tests, right? Or a workflow test as I call to a business logic test. These are tests again because these are exploratory in nature to start with and then they get converted to checks, okay? Integration tests. What are integration tests? Integration tests are if there are two components talking to each other or if my system is talking to a subsystem, I want to make sure that the interface between the two is well established. I don't care about the logic because the logic can be tested in different places, right? I'm interested in making sure the integration, the touch points between the two is tested well. This is where you can do a lot of negative path testing, right? What happens this server goes down? What happens if it cannot take this kind of a load? What happens if this does not happen, right? If a typical, if I expect a lot of concurrent connections happening to a particular system, right? It is possible that it can get into a deadlock. What would happen if it gets into a deadlock, right? That's exploratory in nature. And that's something that you can cover with these tests, these kind of tests. Then you move to your workflow test, which is given a patient comes in, what are the different steps that would happen in my system? What are the different touch points? You move further up, which is from an end to end workflow point of view, how does it flow through the system? One layer below the UI, one layer below the actual user interface. And then the tip of the iceberg or the tip of the pyramid is essentially stuff which will click buttons or stuff which will actually have user interaction and then go all the way through. So if you laid out in this way, this is kind of where we want to get. And if we can get there, I believe we've been able to successfully get companies to do continuous delivery, continuous deployment. And that's kind of what we refer to as inverting the testing pyramid. There are also bunch of things marked in terms of where you can do performance test, where you can do security tests because those are important as well if you're building large critical systems, okay? So this is the end state we want to be in. And the rest of the talk, we'll talk about how do we get to that end state? Yes, please. There's one percent coverage. Well, so it will touch other things and you will get a higher coverage, but that is already covered through this thing. There's one additional percentage which was not covered before. That's what you would get from here. I would be able to test a particular user interaction in my unit tests. I don't need to do an end-to-end test to test a particular piece of the UI. Correct. What could go wrong on clicking on a button? What would go wrong if somebody clicks on the button? Can you test here? They don't need to know, right? Absolutely. Correct. So you will show that, right? Why not? That's what we've done. So on the Gmail team, so I'm not talking about most of the time, right? I'm talking about what should be the end state and how it should be done. We'll take that offline because I think there's a misconception.  So here you've got 70 percent coverage. When you go up, right? I'm saying this is 10 percent in addition to what the 70 percent you have. So it becomes 80 percent coverage at this point. As you keep moving up, when you build this, you should have 100 percent coverage. It keeps adding up as you go up. No, what I'm trying to say is as you build up, right, it covers more. But what I will focus is the one percent that was not yet covered. 99 percent of it is just covered. Correct. That's exactly what I'm saying. Yes. But of the two, you don't test all the time. Okay. I need to move on. Sorry. What is 4 percent of the coverage that takes and 20 percent of the testing time? I'm not talking about testing time at all, right? What? Sure. As we go further, we'll talk about how some of those elements will be pulled further. For example, if you see, we are doing some amount of performance test right here, right? Because we don't want to wait to this point to see that. And I'm also going to talk about when each of these are happening and how they are happening, because that will help you understand more from the time it takes and things like that. Okay. Why the name inverting the testing pyramid? It actually comes from this book called inverting the pyramid. Anyone's heard of this book called inverting the pyramid? It has nothing to do with software. It has to do with soccer. And how, you know, if you see, if you go back in history and you look at how teams were structured in soccer or in football, right? You would see that early on, there was like a very heavy focus on fronters, right? The front people in the goal post. And then as you go back, the teams were structured saying, less midfielders, very few defenders, and then it would be a triangle like that. The South American teams came in and trashed the European teams for five years together, right? And then this author kind of went back and started looking at structurally what is different about these two teams. And then he wrote this book saying, you know, inverting the testing pyramid, how the South American teams actually had an inverted structure of the team's structure in terms of how they are on the ground and that helped them actually beat these guys over and over again. And today, if you see, most teams are structured in the soccer and where they have very few forwards, they have a good number of midfields and a large number of defense, right? So that was the book kind of talking about that. And I was like, you know what, this kind of is very similar to what we're trying to do in testing. So that's the name, inverting the testing pyramid. This is just a slide to highlight the problem with an end-to-end test, which is where a lot of companies seem to focus a large amount of their effort, right? The amount of end-to-end tests it would take for someone to cover the whole system is very high, right? And those tests are extremely complicated and extremely fragile and time consuming, right? So if something breaks here, for example, if I run one of these end-to-end tests and if something breaks, I have no clue where it actually went wrong, right? So I would spend the next maybe half a day or a day debugging through my system, figuring out where something went wrong, okay? That's not mine. So what ends up happening is you end up building a large number of end-to-end tests which are extremely fragile, complicated, and they don't give you the pinpointed feedback. So this is, and typically the end-to-end tests can only be returned when the whole system is up and running, which probably is too late to get feedback again, right? So we want to move away from this model. We also looked at, based on some projects I worked, we looked at the ROI from testing, the ROI that we get from testing and where do we get the biggest buck for the bank, right? And what we found is a large number of things could actually be caught at the unit level which we should be doing and it only takes about 10% effort in trying to build them, right? These numbers are at a very high level just to give you an indicator. They are not accurate number going to decimal values, right? And as we move up, the effort becomes much more and the return on investment becomes very less. So something to keep in mind why inverting the pyramid is extremely important, right? So now let's talk about how do we invert the pyramid, right? Earlier, I also test upon, a lot of people might have seen this. I had this in one of my presentations back in 2004, I think I was talking about acceptance test-driven development and I was talking about this is exactly how we do things. And since then I've worked on very large-scale systems and we've realized that this is not the whole picture, right? This has many gaps that we've not filled and over the years we've kind of evolved to something more bigger. But if you're working on a small, single team agile project I think this is good enough, right? But I see more and more large-scale, multi-sighted programs running where they have one product being built by multiple teams. You'd need something like that. I'm gonna not go into details of each of this but quickly touch upon this. This is there up in this slide for reference so you can go into details of this and read through this. But I wanna quickly touch through this. This was a very detailed slide that I presented recently to someone explaining all the different levels of testing, why they are done, what tools are used, at what level would you write, what is the objective of it, right? Right, starting from your unit tests. So what is the objective of your unit tests? During your sprint or during your iteration while you're coding, right? You wanna make sure that it's really blurry for whatever reason. You wanna make sure that each class functions exactly as you expected, right? As you move, things start increasing in its scope and things start covering more than the unit tests cover. Who is involved, when they're involved. So this goes into fair bit of detail. I don't think we have time to kind of go into details of this and I'm hoping that you guys can kind of read this in more detail at a later point in time. But these are all the different levels of tests that we showed in the testing pyramid. These are the kind of people who should be involved. Some tools that I have used in my experience, there are many more tools that are available and I'm sure you can do the mapping. And when it's done, what level it's done and what is the objective. I think the objective is the most important thing that I wanna quickly talk about, right? Here we're talking about each class, each module, each unit of your system functions correctly. And I wanna make sure as I'm building them, I'm getting the feedback early on, right? Here we're talking about from a business logic point of view for a particular end-to-end in terms of an architecture flow. Not necessarily from a business users point of view, but an end-to-end from a flow. I fill in the form, I go in, I've given some details and then it goes in, it sits in the database or some data store, it sends out a message and then it comes back, right? That's only part of the thing and I wanna make sure that is tested and it's working fine. As I go next level, it's, you know, whether two components can actually interact with each other, have I flushed out the interface correctly between the components. As we go further, you know, you look at like, for example, a trade flow, whether it can actually go through the whole trade flow. There are multiple steps inside that. Are we doing that correctly? Ensuring further, you know, again, I'm taking an example of trade over here from a bank kind of perspective to kind of describe how each of these things go.  So we wanna make sure all of these are covered, but more importantly, when does this happen? What's the flow in this thing, right? So I'm talking about testing touch points through the project lifecycle and at what points does this come in? So I'm not sure if there's something I can do to make this better. It's been, okay. That's bad news. So we're talking about, I'll just read through this, so hopefully it will be fine. So we're talking about kicking off a product or a project with what we call as a product discovery, right? A typical one week to two week session where you kind of try and understand what is the overall high-level vision of the product that we are trying to build. What are the different release milestones that we think roughly at this point and what would be, you know, the acceptance criteria at each level of your story map, right? So this is kind of, you know, we're building a story map and in the story map, you have the goals of your users, then you go down one more level, then you say these are the activities that the users would do.  You'd go one more level down and you'll say to achieve this user task, these are my epics. Then you go one more level down and you'll say to achieve this epic, these are my user stories, right? So you can build that tree structure and at each level of the tree structure, you can define acceptance criteria, right? So to meet these goals, I need to be able to do these four activities. That's my acceptance criteria at that level and as you keep drilling down, you're defining what your acceptance criteria are. So here we've got the different levels, we've got the acceptance criteria, then we move to the release planning level, where we talk about what are the goals, how do we assess we are ready for this, right? How do we ship this? So as you go down, you will start looking at a release level. Inside a release, you might have multiple themes that you are attacking, one theme at a time, right? So for the first theme, how do we ensure the theme is coherent? How do we ensure this is functioning as expected? So a lot of these you might not know at this stage. You're kind of discovering, exploring what do you want. So you start putting in things saying I would wish this happens, right? You don't have a system yet in place to ensure that will happen, but this is what you're trying to say is my acceptance criteria. You might have different work streams that will work inside a release and you want to make sure that all these different work streams are collaborating. This is where you will define your integration kind of tests, how each team will communicate with each other and stuff like that. As we go through, we are going from a high level, product level, product vision point of view, writing acceptance criteria, all the way up to when you do your sprint planning, this is where you define your story level acceptance criteria. At this point, you've done a lot of definition or exploration of what expected behavior you would want from the system, okay? And here onwards, during the sprint, you're actually collaborating with the devs, the business guys, the business guys, you're writing automated acceptance tests and before you write any tests, you're actually writing what you have captured before this, you're converting them into something that's executable. They will not work yet because you don't have a system in place, right? But you're capturing them. I'm gonna show you a few examples of how we do that, right? And then as we move further, we're kind of ensuring that at each of these levels, what you had defined is actually being fulfilled. So this is more about the definition and that is more about the validation, right? Some people look at it and say, oh, this is nothing but old wine and new bottle, the V model, right? It is in some sense, but in a lot of sense, it is not because we're talking about along the way you're actually kind of making them automated, right? It's not a manual step at every stage, but that's still a thought process. It's a valid thought process to have, all right? So far is this clear, right? As we move through the project lifecycle, at what point we are defining what and how we are capturing them, different kinds of tests will come in. So where would you typically see riding unit tests, for example? When would a unit test be written in this lifecycle? Sprint level, right? Fairly obvious. What about integration tests? I'll come to you in a minute, one second. This would be at the work stream planning. When you're doing your work stream planning, you would start defining what your touch points would be, how you would expect them to behave, right? Much before somebody's actually started building code, but sometimes these will be very ambiguous. You have no clue. You might want to do a spike story, right? To figure out what this would actually look like, how these two systems will talk to each other, right? Sometimes you don't even have the other system in place, right? Again, you wanna try and capture some stuff by actually doing your spikes by doing different technical stories, but it is all centered around doing stuff rather than just spending time in meetings and doing triage meetings and whatnot, right? A gentleman over here had a question. You need to speak up. We'll start automation right here because we start defining these and these would be executable right from the get go. These will not work, these will fail, right? But as we keep going through, some of them will start working and when a whole feature is built, that's when all of these would be automated, already automated but executing at this point, right? Again, going back to, when typically we say automation, people think of using an automation tool which will go record the screen and things like that. We have moved away from that at least for a decade now, right? We are looking at tests which will essentially go for large portions of stuff going one layer below it and actually driving it, right? And then you can easily put a driver on top of it which will actually go poke a UI for you, okay? So automation starts, is a part of this whole process. It's a very integral part of it. It's not something we can easily say now is when automation will happen, okay? Some of these are ambiguous at this point. It might not be clear. An example would be good to look at and that's what I want to get to. But also I want to kind of jump into, I don't know if you've seen this collaborative product discovery, what are the different steps and how this flows through? What happens at what stage? What kind of things we are capturing? But throughout this we are actually capturing acceptance criterias which are then to be automated using your acceptance test. I wanted to take a quick, actually let's, I'm not going to go into details of this, but any questions on this in terms of have, have people seen this before? How many people have seen this before? Okay, not many people. So let's actually spend a minute on this talking through this, right? So we talk about the elevator pitch, the business goals, what we are trying to achieve, which is what is chart-fring.  You come up with, you know, pragmatic personas, different kinds of personas that you have, which are your likely users that are going to use. For each of them you define user goals. What is the goal that they're trying to achieve from your system, right? Why should somebody looking at testing care about this, right? Testing unfortunately in large companies that I go is about given this requirement, this spec document that is given to me, can the system meet that spec document, right? And we've seen this movie before, right? Which does not have a happy ending. How many people have seen that movie before? Right, and it doesn't have a happy ending, because yes, I verified what was there in the document, but that completely missed the point, right? It did not capture whether at the end of it, the system did not really solve the end users problem. So one has to obviously throughout the process be very cautious and cognitive about what is the end users goal, and are we actually meeting the end users goal, right? So as you go through this, you have to make sure that at each level you have acceptance criteria defined. We talk about the activity map, the task map, and as we go through this, when we get to the user stories, a lot of people only write acceptance criteria at the user story level, which means I will test that this story in isolation does what it's supposed to do. But that does not tell me anything about whether the system will solve the end users problem or not, right? And that's a big gap that I see in a large companies. And when the focus is so much around user stories, imagine now deploying at the end when the user story is built. Do you think you'd actually be able to deploy it into production? You will not, because you saw down the lane that you've only tested one little piece and you have no assurance that the system actually will solve the problem for the user, right? So which is why I think this is extremely important to cover in terms of this is, these are the different stages and this happens right at the beginning of your release, okay? If you need more details about this, I suggest you talk to Jeff Patton. Jeff Patton is here. He can actually go into a lot more details about this. I learned a lot of this from Jeff, in fact, and then I've kind of brought in the whole acceptance criteria-driven discovery aspect of it, where we actually use acceptance criteria at each level to drive our discovery process, right? So if you use the acceptance criteria to say, okay, if this is the goal, what is the least amount of work that I need to do to have a delightful feature, but also to meet the actual goal? And can I actually do that with as little possible, right? So you kind of drive using acceptance criteria. Everyone must be familiar with the typical continuous integration process, but I want to focus more on what we refer to as the build pipeline. And I'm gonna talk about at the build pipeline level what kind of tests get executed, what gets verified, mapping it back to the pyramid that we talked about, okay? So the simplest form of your build is the local dev build, right? Which every developer would run on their local machine. What would they verify as part of this build? They would ensure that all the unit tests are working, but also the acceptance tests that they wrote, right? I'm trying to satisfy a particular story's acceptance criteria, are they working, right? Have I broken something which I'm touching in a big workflow? I'm touching one portion of it. Have I broken anything in the workflow, right? So that's what you want to verify at the local build level. Correct. Which is why the business level acceptance criteria that I talked about, that is the business level acceptance criteria, not the end-to-end workflow acceptance criteria. So that might have like, I'm sure you are aware of that, that might have talking to hardware, that might have things like doing that which might not be available, right? So this is still at your component level in that sense. But if it's a simple project, I am building a web-based product for someone, let's say Gmail for example. I could actually do the whole thing here, right? So it does obviously vary from system to system and I'm talking about more complicated system where the hardware dependency or other kind of third-party dependency, where you will not be able to do everything over here. So we want to be pragmatic. What is possible in this environment, we want to do that, right? You check in the code, then each team has their CI server, right? That does a check out and if you actually see these guys, we also have some time indicators over here based on what should be the ideal time these builds should take. What is the feedback cycle you should have? If it takes too long, then you're not getting the right feedback cycle. You've put too many things in it. You need to break them apart, right? So you get the most important early feedback as early as possible. So the smoke build essentially ensures all the unit tests, all the tests that you're working inside this print are actually working, right? This might still not test the end-to-end thing. That kicks off when on success, that kicks off your functional build which essentially ensures that from a functionality point of view for this particular module, this particular team, the functionality works as is and it's not broken anything, right? Then you go to the cross-stream build which means if I'm integrating with other teams, you know, let's kick off. If you notice at each of these points, there are some things being published in the dependency management system which means at every level, you're not gonna be compiling code and doing things.  For example, if you're building using Java, right? This thing would have built something and pushed a jar back into the system, right? This thing would come, pick the jar, take the jar from the other modules, build a system out of it and then run it, right? So you're not gonna, you're gonna try and reduce as much as possible in terms of not doing everything from start, right? Some of these might not even be a clean build. What a clean build means is you're not gonna clean out the environment and start from scratch. A lot of them can just use the existing database, can use the existing thing and just run on top of it which we call as the incremental build. So there's the next slide after this which talks about which one is an incremental build while which one is a clean build. So clean environment versus an incremental environment. Both are extremely important. We try to do both of these. So anyway, you have the theme level build which will essentially run in your system integration test environment and ensure that at the theme level all of these things are connected and working together. Each of these have the test that we've written before, right? As we walk through those life touch points, we said we will write these kind of tests, we will write these kind of tests. At each level, those are the tests that are running. Who's responsible for those tests? Each respective team that's gonna be touching on that portion will be responsible for that test. There are times when you will form a task force team, right? A team which is people from three different teams getting together, forming a task force team whose responsibility is to make sure the theme level tests are working fine. A nurse comes in, she wants to see which patient needs immediate attention. That touches across three different teams, right? So let's bring one representative from each team and let them get together and write these tests. Then they would go back to their respective teams and actually implement that functionality. Not after you build a whole thing, get everyone together and say how the heck do we test this, right? Let's get them and say this is the expectation, this is the goal we are trying to meet the nurse when she comes in in the morning into the hospital. She immediately wants to see which patient needs attention, right? That's a workflow. For that workflow, let's bring the people together form a task force or just-in-time team and get them to define what is the expectation from each team level. Then they would go back to their teams and implement that stuff, right? This test now is already there in the system. This would be ignored at this point. If you're using a tool like cucumber or something, it would mark it as ignored. This is not yet working. When the last person commits the code saying, okay, this is working, then it basically turned from ignored to working now and it should now continue working from now onwards, right? So like this you proceed and at some point in most organization there is a cut off, which means no longer you will be able to access your version control and the configuration management team will take over from there. And from there it would essentially go into production, right? So what we want is ideally on a large complex project right from here when somebody checks in, without any manual intervention, going through all of these builds and doing the build promotion and essentially being in production. That's what is continuous deployment, right? So if we do all these different steps, if we have this environment set up, this is obviously the most complicated ones that I have worked in. There are much systems where only there are three environments, right? The smaller the scope of the project, the less of the complexity, the smaller it would be in terms of these different stages, okay? The, so we talked about the cross stream planning, right? Which is where you sit down and you say what? Mm-hmm. We are saying that there is a lot of maturity that needs to come in to be able to work in this model, right? But this is where we want to evolve to because this is what in my belief and this is what I have seen on few projects is how does actually achieve continuous delivery, right? This is a prerequisite if you will. That's okay in my opinion. We get started and we realize that we miss this, we miss this and we inspect and we adapt. That's the agility part of it, right? And we figure out that this is where we want to be. Let's get started today, right? As we go through it, we realize, oops, we completely miss this aspect.  We never thought about that because that would become an issue. That's okay, right? It's not the end of the world. Let's come back and say, okay, how are we gonna handle data across these different teams? Too bad for them. Yes, sir. I think that seems a little bit wrong. It is not, right? These don't have to be same as the production system. I think that's a big misconception. I'll tell you, for example, this one would not even have an actual physical database. This would just have an in-memory database, right? Why? Because can that help me validate what I'm trying to validate at this stage? Do I really need production data with all the information? No, not really, right? Because this test is very focused on what it's trying to do and for that, I don't need the whole system. There's another slide where it actually goes into details of what system is required, whether it's a clean system, a clean environment, it's an incremental environment and that will address a little bit more. But as you actually go through this, your system keeps getting closer and closer to the real production system. I've never worked in a system except some projects that I have built on my laptop and have made money out of it where my laptop is essentially the production environment, right? Or I'm working directly on the server. Those are exceptions. Those will be 1% of the projects that are done, right? Most projects don't have the luxury of having production-like environment even at this stage, but it is not required. I don't think that's required. As you progress through, you wanna get closer and closer to your production system, right? I would say at your UAT level and at your staging level, right? You need to be as close. In fact, I would say at the staging level, it has to be exactly same. For example, you need to have DNS routing if you're doing a web-based project, right? You need to have DNS routing. You need to have all the different infrastructure at the staging environment. And most companies, I think, they do have that some level or the other. Okay. All version control is happening, right? In your version control, or it's happening in your configuration management system. But it's all version control. It's all automated use, puppet use, whatever scripting you want to do. It's not something that we're saying that this is zero manual intervention. Yes? Correct. Let's go back to that, right? If you look at this, where you're working in the UAT level, you're working in the UAT level. If you look at this, when are the integration tests written? We talked about the integration tests are written at this point, right? These are written. You know, you could form a task force team as I was explaining.  So this would actually to some extent drive out your specific stories. You might only be at the epic level at this point. This is essentially drive out the stories for each of the teams that have a touch point, right? So this is where you would write your integration tests. This is, you take it as a part of your sprints. I mean, I'm gonna even talk about many teams we don't do sprints anymore. That's a 19th century thing for me, right? That artificial boundary of doing sprints, we've moved away from that. But I don't suggest you start with that, right? It's an evolution. You get into the rhythm of doing fixed time boxes. Once you get good at it, you can do away with the ceremony, right? Many teams we work, when we actually talk about continuous delivery, we don't have these artificial boundaries of sprints. But when you do have them during your sprint, right? A portion of the time is going towards, you know, doing the special task force or a specific individual from the team is taken out from the capacity for that sprint who is responsible. There are different techniques to handle that. And those are, again, we have a whole bunch of different techniques, but I mean, that's not been a problem per se. So I didn't highlight that fact, okay? Any other questions at this stage? One thing I just realized is, you know, a lot of people say that it takes a, you know, six days to go through this process. And you're like, do you really need to actually simulate the six-day thing? Or can you actually write tests which will essentially give you the same experience, but it'll do the same thing in less than 20 seconds? And the answer is yes, most of the times we're able to find ways of doing that in 20 seconds. Can I be 100% sure that that will not, anything will not go wrong? No. After doing all of this, you can never be 100% sure. After doing all of this, do you not spot bugs in Gmail every now and then? Right? Do you not spot issues every now and then? But are those issues something that will bring down Gmail? No. There are small issues here and there which has slipped out. There's monitoring going on. They figure out that something is, you know, not the convention, and they catch that early on. And then they fix it immediately. So I'd rather have some of those things slightly slip out, right? If you're working for an energy company where even a small mistake is costing you a billion dollars, right? Then can you have a system which will dog food your stuff, right? For example, they have a power plant and they actually have monitoring software which will run on the power plant for their office. And they continuously deliver to that environment.   So you can reduce the impact by trying and finding ways of either dog fooding it or doing other kinds of things. I understand that that might not be possible every single case, but can majority of the things be handled like that, right? I wanna move ahead. Just wanna make sure I've covered everything. So this is the one thing that I was talking. I'm sorry about the projector. It's really hazy to even see anything. So this one actually talks about the objective of each of each build. What is the objective of each build? What are we trying to achieve? What is the job that happens? What are the stuff that happens during this build, right? What is the duration? Who are the stakeholders? What triggers this? What frequency are you triggering this, right? And where is it happening? On what kind of an environment it is happening? Here, I also talk about it's incremental. This is also incremental. Here, we say that there are stubs or marks put in place as we move forward, you know, this thing. And then at some point, we actually do a clean environment. So this is a clean environment. We actually set up the whole dependencies from scratch. We build the whole thing from scratch and we wanna make sure that a clean environment, this goes fine. So you actually go into details of this and it talks about what happens at what stage, whether it's a clean environment, whether it's an incremental environment. So what I mean by incremental environment is, let's say I had a database, right? The last build that was deployed on it, you know, did some changes to the database structure. The next build that's gonna run is not gonna tear apart the whole database and build it from scratch. It's gonna run a migration file which essentially will bring your schema to the level you expect it to be, right? So it's an incremental system. While there will be a system where you would essentially say, I'm gonna tear this whole thing apart and I'm gonna start from scratch. I'm gonna use a virtual machine. I'm gonna deploy this and it's gonna be, you know, exactly what I want, okay? So that's a difference between the, let's see if we can fix that. No? Okay, that's fine. These are really bad from user experience these slides because these are more meant for someone to kind of go into details and look at them, not ideal for pretty picture presentations. But I think these is, I'm kind of running through this as a food of thought and these slides are already available so you can actually go into details of this into much more. I wanted to get to this point and I wanna talk about, some people did highlight issues that they've seen when they're trying to do this. I'm curious to hear if anybody else has tried something like this and what issues they're faced. Anyone? But I saw the frequency that they're trying to talk about when you're doing it. It might not, so there are two ways of slicing stories. One way of slicing stories is from a technical point of view that this is easy for me to build. It's still end to end, but this is easy for me to build. Another way to slice story is that this is the smallest, thinnest slice of things where I will just automate a little bit pieces here and there and that will still give you an end to end experience but might not cover all the cases, might not give you all the sophistication. So one is slicing it around sophistication, one is slicing it around how you build it. So I can still slice it where it's very bare bone. It doesn't give you all the sophistication but it gets someone to experience what the system will feel like. Can I get feedback at that level?  We don't want that, we want that early feedback. And in a lot of systems I have worked where we've actually done continuous deployment where we shipped software 20 times a day, 30 times a day. We only do one small portion of that and we deploy it. It makes that one bottleneck go away in your workflow. Then you attack the next bottleneck, then you attack the next bottleneck and eventually maybe over a month you would have cleared out the whole thing but I can still get feedback and still address a small issue right now. Like one of the systems I built which is an e-learning software for industrial logic where we would deploy things multiple times a day and we would say, okay, now we need to add a new feature which is we have students who are coming on an e-learning course, they're interacting, they're doing a lot of things, they're asking questions and we get emails, we respond to it, we have an SLA that we respond within 24 hours since the email. But we said, you know what, that's a long time. Can we actually have a live chat, right? Where if we are online, they can directly chat with us if they have a question around, you know, what this topic means, we can immediately answer that. That would take us a while to build the whole thing, right? It wouldn't happen in, you know, we were not even doing sprints but it would not happen, you know, immediately. So we said, well, what is the smallest thing I can do and deploy? Guess what we did? Would you like to chat? What we ended up doing something similar is we actually built a fake feature. We said, let's put a chat button over there. 33 users and six experts online, right? And see how many people actually clicked that and see what the response is, right? It turns out over a week that we had it out there, 94% of the people who logged in to the system clicked on that, right? So there's suddenly that curiosity and then it said, would you like to continue? And most people said no. So we said, okay, good enough, let's grab this. We don't need to build this. And then we sent out a survey to all the people who took that, right? And we asked them, why would you not want to do this because isn't it so much more convenient? They said, I don't want to chat with some random person. I want to chat only with my company-specific individual. So then we pivoted and we said, can we do it in that way? And there we got a lot of traction and then we went about building that gradually and it was again piece by piece. It would still send us an email and we would respond back. The email would show up as if it is a live chat, right? So gradually take the infrastructure we have and we slice it. But were we able to deploy it every time we checked in? Yes. I actually would say no because what ends up happening is then you're not getting the feedback, right? Then it still ends up becoming, because that's a slippery slope. We can see where we can slice it, where we can add some feature and as far as slicing this, but we want to get as early feedback as possible because if I keep building it and I'm not activated it, yes, I'm deploying it, but what's the point? I'm not getting any feedback, right? So it's just ceremony now at this point rather than anything else, right? So, yes, that's a problem and we need to find creative solutions to how we can do such that we actually deploy it and we not just ask people for feedback. We actually monitor and we see quantitative data, right? We do A-B tests. You do two different kinds of things and you see what is the reaction, right? That's all kind of going into what we were talking continuous delivery, right? Where it is not about continuously building a whole bunch of software and putting out there, it's about trying to figure out, this is the goal I'm trying to meet, right? What is the cheapest, easiest way for me to meet that goal and can I even do fake features now put it out there, ship it, right? And see if that's something people want. When I go through that, I want to make sure I don't break anything else because a lot of times our experience is I'm only doing a fake feature but I actually break something else, which is where this whole infrastructure becomes important to have in place. Yes, please. Support and maintenance is actually the easiest one in my opinion to put continuous delivery. When I'm building a new product, I have no clue what users want. My minimum viable thing might actually be so big in a lot of cases that getting to continuous delivery will actually be a challenge. So I hear you, right?  So my take is that the system is already working to some extent. If I start doing things to it, I'm not going to make it much worse than what it was before, right? So do I really need to have all that automation to start with? Or can I say we have this, I'm going to only do this one micro check. I'm going to do pair programming. I'm going to do code review.  I'm going to do a whole bunch of manual checks in it before we can actually do to that. But that gives us the confidence of pushing out small things one at a time. So if you don't have any CI framework, which is what I'm assuming, I don't think you need to have this elaborate setup to start with. You incrementally build it as you go along, right? You incrementally build it, but it's easy because I know what I want to expect, right? Which is where I was coming from, right? It's easy because I know what is the eventual setup I want in terms of the infrastructure and things like that. It's hard to put all of those things in place. It might be time-consuming. It might take three months. It might take six months to have this.  So the customer might have done it in the next box. You're doing that in a indoor box, right? That would be a big risk, right? And you wouldn't want to push that risk too late.  Let it run on your virtual environments, all different configuration. Gmail, if you know how many combinations we tested on, it's mind-blowing because it has to matter with what version of the browser you have, what version of the operating system you have, what other plugins you have installed. I once built a plugin for Outlook, right? It's like ancient, but that was very important. And Outlook, you wouldn't believe how many different environments we had because if you have a small plugin in Outlook, that can completely destroy your code, right? It can completely destroy your feature. For the e-learning that I was talking about, we have people using Eclipse, we have people using Visual Studio, we have people using IntelliJ, we have people using all of these, and then operating system combinations, right? We had virtual environments which would have been configured and we would run all of these, right? Did we do that on day one? No, but we gradually built it. We said, which is the most likely environment, which is the most riskiest environment? Let's put that in place. So we'll gradually get there. I think I'm pretty much done except this test-driven development cycle, which I'm sure all of you've seen before. I kind of ran through this, but this kind of explains at a nutshell level how this acceptance test-driven development is built into the development cycle, right? But notice I have something called exploratory testing over here, which is not automated, which is manual. But it happens as and when inside my Sprint, the developer says, I think I am done. I have all my acceptance tests which are automated working, right? Okay, you're saying all the automated tests are working. Is there something that could break from an exploratory point of view? Let's actually do a quick exploratory test on that and verify that, right? So yes, we can automate 100%, but there are things that are important that needs to be checked and that needs to be done as early as possible. It would be a risk-based exploratory test which means you would see what impact it would have and do I really need to do a whole end-to-end one or do I need to just test this and I would be good enough, right? So it's again a risk-based exploratory test, okay? I've blazed through a whole bunch of stuff. It's time for a lunch break, but this kind of gives you at a very high level, not at a very detailed level, how you could possibly look at inverting the testing pyramid to get towards continuous deployment. We didn't really talk about what the system would look like if you're doing continuous deployment yet because this is still talking about how to get there, right? I gave a talk last year which talked about an example of continuous deployment. The video is up online, but that tells you what the system would look like if you are doing continuous deployment and how to get to continuous deployment more from the infrastructure point of view, okay? So I think that's it pretty much the end of my talk. If there are any questions, I'll be happy to take them. Thank you for coming.


Mr Secretary General, your Excellencies, Ministers, distinguished delegates, ladies and gentlemen. Let me begin by thanking you all for the confidence and trust you proposed in me and it is a great honour to serve as the chair of the WTP of 21, the first we're having in eight years since the last meeting in 2013. I'm truly grateful to members of the informal working group, led by Chairman Mr Fabio Bigi, who have worked tirelessly over the past two years or so to present this forum with visionary draft opinions. I'm also mindful of the high expectations attached to the new and emerging technologies at the centre of our discussions. I was looking forward to seeing all of you again after all this time. As the WTPF was to have been the first physical ITU meeting since the pandemic struck, but that was not to be. COVID-19 may have prevented us from coming together in person in Geneva, but you can rest assured that it won't prevent us from discussing the important topics that are on our agenda. We will do it for all the digitally excluded, the women, men and young people everywhere who are still not using the internet, let alone the new and emerging technologies that hold, as Secretary General Zal said, so much promise for all humanity. And we owe it also to ourselves to do our utmost to harness these technologies effectively to accelerate the achievement of the 2030 agenda for sustainable development. Because when poverty is on the rise, when our children's education is on the line, when our planet is suffering, when young people have no hope of getting sustainable jobs. When our health is impaired with existing and new diseases, we believe that these innovative technologies can and will make a big difference in people's lives. I'm looking forward to constructive deliberations always in the spirit of consensus building and cooperation that characterizes our work in IT and to a very successful form in the service of humanity and promoting sustainable development. Let's make history together. Thank you.


Hello and welcome to the session. This is Professor Farhad in which we will discuss the integrated audit What is the integrated audit? It's when the when the auditor audit the financial statement and issue an opinion, which is the traditional audit also ordered the internal control over financial reporting and also issue an opinion in This session about the integrated audit We would look at testing and evaluating the design and operation of the internal control over financial reporting In the prior session, we looked at what we call the top-down approach Remember our objective as auditors in an integrated audit is to issue the opinion about the internal control over financial reporting What steps do we undertake? We plan the engagement. We use top-down approach, which we already covered We test and evaluate the design of the internal control over financial reporting We test and evaluate the operating effectiveness. So we're gonna Test the design or evaluate the design test and evaluate the operating effectiveness So I decided to cover these two steps in this session Then at the end we will form an opinion, which is our end the product Which is that's our objective about the effectiveness of internal control Before we proceed any further. I have a public announcement about my company farhad lectures comm Farhad accounting lectures is a supplemental educational tool That's gonna help you with your CPA exam preparation as well as your accounting courses My CPA material is aligned with your CPA review course such as Becker, Roger, Wiley, Gleam, Miles My accounting courses are aligned with your accounting courses broken down by chapter and topics. My resources consist of Lectures, multiple choice questions, true-false questions, as well as exercises. Go ahead start your free trial today No obligation, no credit card required Let's start by looking at testing and evaluating the design of internal control. What is the design? What do we mean? What do we mean by the word design? Well, simply put What does the internal control looks like on a piece of paper? Simply put you are looking at the map You are not doing anything. You are not taking the trip when you look at a map You're just looking to see where you're going. You're not actually going anywhere. Let's talk about something even simpler Let's assume you're studying for the CPA exam. So that's your plan plan is to study You will have a design for that plan and what's the design at the design to read the notes watch the lectures solve MCQ's and Solve simulations. So that's the design. That's the plan now if you do so You might pass but that's how the design but it doesn't mean you are going to do that The design is just basically what you put down on a piece of paper Now, why do we need to learn about the design? Because we need to learn about the design or that or test the design effectiveness and We assume after we learn about the design if that design is working properly It should achieve the company's control objective and effectively prevent and detect errors on fraud that could lead to material mistakes All what we are doing first is we want to see is the design good Maybe the design is no good right from the get go and then you have you have a serious issue here But first we need to learn about the design. Now, what procedures do you have to undertake to learn about the design? One procedure you can use is called walkthrough and basically tracing transactions starting with a transaction from the origination through the company information system Till it appears on the company's financial statement. So taking the Taking the statement from A to Z now the auditor can do this themselves or with the help of the internal With the help of the internal auditors. So the external alone or they can get the help of the internal auditor During this process, they might conduct inquiries talk the management inquiry of personnel observe operation Sometimes they let you know they're observing sometimes they don't and inspection of documents and all what they're doing now is learning is Learning about the design of the internal control. That's all what they're doing But that's not good enough. Why because you can have a great internal control on a piece of paper But if it doesn't work properly that it's useless. So now what you want to know is is it operating? effectively So is the internal control over financial reporting actually working as design? Is it operating effectively if we go back to your CPA plan? Are you going to actually read the notes? Watch the lecture solve the MCQs and the Sims? That's what you said you you want to do that was the design that was the map But are you actually doing it? Maybe you're not reading the notes Maybe you're not watching the lecture. You're just going into the solving MCQs or something like that So you are not following What you said you will do but if you do so you should pass or you should learn the material you should be prepared You should doesn't mean you will pass, but you should be prepared. You should pass eventually Okay, so why test the operating effectiveness because okay, we learned about it now We need to increase our conviction that the internal control over financial reporting is actually working We know by design. It's good if working properly But we need to know whether it's working properly or not When we test the internal control over financial reporting We need to be familiar with three terms the nature of the testing Timing of the testing and extent which is basically very similar to the testing that we learn about in your audit Regular audit which is auditing financial statement. The nature is basically the type of test What type of test can we undertake? Well, we already learn about inquiries asking questions Observation inspection and we do this through in the learning stage. We would also do this during the testing stage Why not we can do it again? In addition, we have to do Repreformance recalculation of the application of the controls now We have to test it actually go through it and run some numbers and see if it's actually working as expected If they're saying they're doing they're getting the approval for that sales order see if they're getting the approval See if they're properly computing the sales tax see if they're properly computing the sales tax go through it For example, if you want to test completeness for recorded invoices Maybe you could use a general audit software something like idea or program Excel to find out if there's any gaps in the shipping documents Now you're actually testing the control Well, you looked at their shipping Design everything looks good. They have a good shipping system. But is it actually working? You will test it Now bear in mind you have to know that inquiries alone are not sufficient to test the operating effectiveness inquiries are good to learn But not the test you could use it during testing that it's by itself is not enough So that's the nature of the test Timing timing basically answer when are you going to test? Okay, it should be over a period of time. Okay Now bear in mind we have certain routine transactions transaction that repeatedly occur and certain transaction or events that are not routine for routine transaction, we would assume that the control is continuously operated as Tested so for example, if you have sales transaction through a computer system You would assume if it's you tested it for a week or a day and it's a computer system and there is no changes You would assume it's routine. It should it should be it should be functioning properly You might have non routine events for example preparing monthly or quarterly financial statements You really cannot test until after they're done to see how well the process went now You have to introduce unpredictability during the process. You cannot tell Management when are you gonna be testing because if you do well, they might Be a little bit more alert and change their method. So that's how when I said about observation Sometime when you want to conduct observation, don't let them know because when people know they are being observed They might behave differently So that's timing extent of the test Well, how much are you going to test like basically the volume? How much are you going to test and the answer is to obtain sufficient and appropriate evidence? Why because you need to provide a high level of assurance that the internal control related to the assertions are operating effectively Now again, we're still talking in general terms and we're gonna keep talking in general terms But basically what you have to know is this if the control is manual. It's not automated It's gonna require more work because automated controls once, you know, they're working properly. You test them once a Computer system process something properly. It's programmed to do it at all.  So you have to do a little bit more testing The more important the control the more testing it requires so if a control is important and we should know this because Testing internal control is a risk risk based approach. The more important the more risky the more work you would give it The more frequent the control used the more the testing as well. Okay, so also the more Frequently we use this control because we use it continuously. You want to make sure it's working It's not a one-time thing and you cannot accept less than persuasive evidence So the evidence has to be persuasive how well has to be sufficient and appropriate how to provide high level of assurance How follow these general terms? So this it's a judgment at the end of the day Now we have to understand Because we do test of controls when we audit the financial statements and we do test of controls for integrated audit So we need to understand the differences between the two because you could be asked a question Okay, first the purpose the purpose is different the purpose of Audited financial statement is to assess control risk Why do we need to assess control risk to decide whether we're gonna rely on this control or not rely on this control? Remember we have low control risk and if there's a low low control risk that means we're gonna rely We're gonna obtain evidence to see if that control was operating during the period We are auditing if it's a high control risk. We don't rely. We don't test anything so that's why that's the purpose of auditing financial statement when you are not issuing a report on The controls over financial reporting now if you are issuing a report about Internal control over financial reporting, which is you are conducting an integrated audit The purpose is different the purpose to show an opinion about the effectiveness of internal control taken as a whole to determine Whether it's fairly stated at a point in time. So the the purpose is different Because the purpose is different the scope is different and the procedures are more extensive The scope is definitely different. Maybe the procedures could be similar inquiries observation, but they're more extensive So to answer the question, it's different. They're totally different for purpose scope and procedures. We say they're different Just make sure you know the difference and hopefully the slide kind of explained the difference between The test of control that you learn about during the regular audit the audit for financial statement and the integrated audit is totally Different now we are ready now to form an opinion on the effectiveness of internal control and this is gonna be heavy-duty stuff Forming an opinion and issuing actually the report itself. What should you do now? Go to far hat lectures and look at MCQ's That's gonna help you Understand this concept better in this in this concept. They test you Not only there's some memorization here But if you understand the purpose the overall picture you have to use less memory But if you have to use your memory 100% You'll be in trouble because you're gonna have to learn about this about non attestation services many many different reports So that's why understanding is totally The right thing to do for the CPA exam because think about how many reports you have to deal with reviews Compilations and DNA Suck report should I keep going? So there's a lot of the point is There's a lot of reports relying on your memory Big mistake. There's agreed upon procedure prospective information for former financial information Okay, so bear in mind that memory does not work for the CPA exam Understanding appreciating will work much much better. I'm always here to help you study Good luck Attack each topic one at a time Understand it and assume that's the only topic you're studying for good luck