All right, so hopefully that kind of gives some light into why JP Morgan is kind of interested in this conference And you know, we would love to hear more from other companies as well who are going through this transformation We do this public submission system where we try and get as many speakers locally internationally to share this story next what I want to do switching a little bit of gears is kind of We've we had actually a keynote speaker today, which was Jim high Smith. Sorry Jim high Smith was not a keynote speaker. It was Jim McCarthy. Who was a keynote speaker? Jim McCarthy could not get a visa to India. So his trip was cancelled last minute But we thought we'll take this opportunity to bring some of our other speakers on stage and do You know like a panel where people can ask questions and each panelist will get a chance to take a stab at it We'll I'll keep a strict time on each question and I'll time out For each question not more than five minutes. Every speaker will get a chance whoever wants to so let's welcome Arlo Bell. She Ravi Ravi Kumar Ravi's the team chair We have doc and Chad all of these guys have worked with a lot of distributed teams and In my opinion have a lot of great insights in in you know How distributed teams work and how agile teams in general work? So I would you know be delighted to hear their opinions of things that could not get addressed today things that you still have in mind So anyone wants to take a stab. We did have a theme for the panel Which was more of you know offshoring and agile, you know, are they kind of oxymoron are they really? You know that can they work together? That's really the theme, but we will see how it goes and also if When someone puts up the hand and ask a question if that you know if you feel that the speakers could not answer that well Then we welcome you to take one of the chairs And you anybody from you can come pick up that chair and answer the question All right, and I would love to see those two chairs filled We will use a fishbowl format. I'll just quickly explain what a fishbowl format is So these guys are fixed we put glue on their seats and they're stuck They cannot get up, but those two chairs don't have glue on it So when we have someone from the audience who wants to Give an answer to the question you come grab a seat Someone else wants to give an answer can also come grab another seat But the rule of a fishbowl is one of the chairs have to be empty So if the second person comes and sit then the first person has to leave In terms of after answering the question after answering the question. All right, is it clear if not we'll figure it out So who wants to be the lucky one who wants to get the first question right there go Can we have the mic please? It's cost reduction if my primary intent of distributing or offshoring Will Agile work there? That's something that I'm trying to understand So let me repeat the question So if cost reduction is the primary factor for distributing the project for offshoring the project Will Agile work in that kind of a context? Amler has some good data about this from several years back. He was looking at Distributed teams versus co-located teams and comparing that to Agile and non Agile So there are four possible combinations And it's worth pointing out. He was looking at adaptive iterative not just Agile So nuance that doesn't make much difference here, but but includes all the Agile ones So in that mechanism or in that Organization you found a fairly strong swerve in terms of productivity in terms of how much they would deliver for a particular unit of cost He found that co-located Agile teams had the highest productivity And then below that at some ratio, I'll just say 80% I don't know what the number is is co-located traditional teams And then there was a huge gap and then there were distributed Agile teams and then distributed traditional teams And so if you are distributing anyway agility is still a benefit However, co-locating has more of an effect than switching from traditional to Agile in terms of productivity Yeah, go ahead. So I Distribute for my cost reduction because my cost is less there and then I go back to the Off-show team and tell them that You know, I've started just you know giving you work now get me better productivity because I still want to reduce the cost And I don't want to invest on you know I don't want to increase my cost out there because I'll get I want to get more value from that The center, you know, how do we tackle that kind of a situation? They want more value, but they're resistant to cost How do you tackle that? So we'll pause on that question We'll let the the speakers respond to your first question Which is you know, uh, you know if if cost is the factor, would you go distributed and distributed agile specifically? So What I think is What what I think is typical is there is a cost productivity trade-off, right? So what do you mean by cost, right? Is it hard dollars because the productivity? Hit that you take actually is a cost as well and sometimes it is worse for you In terms of if you would do a team that's co-located and together Versus what the productivity hit was now we do a lot of distributed agile projects And we found ways to make it work and we actually help our customers do it But some of the things we find is can you Take the distribution and still make it kind of co-located like so you're talking about how you could empower those teams Well, is there more work that they can do together as a unit versus You know 50 percent of people being over here 50 percent over here and they have to figure out how to collaborate So I think it it the parameters Um Really matter in terms of you know what the cost outlay is how your teams are working to get that The proper benefits that are That makes sense I think Cost yes, you can reduce it But so long as you have certainty in what you are expecting So if you have got requirement, there is certainty and you want to drive low on cost Yes, you can outsource it or Distributed and get that But if something is on an emergent nature you are trying it out and then expect Cost also to decrease while you are distributing then I don't think it would work out They don't go linearly in that way So you'll have to know what is it that you are distributing and why is it that you are distributing and what's the outcome that you are expecting So that would be the fundamental basis. Why would you want to try that? Hey, hi My question is In your own experience What kind of fixed price contract models have you seen succeeding in an agile outsourced projects? What kind of fixed price agile outsourcing contracts you have seen work Essentially the key factors on which this fixed price contracts were built which you saw being successful Especially in the outsource model Yeah, so You know the projects that i've done where it was, you know fixed cost and and and we were providing You know the actual staffing to do the work Really the way that you know, we made it to be successful was um focusing on value focusing on Um, you know being able to to flex the scope of delivery So we can we can guarantee that we're going to have something of high value by that timeline and with it You know within cost, but that does mean that you know, we've got to be a lot more Open about the scope and we've got to be disciplined about making sure that what we're building is You know high value stuff and you know that we're not over architecting etc And in getting something out there fast and iterative So, you know if the if the cost is fixed and the timeline is fixed then scopes got flexed in some way To me Fixed price is an evil It's a crime You cannot be doing that. I mean in the in the sense that how it is being Distributed or offshore today because everything it's from a cost perspective You want to drive it and Often times not often time most of the time in my experience. I've seen it's always been the iron triangle It's always going to be fixed fast fixed scope and your fixed quality etc. No, there are there has to be a trade off And the moment you see fixed price and trying to drive down that aspect somewhere There's a waterfall trap because vendors are also going to hit back and say okay Give me something of certain then only I'm going to commit to this So then you are killing agility You can still be running your project under the banner of a jail, but I don't think truly you will be a child And just a quick additional point. I think rarely is scope actually fixed in any way So When you get into a space where the scope is fixed say there's a Master story list that you've agreed upon delivering rarely is that really the list And that'd be a good thing from an agile perspective, right? Let's deliver the highest value first, but it's like oh, but what's in the contract is this master story list So then you end up trying to deliver the value and the list Which that's a trap as well. So I think if you're going to do a fixed Price thing you have to be very careful about the prioritization parameters and the scope parameters in particular because It usually up front. You don't know what you're going to build. That's the point It's it's all about the feedback loops and and being able to use those to maximize value That's the reason we do this whole agility thing Feedback's only useful if you integrate it into what you're going to do. Otherwise, it's just hopes and dreams and wishes So to include that you want to give yourself options in the contract Toyota's variable scope contracting is a great mechanism for that It does a good job of lining so that both teams win if scope is decreased So, you know, there are techniques like that that you can write into the contract so that you can incorporate future feedback and learning And I recommend pitching it that way as we want to include learnings that we gain over the project in order to drive the costs down and get more value Yeah, there's a lot of organizations have multiple divisions, right and and You know, you've got r&d and you've got production And I think that one of the challenges that we face one of the things that one of the changes that we could Look at making that would really help is we look at software development as r&d as opposed to as production And if funding were done in that way, I think it would change a lot of this a lot of these questions, right? It would change the context of the entire conversation And I think that's a much more appropriate paradigm for software development Thanks Go go there. No one from the audience wants to add anything in Is there's two seats still left open? So there was a time when xp was I think we need to just hold on to your question. That gentleman has been waiting for a while. We'll come next to you. Sorry evening all uh My question is related to release planning Uh, often I mean I come across project where I'm I'm given a prd document and uh, people say, I mean They just say that, you know, I'm okay with whatever mechanism or Approach you want to take to execute this but I need a date by which I can come You know communicate back to the client or the customer as to you know, let's say 30th of May you're going to get this How do we go about this because if you if you want to arrive at a date then we are going to calculate velocity and then you know in In a way effort and then days and months and that that kind of thing. So Is there is there a silver bullet to you know, actually Do that So I think um From my personal experience, how much is the certainty that you have on hand To actually go back to become a customer and say that I can give you by this date if you ask for a date If you don't have certainty, then you obviously cannot commit for it At a at a high level. Yes, you can and that also is a high level expectation from them. That that is that is okay Uh, they would want to go reprogressing it in that direction and if it's on an emergent evolution kind of a thing Yes, it's perfectly valid to be there and you can benchmark against that and progress but the point is uh, if you are Giving me a high degree of certainty and then expect driving it then again if you are getting into the trap So it's not it's again killing back going back on your agility In my mind, there is a silver bullet. Um, and that silver bullet is data data data and more data So, uh, there are teams that make estimates by asking people for numbers of how many days it's going to take That's a fairly data free way to make estimates And you don't have a high degree of predictability In your results the teams that do it by Just counting the number of stories or maybe estimating the their complexity in a unit and then measuring a velocity Tracking that over time checking for your variance and velocity as well as the means using that to predict measuring the The incoming rate and outgoing rate of stories Outgoing being stories you choose not to implement When you start incorporating that sort of data into uh your analysis of of what's going on in the project Then you can start making very good predictions. And in fact, you can start making predictive models that I can then Uh rather than me making prediction for you I will hand you the model and say go ahead and adjust all the backlogs However, you want freely in this model and it will continue to make predictions of What you're likely to get given those decisions and then pick the right decisions So data Two points. One is uh, I think you can make escalating commitments, right? You can say At this point, I can give you this broad brush stroke with this level of certainty So you can create range of date and certainty And as you progress with data You you can tighten up the bounds of that second thing that I find really useful is understanding cycle time So if you understand From soup to nuts, okay This is a story these are how long it takes me to do this from soup to nuts This is how much work and process I typically carry That I can carry that's good for my team You can kind of look at it and make a very good guess from there because you know, okay My cycle time is 15 days and I haven't started it So there's no way I'm going to do it in 13 Right. So I think that's a very useful mechanism to start to to make those choices Yeah, yeah, absolutely. I agree So, you know when I I'm working with teams a lot of times we go in and and some will say We've got to get this team to be faster, right? Um, and a lot of times the first step is to actually slow them down Um, step back take a look at cycle time Take a look at lead time take a look at, you know, all of the data that you can actually gather from that team And figure out what the root causes are to actually stabilize the throughput once you've got that predictability Then you can actually start making, you know, some level of commitment with certainty You know, we'll do burns with standard deviation rather than just a standard burn So it shows what the range is of, you know, our confidence interval against that But you know, if the team doesn't isn't isn't stabilized first One you can't actually get them to go faster because it's There's there's too much gunk in the works and you can't make those kinds of of commitments not not with any kind of certainty So I'll take a stab at that question as well. I'll jump in from So I think this is the first I think Jeff Patton was kind of explaining this model and I thought it was pretty interesting So he talked about the you know, trying to look at the iron triangle And he said if you look at the iron triangle, you have basically your scope your cost and your time And typically people would use Uh, you know, this is fixed the the scope is fixed, right? And then you would use uh estimation to figure out what would be the cost and how long it will take Right and then he talks about in the traditional models. This, you know, is kind of what we focused on But if you look at, uh, you know, what some of the agile methods are trying to do is they're actually trying to invert this Triangle and they're trying to say well, let's fix the Cost and the time In in other words, you you give me a deadline by when something needs to be delivered, right? And we'll fix the cost because we We have a team of six people working on this or a fixed size of team working on it So we know the cost is fixed for that duration So the cost and time is basically fixed and then you basically estimate the scope what that means is you you you Work in a short cycle You work at regular heartbeat and then you see measure how much you're able to achieve in that short duration And then do constant projection going forward to estimate what scope you will be able to cover in the given time Or in other words, you're kind of trying to uh, also budget things saying, you know This is how much i'm willing to invest on this kind of a thing and then, you know We will play around with the levels of sophistication How sophisticated something will be or how less sophisticated it will be depending on your budget So that's uh, in my opinion, that's kind of a very different perspective of looking at, you know, this this whole iron triangle problem All right, we'll go down there So there was a time when xp was uh, quite popular Extreme programming is one of the agile flavors. Um, at least I haven't heard about that being practiced anywhere Hello, sorry. So there was a time when um xp was quite popular as one of the agile flavors. I mean probably couple of years back Now off late. I haven't heard any of the organizations at least to the best of my knowledge practicing xp So in your experience, have you seen xp being practiced still or is it that scrum or others are kind of defective for practicing? So, uh, as a strongly xp biased person I'm going to answer accordingly. Um, so scrum has a number of very strong advantages over xp. Um, it is marketable Um, it is easily transmittable Someone who has done scrum for a short period of time Is able to transmit that knowledge to somebody else And therefore it has a much shorter doubling time in the marketplace So it's not that xp has decreased the amount of xp in use is actually increased But is increased with a very slow doubling period. It takes a very long time to to spread scrum has been driving all of the agile adoptions for a while because it's marketed well well And it's easy to transmit and so as we start getting more and more agile visibility. It's really a lot more scrum visibility But we are still finding that The teams that are really delivering well If you want to get to two star fluency xp is still the game that that gets you there Scrum is a great way to get to one star fluency We see a lot of teams get there and then fall into certain technical debt traps and then go look for xp So one of the things that i've that i've seen happening is in a way scrum is consuming xp You know, so i was involved in Creating some of the scrum certified developer curriculum and teaching it a couple years back And it was a scrum course and it was a scrum certification, but Throughout that course we were talking about pair programming You know test driven development continuous integration You know, we were talking about all of these you know, many of these practices that actually came from You know xp And just kind of folding them into You know into scrum training. So it's a little convoluted these days a lot more than it was I would say You know four or five years ago I think there's a bifurcation of people thinking about scrum in terms of project management and analysis and There's probably an overlap to the extent that xp did talk about stories and using stories and pushing those through But that now when you hear people say i'm doing engineering practices as the the folks Forget what company that was up here Yesterday said, you know, we're moving from doing all scrums and now we're doing engineering practices. Essentially they're adding xp So people people might not be saying that they're adding xp And the word also became dirty for a little while because people said extreme programming Why did you name it that so now people say i'm doing agile engineering practices. They're doing xp So xp is is alive and well and flourishing I've actually not been to a single organization or a team which is doing code in code scrum And is not doing mini waterfalls I have not seen a single company in 12 30 years of my You know experience with agile who's doing code in code scrum and is not actually practicing mini waterfalls This is essentially mini waterfalls every month Or two weeks if they can sustain it one month itself is hard Two weeks is burning people out All right We'll go Yeah, there's a gentleman who's already got the mic and we'll go there next Good evening everybody. My question is this is going to sound a little bit skeptical Last two days would be my first exposure to agile and hearing lots about agile And I've been hearing nothing but um, almost like a nirvana state, right? Everything is going to be great customer gets early We are going to have less surprise and low cost self-organizing and it list goes on My question is if this is so true and is and and I believe so based on what I heard What stops some of our executives, you know, I assume all the ct os and c o's are the best in that Why isn't that um, there is a big of jump into this bandwagon and we see almost agile development across the board Versus what we are seeing today just more waterfall and then less of agile So I would like to hear a little bit more on what do you think is some of the blockers from the executive standpoint? Are there some of the blockers for adoption of agile? So I think that this is a highly culturally nuanced question So the blockers you're going to see in the us are totally different than the blockers you're going to see in india And the rate of adoption that you're seeing in in the cultures are going to be entirely different Like in the us at this point people are talking about yeah, it's pretty much everyone's doing agile Only the laggards are left and they're already flipping The waterfall traditional stuff is mostly dead And the challenge is that a lot of at this point the transitions are in name only They're doing the same things they were doing under waterfall which weren't waterfall anyway And now they're calling it Agile just as they were previously calling it waterfall. It's really cowboy So it's it's not that people are You know are avoiding the agile transitions. It's that they're we're down to the point that it's the people who are white labeling Whatever it is that they're doing anyway That's not that's totally different when you go to different cultures One of the big things that I think is is a driving part of that is Agile done well fundamentally decentralizes power Uh it fundamentally shifts from central decision making to much more distributed decision making And if status is associated with who makes what decisions Then that is a shift of status power and influence all at the same time And that has huge cultural ramifications In companies and uh creates quite a lot of resistance and back in the the days when agile wasn't the dominant We very often saw agile show up in one or two portions of the company And then the antibodies would come back and come out and destroy it. It was because it was decentralizing that power So to me, I mean piggybacking on what Arlo said um My experience has been I I've worked some time in the u.s Uh while I'm from India I worked some time in the u.s And that cultural thing what manifests in the organization makes a huge difference We never had agile as in agile as a banner, but then there was always customer collaboration working with the business Uh, I had not read about agile at that point in time, but I remember The manager making a statement if something goes down in production, then I'll get to know I'll come and talk to you Otherwise business will not complain to me. You are doing a good job Right, so that was a perspective coming back to India there was a whole lot of process and all those things and The experience was that we never did a perfect waterfall In India while we talked so much about waterfall and things like that. We never did perfect waterfall Because it was always chaotic Requirements was never frozen. In fact, we got to know the requirements only in uit The point is when you go to a waterfall and it's kind of I have I think stopped making a comparison between waterfall and agile That way but then the good aspect of agile what has to be taken in or inculcated is the the value for the system and what Arlow was mentioning the decentralization aspect You cannot be chaotic and then say that is agile Which is again a misnomer there. It requires high discipline to be agile And it also requires some of the authority and etc The powers to be delegated back to the teams and they can govern themselves So that takes a fundamental shift and we are not in that kind of culture if I have to talk We are more control oriented as opposed to giving off that so I'm going to say something that might be slightly offensive, but I'll say it anyway You have to just be yourself Jared So I think there's actually what I call a generational lag in technology management And that is this generation of managers was last generation of technologies developers Right that makes sense right and so Oftentimes you have people who are managing now who are disconnected from What's the current state of practice both in technology and in process and When you have to explain to someone who you know Cracked their their life on cobalt mainframes and their worldview is developing on that And then you're saying pairing unit testing all these things that are kind of foreign It's hard to make the intuitively that all these things are wonderful and I should do them Right. I had a conversation with a business leader about continuous integration once many years ago And we went through this whole thing about continuous integration and why it was wonderful And she said at the end. Okay. Yeah. Yeah I'm just trying to make my merge go from eight weeks to four weeks So how can you help me with that? And I said but there would be no merge and she's like, I don't believe that that's crazy talk I'm like, no, no, no people do this all over the world And she was like, you got to be kidding me This make I want four week merge forget that see eyes. I don't believe And I think that's um, it's hard for people to crack Yeah, so uh, I'll we because we have just started our agile transformation a year back And we are a 260,000 people organization and thousands of people in technology So the first thing I think that the organization has to believe that that is pain We we are in a banks are in a highly regulated environment Regulations are changing almost every day And we realize that our technology is not agile There was another problem to to make it Make the organization agile what we realized it that most of the people They stopped learning the moment they joined the bank. Basically we we learned through our college But once we join a company That's when we stop learning that is we don't pick up those books Well, there are of course passionate technologists like the folks sitting here But mostly in most companies people don't learn So what we have decided is that we first want to enable a learning organization And and agile is very very difficult to get it right, especially in a large organization for a small company like whatsapp It's very easy. Uh, I have worked for a startup for 10 years in the u.s. So I know but but here The to implement agile first things is those managers Have to go and sit with the teams and work with the teams in implementing those best practices Well, I would even say it as good practices because you evolve them And uh to to just tell a team to implement continuous integration It's not going to work to make them believe that continuous integration is like breathing air It takes time and it's going to take a long time For any organization that starts with so we have started at the sea level first So when we hired this World famous coach in lean agile the first thing he said is I am not going to speak to anybody but the sea level We are going to first spend time a lot of time with the sea levels We are going to coach them and then we'll go Down the organization. So It has to start the coaching has to start started the top not at the bottom Yeah, so The reason I came over here was because I went through that transition from, you know, doing sdlc to agile and I've also spoken on that So typically we what we've noticed is especially if you are in a service provider setup In many cases, uh, the customer would say we want to go agile But why they want to go agile? And what agile actually requires are two different things. They want to go agile because they want things faster And and in this case when I say customer it could be the management, but they don't realize that they need to change as well So it's it's like we need to go agile. I need things in two weeks And you guys go and do it. So that's the first problem Expecting that we don't need to change You know, you guys change and then start and the second thing is investment So there is a process side. It's sometimes easy, you know, you can start stand-ups What's a big deal? People just stand up. You can do a retros. What's a big deal? You know, give an hour do it but True transformation happens when you when you have to invest in the platform, which is, you know, continuous integration Automation test maybe performance test. You actually need to pay money to get licenses in these cases. You probably need to get hardware People don't think about it. You don't actually put in that initial investment to make those transformations So you expect with a stand-up and a retro for the teams to just transform and start doing things You expect that teams will do showcases With just a stand-up and a retro being instituted that doesn't work That's been the typical challenge that we always have All right I want to add to that basically I have two perspectives, right? So you were saying what is stopping companies from jumping on this brand wagon? I see every company jumping on this brand wagon Tell me a company which is not jumping on the brand wagon I can't think of any. I mean I've spoken to so many companies. Everyone's jump on the brand wagon A lot of them are dropping off the brand wagon as well It's it's 40 years now. And I think it's pretty Old in my opinion If you look at a lot of Indian services company, they are jumping on this brand wagon for one specific reason Any guesses? No Customer required it if I do it. I get business if I don't do it. I don't get business I did cmm before this. I did something else before this. I'll do now agile, right? Send 10 people get csm certified send 10 people get some other certified show that as the Profiles we have in the proposal that we send and there we get a project, right? So a lot of services organizations in my opinion are falling into that trap, right? So I think to your question. I am not aware of a single organization I'm sure there are but I'm not aware of a single organization. Which is not jumped on this brand wagon right I'm talking about services organization now. Let's look at India has a huge number of captive product companies, right? So you have a product company in you know in europe or in us or in australia And then they have an offshore development center or an odc over here, uh, you know And why are those guys jumping on the brand wagon? Because they want to bring more control over here They want to have better say in how things are being done, right? So that's another direct motivation for people to kind of Show that we have agile skills and trying to get better work instead of just doing glorified generator's job Right, so there is that other factor for people trying to do Bring a lot of agile into the organization. So in my opinion, I don't see what you're saying as Happening at all. I see the exact opposite happening I don't have the numbers if top of my head. I would have to say 90 percent of organizations claim to be doing agile 90 percent of organizations claim to be doing agile Doing agile itself is an oxymoron in my opinion. Uh, if you really want to go more into it, I would strongly recommend watching Andy Hunt's webinar that we did before the conference where he talks about, you know, the day we wrote the manifesto It was dead for him Because he says that, you know, it was essentially We try to make this as a static thing while it was something dynamic heading somewhere that we didn't actually know And we've kind of made it very static now So if you want to see the numbers, um There are two reports state of agile report and state of scrum report And they have a whole bunch of data on who's doing what and how many Uh of all it projects are are in there or whatever. I don't have the numbers off the top of my fingers But lots of people at this point are at least claiming to to do those practices Right and there's a lot of fudge for example Historically continuous integration on those reports is like 50 percent When you dig in into it with people and you say are you doing continuous integration? Yeah But my nightly build is running just fine No, that's still better, right? I have the ci server sitting there Right. We are doing continuous integration because the server is there Right. How often do you check in? Uh Once before the end of the sprint Right That I'll be lucky. Yes, we'll go there Hey guys the so it's quite clear that agile is certainly booming. We now have frameworks to scale agile Thanks for the sarcasm So, uh, my question is uh, what's the panel's thoughts on the scaled agile framework? You want a public answer to this question? You're outing us here So, um So, okay, first I'm gonna gonna Again state my biases. Uh, so I work at microsoft. I do uh company transitions all over the place. Um, You know, I when I do a pilot a pilot is 150 people at a time. That's just a small thing you try on the edge Right. Um, so when I'm talking about scale, I'm thinking large scale Um, so I'm not going to propose an alternative Way to to handle scale until it works at least 600 700 people at a time Um, from the conversations that I've had with Alan shallow way on scaled out of a framework It's really well targeted at the 150 person size for which I say well, that's interesting. It's a pretty good pilot Let's see whether it works now. Go ahead and see whether it works. I'm not going to bring it in in my case Um, there are some things that it says Are important, but I don't feel this is enough emphasis on those and in particular The technical practices If you're doing the technical practices if you're doing xp like or adaptive engineering at those teams You just drive a lot of risk and cost out And when you drive out a lot of that risk and cost then you have fewer problems to solve at the higher level There's a whole lot of weight in that in that framework that's There to prevent Screw up cascades to prevent an error in one team from cascading to another and another and another Um, if you're doing the technical practices You just don't have as many screw ups and each team has more ability to recover from a dependency screwing up So you don't need to prevent those cascades as as much and it makes me wonder why you have so much Heavyweight process at the top for dealing with what can be a non problem if you simply solve it to me I I I cannot give a direct answer as I log here because my work is in a different kind of an organization It's predominantly it services. I really don't see a point of scaled frameworks in an it service organization We are so dynamic There are multitude of complexities multitude of service lines and industries that you support I really don't see how exactly can you scale other than the verbates that you go into carry with you Because fundamentally context matter and these contexts are so much of variability in those contexts And when you look at scale and things like that, there has to be a certain amount of standardization vocabulary and those kind of things all put together And commitment from the management and also the customer There are too many ants and I see lots of ifs and ifs says if they work then it happens I don't see the first if happening So I don't see how it would actually do that We can turn off the video I think obviously there's a need to give people And I said this in my process talk about an hour ago guideposts on thinking about how to think through things But I'm I'm suspicious of any Framework in general that says hey to be successful you should work like me because 10 other people have done it So I'm biased there and I'll put that out There are a couple other issues one is I think the skilled azure framework doesn't say enough of what not to do So there's a lot of stuff in the framework to do this do this There's some heavyweight stuff in there, but there's very little don't do this And my what I've seen observed is that organizations with bad practices And bad ways of thinking that want to scale agile can wiggle bad behavior into lots of the little Nooks and crannies of the framework, right? So you'll see people do things like analysis decomposition Which the framework talks about in a very nice way about you know at this level do this at this level do this But then you see people where essentially at some high level of the Organization the analysis is what's law and then it's being pushed down and shoved to teams And at no point do people say this is bad Don't do this kind of thing as part of The framework itself. It's so heavy on evangelization at this point Do this get out of there is great that people aren't saying do the stale azure framework, but don't do these bad things The other bit that Concerns me is I think there's some There's some voodoo math And how certain things are set up so the normalized estimates across teams It's just I look at it and go First of all, you can't normalize estimates across team getting people to estimate the same way Is a fool's errand in my Opinion and even if you could The point of it in the framework is to help you make ROI Judgments about what to do next That would assume that not only the estimation itself is consistent, but the execution Is consistent So if I estimate something as a 30 the other people estimate something as a 30 We both on average have to do it as a 30 for it to make sense at the ROI level But if you do it as a 15 consistently and I do it as a 45 consistently Well, then Did my estimates mean anything? So I think you know and you see this too with people doing Scrum at scale and rolling up release Burndowns across story points of many teams. There's just lots of stuff where you're like Man, did anybody really think about the math of that? So I think that there's some Things like that and scale that as a framework that I don't think make a lot of sense either Overall I think there's some good nuggets and some good things for people to learn But it's not something I would like be excited about so So also, you know, I haven't seen the scaled agile framework work in practice And that's not that I've seen it fail. It's that I haven't seen it work in practice That said I have seen a number of other things scale Very easily to similar sizes So when I look at the teams Inside microsoft that are scaling to thousands of people They're pretty much using one of two approaches. They're either using a waterfall like plan Which works pretty well and I actually highly recommend it Adaptive or agile waterfall great technique And it works do simultaneous phases But go ahead and plan like you have for large projects with very fixed delivery dates. It works for that system Or the other one that they're doing is a very decentralized metrics based approach This is the way that yammer works for example, where they do continual analysis to understand What are the leading metrics that correlate with future revenue to what degrees of correlation? And then what are the experiments that we can do that have what impact on those leading metrics? At that point, they've now decentralized the decision of who of what should we execute on Such that each individual team can start making calls that optimize for the global whole That breaks the whole planning problem and they don't need central planning in order to get unified results and they can scale very high So I enjoyed very much what you said about scaling agile that The framework doesn't say What not to do and this reminded me of What we used at Siemens for scaling agile and we still are using it in different organizations is the book from bass potter and craig larman Scaling lean and agile Development and this is something that was very very helpful because I think in basically Every organization is different and every agile or lean adoption is different And we need to have to look at different Things that will work in this context and not work in another context and we have to know Under which circumstances we should try something and Not try something else because it would not work there And there this book was of tremendous help for us for the organization with several hundred developers also all over the world to Get to a good own agile framework that is maybe it's in some Things it's similar to the scaled agile framework But I think out of the box an agile framework will not work anywhere So I once heard someone really authoritative in the agile space say that You know, they looked at the scaled agile framework They looked at dad, which is the disciplined agile delivery and they said You know, this is exactly what we were trying to avoid 14 years ago Right, so that's someone who wrote the manifesto Making a public statement saying this is exactly what we were trying to avoid But somehow we don't seem to be learning from our mistakes That was well summarized. Thank you We had a gentleman over there and then we'll come here. Sorry guys Well, you already have yes, go ahead. Yeah Um, can I add a quick comment on the last one just We have used in our work with quality system and agile We used action research, which is not known as an agile method But something which was around since 1920 as a research method within organization is a collaborative method I just recommend you to look into that when you want to change in the organization to look into action research I think it can be very helpful. It's an agile method, but it was before agile Now to my question I have worked with I have pioneered agile in my organization and It worked very well for outsourcing from Scandinavia to India as such But one area where we have struggled has been with the product owner role we use scrum and xp But the product owner role has been a challenging one either the customer have Provided one person and it's mostly non-it companies for our customers They provided one person who've been a product owner, but they have not fully understood that role They have not wanted to participate on a regular basis Therefore we tried with having a proxy in India Who has worked together with that customer product owner and do it sometimes it worked sometimes it has not worked I'd like to have some feedback about how to work with this critical part of To get the agile to work together with customers in particular End uses and customers who are not mature and understand the agile methodology itself So I'm another bias in that I'm a product manager by trade Spent the majority of my career doing that and I think that the scrum product owner role Because it doesn't talk about the discipline of product management fully and take some pieces of it Uh It's open for sort of an asymmetry of people Not committing to doing what you need, right? I'm a smee I'm some person over the side and I want some product built But I'm going to give you a little sprinkling of my within my head as opposed to actually sitting down and going Okay, I'm building a thing a product So I think there's a little bit of How the the role is defined in the community Lending itself towards bad behavior But what I think the ultimate answer to what you're saying is that we need to evangelize that we're actually out in the world Creating more products. What do they be internal? What do they be external? We're creating a lot of products in the world We need more people doing product management and agile project product management And I think when people Have it as at least a role, maybe not a full job But when that's clear and the the norms of it are clear and what what the responsibilities are in a larger way than the product owner I think You'll you can get better engagement because it's It's core to the the responsibility of building a thing that the product becomes real to people So I think it's hard to get a smee that In a case where you get a sprinkling of their time to engage in general I think that's a kind of a flawed thing that we have So I look at this from a systems thinking perspective Um, I find that a useful way to analyze the various options. Um Fundamentally the thing that seems to determine the effectiveness of the Of the product design or the product direction will call it Is the length of the feedback loops and the number of delays And the amount of information that makes it through each hop from brain to brain loses Like about 90 of the information And every hop Causes delays and then a hop that's international Yeah Wow Yeah, so an international hop is going to cause even more delays And or more information loss depending on how it's executed So the teams that I've seen that have the best Direction with the customer have They don't even have product owner roles They have developers that are able to directly analyze the data of direct user behavior and experiment on those users directly In order to gain very good clear information. There's one level down where you have a product owner role That's doing that experimentation. There's one level down below that where you have a product owner role Who's making guesses based on their opinions and their experience? Which is now lagged information on the basis of a direct customer and then you can go further to You know distribute that or have a customer role proxy, but just from a peer systems thinking perspective It's all about how could I eliminate more hops and how could I eliminate more delays go all the way towards zero? Can I just comment on that? The problem has sometimes been that the customer is not able to explicitly communicate what they want That one I've called as a proxy has often been the one who has facilitated that Discussion that sometimes worked very well, but it's not always work. That's why I asked I've actually never met a customer who can clearly communicate what they want No, neither have I but I meant There are those who are better than those problems If they think that they know exactly what they want then you you pretty much can shut down your project on day one We don't need that yeah Proxy cannot be the conduit for successful You know product owner or a manager Proxy may not be the right word Because oftentimes when you when you send it off like this in a proxy manner, there is also he is proxy and passive also He is not empowered to take any decisions So how can he communicate to the teams? So for every decision he has to make he still has to connect back to the product owner and he's not available And he himself doesn't know So that's where the challenge is So I think that part of the so I think we all agree on But this is a problem And I think that I might have some of your same background coming from Scandinavian context myself. So how to address this problem And especially in the Scandinavian context. So I work in Norway with teams in Sri Lanka And We have some two features of Norwegian organizations that are Helping us the first feature is that People are usually open to informal decisions. So you can have things kind of happening without a formal decision being made The second one is that people like people Even though there can seem kind of cold people like to have this feeling of Of a nice conversation of like Inclusiveness is an important value. I think so what we've found to be very useful is to create a Workshop in a remote setting where we get the developers in the room and we get the product owner and the SMEs Whoever is there in the room and we sometimes have to trick them a little bit in Why we're pulling them in here? So we're saying, you know, we want you to we want to introduce a team to you We want to we want to have sort of a round table thing. These are things that are very accepted in a Nordic culture, I think and then we make that into a first situation where we Make the developers and the SMEs and the product owner Talk together and from there we're saying we like this we Suggest you do it more And and then we're and then we've got started anyway So I hope that is applicable in your setting as well Cool I have a question You need to add you could come up someone. I have a question on paid programming Yeah, go ahead. So I have a question on paid programming that is a basic question. So Can you just hang on? He needs to complete something. Please. Yeah. Hello Uh, just two lines to share is uh, we had this kind of issue and uh, you know the and we try to Ask for more involvement from real customer and so one of the reason came is I don't have time and other thing was uh Like I told you clearly what I need From their perspective and they say once it is done, then I can I can be in the feedback loop So literally our team build a fake Feature and just like a UI stuff and say, oh, it's done and the guy like a proxy Say thought it's done And then we went to the customer and had the feedback actually So that's something like, uh, you know, I think he also added like you have to come up with something That's a brilliant idea Yeah We do that with real users all the time Do you think The customer need to Need to have the agile mindset for an organization to adopt scrum That's the customer need to have an agile mindset to adopt scrum Maybe an easier question is if a customer does not have an agile mindset, what's the best thing for them to do? So, well, all right, so my challenge with the question I guess is that for for for my current role for my context The customer is an end consumer out there. That's you know, browsing around the web or has a phone in their hand Um, and I really don't care if they have an agile mindset or not. It makes it makes absolutely no difference to us Um And and some do and some don't right? So I could actually the the the question is a little difficult for for me I'm imagining that it comes from In a consultative way And in that, um Having having also done that kind of work It is far more challenging if the customer doesn't have an agile mindset You know oftentimes oftentimes we were doing transformations So they were asking us to to help them Help them get there You know, but we also just found that for a customer that didn't really have that mindset, you know a really high communication Setting expectations up front. Here's how the project's going to work. Here's how we're going to do things Here's how it's going to be funded. Yes, every you know every week or every two weeks You're going to have an opportunity to say all right. That's it. I'm cutting the funding off the projects over or we're going to keep going You know, there were some benefits that that would attract them to the idea of an agile approach And as the project progressed the relationship got better and better So there were some that when we started they didn't have really an agile mindset by the time it was done They didn't want to do it any other way So I I think You know, you don't even have to use the word agile In most cases, I would assume that customers want Value delivered at a low cost and customers want to have a good sense of progress Forget agile Almost all customers would want maximum value at the lowest cost And would want to have a good sense of progress being made So if you could have a conversation around those And expect that you know, we want your feedback so that we can deliver quickly The transformation would happen automatically So The the discussion around agile itself need not really happen. So long as customers have this broad principle in mind I jump in here real quick. I think there are different kinds of customers You know for some customers The reason to go for agile is basically I want to Or not to go the reason for some people to outsource their development Is basically they want to push the risk of their head to someone else, right? Is is that this is risky? We don't quite understand it. We don't want to do it Let's push it off to someone else and now it's their risk. So let's try and get a contract in place Let's try and get a date and time and cost in place. So the risk is off our head, right in that kind of a context How you educate the customer about agile or whether agile is even important for them is Is something to ponder upon right because Fundamentally what they're trying to do is Push the risk off their head and if you understand that and if you can explain to them that Or rather demonstrate to them that you know this technique of whatever call it blah that you're doing Can actually reduce risk then they would appreciate what you're doing much more, right? So it doesn't you don't have to go big gungo Let's do a five-day agile workshop because we're going to educate you or baptize you to to be agilist from now Right, you just you just say well what you're trying to do is push risk off your head We understand that what you're trying to do is high risk So we will try and work with you to reduce the risk, right? So that's one approach to take in some cases people want to do agile because they have heard A bunch of stuff from other people and they're trying to push you to do agile But they don't have the mindset, right and there it's already a failing game because there is a lot of Misconceptions to be clear and that that's another situation where you know, you do need to help Deal with the misconceptions first before you kind of get into the whole agile mindset with them Ironically in terms of customer the word So I build products for companies. So we do enterprise software essentially But we we build products for people wanting to do agile But It's surprising how many people want old school roadmaps So they'll tell you, uh, what are you going to build for a year? I don't know Because I'm agile about it. So I want your feedback because maybe I'll build what you want next But people like no give us the roadmap for a year or two before we buy Or you'll see people saying I don't want those releases. We're like, hey, we release all the time You know four or five times a year now we have sass. We release like, you know, we just put stuff out there We don't want that. That's destabilizing and it's like, well, wait a second. Don't you guys do agile? Yeah, so it's like, well, do you give your customer a two-year roadmap? No Then so in our case, it actually is a fair amount of friction if someone wants to buy an agile tool But they're not quite agile. It's strange So something I just want to add here is a lot of time context when here that's where context matters More than the customer customers are serving a market and certain markets are more dynamic than say other markets Or certain domains are more dynamic than the other, uh, you know domains and that dynamism in that domain sometimes, you know, drives the agility Or, you know, the requirements of the customer So and and certain domains are becoming much more dynamic than what they used to be in the past Right, uh, and that's exactly where, you know, most of the voices that you hear from customer comes So, uh, you know, you we need somewhere the delivery computer I mean the software delivery to be aligned with the dynamism of the market and that's what exactly customer is looking at Really? So a lot of the comments here have been about, uh, it doesn't matter so much whether it's the agile mindset and that it's it's much more about The outcomes and be able to get to agreement on on what you mutually share is the values that agile will help you deliver on Totally agree One of the values that your customers that almost all customers are bringing in That will not necessarily align with you is Uh, sort of what noresh was talking about not only do they want to move risk onto you They want to move executive function onto you as a customer. I want to not have to think or worry about things It's one of the biggest values that Anyone gets from buying something is I can just buy it and then not think about it Um, so that can directly get into the conflict even someone who has an agile mindset Yeah, I I want to for the things that I want to worry about tweak it all the time for the rest I want to install it once and have it work and have magic show up every couple of days and never have any Magic go away. Um, and it's up to you to make that happen um, so that's where uh getting back to some of the previous comments about customers customers won't tell you what they need Um, and they don't they actively want to not worry about this. So look for other ways to get the data user experience designers and design thinking and all sorts of techniques have been developed for Finding the data based off user behavior and the like those can supplement a more rare conversation with the customer All right, I need to call this off because people have been telling me time out time out. So We have dinner that's ready for you guys And I hope that we will continue this qna as we go through the dinner and we will Will be around here to go on. There's one question over there I I will I will take that question along with side with all the speakers and we'll have dinner and we'll take that question Yeah, this is regarding a pair programming. This is a basic question. I'm a verification engineer So we are planning to adopt this continuous delivery and pair programming for the next release onwards So right now our ratio of the developers and the testers is four is to one. So we are happy with that and What the testing has to be done within the release so by one tester So if you're going for the pair programming and pair programming and continuous delivery And this ratio is not I mean not practically doesn't work four developers and one tester Right. So what is the preference of what is the preferable ratio besides for the play programming and continuous delivery? I mean the automated everything within the release. So all this considering that so what is the preference you developers and Testers ratio People who want to leave can certainly leave. We will finish this question and then we will leave later But those who want to leave can go ahead It is all about the context The right proper ratio of devs to testers depends a lot on what the testers have to do because of what the devs are doing Um, so teams that have good strong two star agility the devs are no longer writing bugs And when they're no longer writing bugs then uh testers have a very different role Um, and I've seen a lot of companies that work very successfully with a 51 to 1 dev to test ratio That tester is one of the most important people in the company And they're doing a completely different extremely high value role They're finding all the holes in devs thinking and unknown unknowns But earlier on it should be 1 to 1 dev to test ratio because there's a different job that test is is doing So really depends on your context One of the things that I think we've gotten in a trap of With xp is the word test is overloaded So what is a test is that a unit test can be something to preserve developer intention It could be something to test the boundary condition on a small way And so because it's so overloaded everybody thinks they can test And in some ways everyone can and that's great. That's how you bring full quality ownership to a team But also the discipline of uh testing is really important. And so The ratio is again highly context dependent you want to make sure you have enough testing expertise In what you're doing so that the test you're writing the automations in you're doing is is solid And I I can't tell you what that is without knowing your your actual individual setup But I would caution you to really think about testing of the discipline And the the best functional or you know testers in the world Other people who really think about this are amazing. I don't know if you've ever seen a talk by Michael Bolton He I mean he blows my mind. I'm like wow you thought to do that thing Why did you think to test that? And so if you don't have that In your team you need to get it All right. Thank you. Thank you guys for entertaining us Dinner is served outside. It's uh, you don't need any coupons or anything You can just go have dinner and hang around we'll be around here for at least another few hours All right. Thank you guys


This is stock number HA342B. We are here at Summit Automotive in Fond du Lac, Wisconsin. You're new and used SUV headquarters. Today, we are checking out this super clean 2015 Chevy Traverse. This vehicle has a 3.6 liter V6 motor. From this HD video, you will be able to tell that this vehicle is extremely clean all the way around inside and out. No dents or dings or cracks on that front bumper. It has the projector light, projector lamp fog lamps. Headlight lenses are nice and clear.  So if you have HD capabilities on your computer, tablet, or smartphone device, turn them on right now because it is like you're right here looking at the vehicle with me. You can see just how nice and clean that body is all the way down that side. It comes with the factory 20-inch painted alloy rims and it has bridge sewn dual-er HL255-55R20 tires. They have right around half the tread left. Four-wheel disc brakes on this vehicle. No dents or dings on that front fender. You can really see that metal flake in the paint is a black granite mixture. Hood is very clean. No dents or dings on that. Front bumper was an excellent condition. I didn't see any dents, dings or cracks on that. Passenger side rim is in nice shape as well. No scuffs or scrapes. And as you go down this side of the vehicle, you can see just how clean that body is. How reflective and mirror-like that paint is. No dents, no dings, no scuffs, no scrapes. Paint is very glass-like, very mirror-like and just really nice all the way down the side here. We take these HD videos so if you are far away or even if you're close by and just cannot make the trip down, you can still see the car, hear the car, and have confidence in the vehicle that you're looking at so that when you get here there's absolutely no surprises. You can have confidence that you're dealing with the right people and the right vehicle. Four-wheel disc brakes. Back rim is absolutely perfect as well. Back tires have just as much tread as the front tires. As we come around to this side of the vehicle or the back I should say. No dents, dings or cracks on the rear bumper. Paint is very, very nice all the way around on this vehicle. It does have a backup camera. This is a manual lift gate. Back storage area is very clean. Those third-row seats do fall down for the extra storage like so and they come back up like that. Both sides go down nice and flat and you can see that the rear shocks for the gate are doing their job very well. Gate shuts nice and solidly. Tail lamps are nice and clear. And as we go down this side of the vehicle, no dents, dings or cracks there. For full disclosure this back rim is in nice shape as well. And as we go down this side of the vehicle just as clean as that passenger's side. Very nice condition on this vehicle. It has the side marker lights. They are heated as well.  There are no rips. There are no tears on these seats. They are heated and it does have side curdinair bags. Power driver's seat. Factory floor mats throughout. Power windows, power locks and power mirrors. You get auto headlamps. Tilt telescoping steering wheel. And as we hop inside the vehicle here you can see that this one has 52,769 miles. The instrument cluster is in really nice shape. Steering wheel is in excellent condition as well. No major scuffs or scrapes on that. Bluetooth audio controls and cruise controls on the left. This one comes with the Chevy MyLink system. This is also where your backup camera shows up. Bluetooth audio controls and it has AM, FM and Sirius XM radio capabilities. Here are your air controls. You also have rear air controls too. Your heated seat buttons are down there. Passenger side floor mat and passenger side seat are in excellent condition. No rips or tears on that. This vehicle has never been smoked in. Smells very clean inside. It does have side curdinair bags. And you can see that the headliner is in excellent shape. Up here you have your home link system for your garage door security systems and lighting systems. And it does have on-star capabilities in the rear view mirror there. We will take a quick look at the back seats and then check out under the hood. Back seats are just as clean as the front seats. No rips or tears back here. It does have the latch child safety system. Seats are in excellent condition. You get your rear air controls back here as well. And two USB charging ports. Factory floor mats. Got factory floor mats back here. And the third row seats are in really nice condition. No rips or tears back here. So these seats go up like so. Then you just kind of fold these down in there. And we'll take a quick look under the hood but very clean inside and outside on this vehicle. I would highly recommend it from a used car standpoint for condition and quality. Under the hood we have the 3.6 liter VVT direct injection. Variable valve timing. Not much to look at here but engine bay is very clean. It does run very smooth. This vehicle has been fully safety and inspected by our service shop. It has a fresh oil and filter change. All the fluids have been checked and topped off. And this traverse is 100% ready to go. To see more pictures of this vehicle or one of our other 450 new and used cars, trucks, SUVs, minivans, Wranglers, you name it we got it go to our website www.summitauto.com Full pictures and descriptions of every single vehicle on our lot. Videos of every single used vehicle that we have that's ready to go. And if you'd like to see more used inventory, you can go right to our used inventory page at carson41.com. That's carson41.com, the number 41. And to make this Traverse Series, give us a call right now<PHONE_NUMBER>. Ask for one of our sales associates to make this super clean 2015 Chevy Traverse Series today. Once again, that number is<PHONE_NUMBER>. And if you'd like to check out more HD videos of this vehicle or one of our other 450 vehicles, you can go to youtube.com slash summit auto remember to like, subscribe and share on this video and all the videos that you see there. In fact, in a second you will see a link to subscribe to our YouTube channel on your left. It linked more used inventory videos like this one on your right. If you haven't been to our website on the bottom link to this video, this vehicle on our website, click those check us out. And we really look forward to helping you with this super clean 2015 Chevy Traverse LT. Thank you.


All right, my name is Catherine Chimetero. I am the facilitator for this session, and I have two speakers with me today, Dr. Tarun Singh and Dr. Tanmay Mahapatra. I will start by introducing Dr. Tarun Singh. Dr. Tarun Singh is a public health professional working as the country director for an international NGO, Maternity Foundation. He has 10 years experience and has worked with the Ministry of Health and Family Welfare, IPE Global, University Research Company, UNICEF and USAID funded projects. Currently, he's working towards empowering skilled birth attendants and midwives through digital health platforms. With a team of experts, they have developed the Safe Delivery app. The app aims to create a low-cost digital solution for frontline healthcare workers and to save the lives of mothers and newborns. As the country director, he drafts creative concepts and successfully reaches out to healthcare workers with life-saving protocols to strengthen health systems. He is zealous in bringing about dynamic changes in the ways mothers and newborns are given priority above all. I'm now moving to Dr. Tanmai. Dr. Tanmai holds a PhD in epidemiology from Fielding School of Public Health, University of California, Los Angeles. He has published more than 160 reviewed publications in reputed international journals. He's a physician and public health expert with over two decades experience in research, measurement, evaluation, implementations and innovations in public health and epidemiology. Currently, he's the team lead of Care India Solutions for Sustainable Development at Bihar. He has conducted several research projects and generated evidence-based results to help in policy making. He is proficient in quantitative and qualitative management and analysis using SAS, STATA and Atlas. He also teaches methods of data management, statistical modeling and analysis with interpretation from statistical and epidemiological viewpoints. He previously worked as a medical research director in Kolkata, a staff research associate in California and epidemiologist consultant in Kolkata. Now it is time for Dr. Tarun to start the presentation and then he will hand over to Dr. Tanmai. Welcome, Dr. Tarun. Thank you so much, Catherine, for providing the introduction and setting the tone for the presentation. I also welcome my co-author, Dr. Tanmay Mahapatra, joining us from the Bihar, where the study was conducted, actually. And first and foremost, I cannot miss to congratulate everyone on International Day of Midwife. And thank you so much, VIDM, for organizing this event. So today, we are going to share results of one of our pilot study, which Matanti Foundation in Care India conducted jointly in Bihar. It's on the user experience and effectiveness of safe delivery app in the nurses posted in public capacity of Bihar, India. So first, I will take you through the layout of the presentation. I'll start with the introduction. Then Dr. Tanmay will take you through the evaluation methodology implementation analysis plans. Then again, I'll come back and share the results, keep finding discussion points, and then we can have the questions from all the joining who joined the session. So what is safe delivery app? So this whole study is designed around this particular tool. So it's better for everyone to understand what it is. So safe delivery app is a free mobile application for smartphones and tablets. And it provides instant access to the labor room protocol to all the skill birth attendants or healthcare worker using this application. Currently, the clinical content designed in the available in this safe delivery app is around basic emergency obstetric and neonatal care, which means the key protocols to save the lives of a mother and new one at facility level. Currently, it is available in various languages, English, Indian languages, basically all the regional Indian languages. Apart from that, also it is available in the global languages as well. The beauty of this app is that this app is completely free. We work on the premium model, so no one need to pay for accessing the content or accessing any part, use of any module, use of getting the certificate, nothing will cost you. And also for the peripheral level health settings, we have designed in a way that this app doesn't depend on internet. So even in the areas where internet is not there, they can access safe delivery app. And also once downloaded, they can access all the modules, all the content in a single go. So safe delivery app is a very important tool and being taken up by government of India in various curriculums and various educational curriculums like BSc Nursing. How we started. So Metenti foundation in partnership with University of Copenhagen and University of Southern Denmark developed the safe delivery app. Before going into the field, we have tested the safe delivery app in Ethiopia in 2014, 15, where we have conducted a randomized controlled trial on knowledge and skills on certain modules. And we have found that there is an increase in more than 50% both in knowledge and the skill which actually helped us in reaching out to the other countries. So we have started reaching out to the countries where the need was pretty high for these kind of tools. And as India where you know, maximum number of maternal let's are happening right now because of the numbers, you know, the population size. We have joined hands with government of India and launched the Indian version of safe delivery app. When I say Indian version, it means that it is aligned with the government of India's guideline. In December, 2017 app was launched with the objective to build the capacity of service provider on the latest clinical protocols. Whenever you know, you start any project, you know, there is a need of evidence and we need to test the tool in the real settings. So earlier we, you know, tested it in the Ethiopian settings, but it was very critical to know how it is behaving in the Indian settings. So here India, which here India is already, was already sub providing the technical support to government of Bihar for the health system strengthening. So we felt that as Bihar is an area where, you know, the maternal and newborn health indicators are low and the, you know, we all feel that the internet literacy is a little bit on the lower side. So we, along with Care India, you know, partnered and you know, conducted this pilot study to explore the effectiveness of safe delivery app in Bihar's public health facility. The two aspects were actually seen during the study. The objectives were first was the usefulness of the SDA, how it is improving, you know, helping in improving the knowledge of the staff and also what is the effect, you know, how they are taking it as a learning experience. What are the, is it user friendly or not? How it is behaving? Is it crashing? What kind of technical problems they are facing? What is good? What is bad? So that we try to capture in this study through the questionnaires. Now I'll just hand it over to Dr. Tanmay Mahapatra to take us through the design of this study, Evolution Methodology and the assessment part. Over to you, Dr. Tanmay. Thank you, Tarun and thank you, Catherine and others as well as the IGM for giving us the opportunity of discussing the findings of this study. Just to start with the methodology. The assessment of this pilot was done in an evaluation mode which involved a pre-post comparison of the knowledge of the nurses who are the delivery conductors participating in this intervention. We did a baseline and then after approximately three months of the intervention completion and end line. The, as Dr. Tarun mentioned that here in India, partnering with government of Bihar was conducting a delivery service providers mentoring program since long. It was called an Amanat program and later on it was actually evolved into its Amanat Jyoti part. So in this program, different facility-based labor room post-aid nursing professionals were trained to deliver better, to give a better experience to the mother-child diet. The facilities were for some reason, the Amanat program was not running where the universe of this particular assessment to ensure that there is no such contamination of information coming from any other sources other than this particular app, this intervention. Thus, 33 basic emergency obstetric and newborn care level facilities belonging to six districts of Bihar were selected for this particular assessment of the study. Now, in these particular districts altogether, the study was implemented by during the pre-intervention assessment. The data collection for the assessment was carried out by a staff, a cadup called Knaps Mentoring Supervisors who were the principal implementer of the mentoring program as I mentioned. And they used to visit the facilities in pairs for mentoring the staff. They in these non-mentored facilities in six pairs of NMS made an exposure visit to the selected facilities to give the introduction about the app. During the process, they also administered a paper-based questionnaire, a tool comprising of 24 different questions covering all the selected for domains of this app selected out of 11 all together. Those were active management of the third stage of labor, neonatal resuscitation process and management of maternal complications mainly two, one reserving the postpartum hemorrhage, the hemorrhage that happens after delivery, person risen, a major life-threatening aspect for the women delivering, as well as another condition where higher blood pressure and protein passing through urine, proteinuria actually found to constitute a kind of syndrome or syndrome complex called preeclampsia or in its advanced age, eclampsia. Regarding these, the post-intervention assessment, the same respective nurses revisited the facilities and followed the same procedure as it was followed during baseline. Based on the responses provided by the nurses regarding their knowledge, regarding all these aspects, the information provided through the app, they were scored in a scale of zero to 24 overall, whereas zero to eight points scale were used for the neonatal resuscitation and active management of third stage of labor, a zero to four scale were used for the management of those two maternal complications, the post-delivery hemorrhages, as well as during delivery higher blood pressure and proteinuria. Based on the distribution of this score, overall knowledge-based question answering performance of these nursing professionals were categorized into either poor if the score was less than 50% or average if the score were between 50% to 75% and good if it was beyond 75%. All together, there were selected 268 nurses listed to start with, but it was required that they should actually be participating in the pre-test question answer session, the post-test question answer session, as well as through intervention throughout the in-me-twee period. Some issues were there, for some reasons, some technical glitches, either the pre-test or the post-test or together data could not be actually partially or completely collected. After removing all of them, there were 229 altogether subjects whose IDs could be matched during the intervention pre and post assessments also. Thus, this particular assessment came up with certain findings and I request my colleague Dr. Turun to take us through the findings and the discussion of it. Thank you. Thank you, thank you so much, Dr. Tanmay for simply finding the complex study protocol for everyone. So now we will see how nurses have performed after the three months of the uses of the safe delivery app. So as shared by Dr. Tanmay, in earlier slide, there were two criterias. One was we will be going to review the scores on the scale of zero to 24 and second one will be, we are going to talk about the categories that were poor, average and good. So these are the results for the overall scores, you know, the scale of zero to 24. So the maximum marks for 24 for the overall questionnaire, but we can see in the baseline that the average score was seven. And when we compare it to end line assessment, we can see in the end line assessment, the score was improved to 9.10, which reflects that, you know, there is improvement in knowledge, which is also statistically significant in point of view of statistical, you know, calculations. If we further subdivide and see the, how each module wise they have performed, we can see there was a upward trend in AMTSL earlier it was 3.06, but now it is 3.80. Similarly, in other module also, we can see the upward, you know, shift in terms of the knowledge, which actually strengthen the evidence that app actually improves the knowledge of the healthcare workers if they use it on frequent basis. Second set of results was the around the categories. So we have divided the participants into three categories, who have scored less than 50%, average was scoring between 50 to 75%, and the good who scored more than 75%. In the baseline assessment, we have seen that across all the module, including overall performance, there is high number of participants were categorized under the poor category. So we can see in the overall score, it is 95% of the people, you know, participants were scored less than 50%. When we implemented safe delivery app for three months, we can see there is a shift from poor to average. So it means their knowledge is improving. So in overall score, we can see the poor category is now reduced to 75%, which means only 75%, people are scoring less than 50%, which is earlier 95%, which is a quite high number. But now we can see that the average scores have increased. So from 5%, now average scores are increased to 21%, which means the people are scoring more between 50 to 75%, which is a good sign. And earlier during the baseline assessment, no one has, you know, came into the good category and no one scored more than 75%, but now 4% of the participants, you know, are there who has, who has scored more than 75%. If we further break it down, the AMTS and Neonatal Recess Citation, PPH and Pre-eclampsia-eclampsia, similar kind of trends we can see where there is, you know, there is a decreased number from the baseline, there is a reduction in the number of the poor category and we can see there is a shift from poor to average and average to good in across all the category, which is, which was very encouraging to see. And again, all the results were statistically, you know, significant in pre and post. As mentioned earlier, we have also checked the feasibility of safe delivery app with the positives and the challenges. So I'll share a few positives and the challenges with you, which were shared by the participant. First was it is available, safe delivery app is available in the multiple languages, which is a good thing, which is shared positive thing shared by the participant because in India, you know, various regional languages are being spoken. So to scale any application or any digital health solution, especially related to the training and the content, it needs to be available in all the regional language. So it is already available in eight languages, which they found very useful and very easy to, you know, scroll the content as per their preference. Second was the audio visual aids actually helped to make concept more clear and learning more interesting, make learning more interesting. So they have found, so in safe delivery app there are various, you know, features available one and one of them is audio visual, you know, animated videos. Other are action cards, practical procedure and drug list, but one of the mostly used feature is animated videos and they found participant found it very, very helpful because we have from our studies, we have seen that, you know, that what you see will retain better instead of just reading through the material. So they can see the animation and they can retain the knowledge for longer time through safe delivery app. Third was helpful for self-learning and knowledge enhancement. So this app is different from other app which are available in India because most of the app are collecting the data and collecting the data for, you know, monitoring and evaluation and also for the monitoring of the healthcare services. But this app is for them advice for the nurses which don't monitor them, it just, you know, they can use it for the self-learning and knowledge enhancement, which they have also felt. I will take fourth and fifth point jointly. So participants have shared that it has a certification process and they can also do a self-knowledge assessment. So Safe Delivery app has a My Learning platform where a user can assess their knowledge on their own and also can get the championship certificate which is signed by highest authority Indian Nursing Council and also carries continuous nursing education points which makes it actually a win-win situation. It is free, it is offline and also it gives you a C&E credit hours which actually helping them in renewing their licenses. Sixth point was content for simple and easily understandable. We have tried to, you know, make it very simple content is to the point, not lengthy, not very elaborative so they can, you know, quickly revise everything. Seventh was comprehensive drug list helpful for reference. It has a drug list feature which is available on the home screen. So quickly in case of any emergency you just click on the drug list and you get all the information about the particular drug, you know, doses indication, contraindication adverse reaction which they have found very useful. Eighth was question changes with each round of tests which help in brainstorming. So app is very dynamic. We have a pool of questions and pool of responses with changes after each use, each, you know, level. So it actually, you know, help them in brainstorming and think more deeply when they were going through the championship certification. Two more positives help to clarify doubts and correct practices as staff nurses come from different part of India and they come from different colleges. Save delivery app actually help them in coming on the same page on the labor room protocol because we have, you know, different teaching methodology across all colleges.  Tenth is helping revision during laser time. So definitely app, as I mentioned, is free and can be used anywhere they want. So they have used whenever they are free which was a good thing. Helping identification of complication. So it has animated videos where we have shown them how any complication looks like. So they can, you know, as I mentioned, they can retain it better when they are seeing the complication because regularly they are not experiencing the complications in the facility. These are, you know, very less in number they are coming to the facility. So they, it actually improved the confidence of the healthcare worker on identifying the complication through the app intent. It is helpful in preparation of short theory sessions, clinical discussion, can repeat the test and check one's progress each time they log in. So it has a interactive dashboard where they can see their progress, how they are performing on the save delivery championship certificate path. 14th is very interesting. No special training or trainer needed for the operation. Very simple application. You need only one hour or two hour of orientation and you are ready to go to use and explore all the modules and features of the app. So it was quite encouraging for the digital point, digital health point of view. It gives intermittent notification or reminders to the user of the app. So through our channels, technical channels, we know that, you know, if some users are not using the app regularly, we provide the notification and push notification and reminders to make, you know, to improve the engagement of the user on the application. We use an action card for management of complication seems to cover all essential points. As I mentioned to the point information is keeping and only essential information is being given in the app. Keeping Bragg as homepage icon is easy to access and quickly draw our attention. Step by step information make it easy, make it a fun learning. So they have enjoyed, you know, the learning platform a lot. It's a pocket diary which can clarify doubt in user language whenever necessary. So our idea behind designing the safe delivery app was that it is, it is kind of a body for all the nurses, which, you know, they get the safe delivery app in their pocket, in their mobile phones and information is there in the cloud. And whenever they open the mobile phone, they can access all the information which is there in the cloud for them and to, you know, make them more empowered on these labor room protocols. There are challenges as well as because it is a digital health tool and very, when we started off, it was in very early stages. So there were few challenges, you know, shared by them. I'll quickly take you through those challenges. First was the agent nurses are not interested so much. So it is around the behavioral change kind of problem where, you know, some, there was a blockage of not learning new things, but definitely there was a challenge that, you know, agent nurses were not that interested knowing the safe delivery app and they were seeing as a extra burden for them. All the PC nurses are not having Android phones. It is in the low resource setting, it is a quite huge challenge where, you know, Android mobiles are sometimes not available for the PC nurses. But government of India is also providing and more tablets in the field for each and every facility. So we have tried to integrate the app into the more tablets in various state to overcome this barrier as well. Sometimes safe delivery have become monotonous. So adding frequent educational games in between would help to refresh their minds. So it was a very good suggestion we have received from the feed from the participants that the safe delivery app follows a standardized structure, but they need more games, more, you know, kind of interactive platform embedded in the safe delivery app to, you know, make it more interesting and, you know, would help them to refresh their minds. So we are working on that. The theoretical or conceptual part need more elaboration. So safe delivery app right now work as a, you know, provide to the point information at the time of emergency or a time of, you know, mentoring or trainings. But few participants have shared that, you know, more elaborative information is required around theoretical and conceptual parts, especially physiology and anatomy part, which was well taken by us. The app shuts down after repetitive use. It is a technology related issue. So many have experienced that, you know, that app is getting shut down after repetitive use. If any question goes wrong, then the user has to re-attempt all the questions from the beginning, which is pit-time consuming. So for us, a labor room protocol is the thing where you have to score 100% because it is related to saving the lives. So we have kept the standard of 100%, but many have, you know, few of the participants have shared that it should be reduced to 70 or 80%. So that, you know, it will be easy for them to get the expert level and ultimately achieve the championship certificate. The rational of the few questions was not up to the mark. It created confusion. So in the language versions, you know, many have, few of the participants have shared that the rational was not up to the mark because of the translation purpose. But after this study, we have rectified that issue. And now, you know, it is available in various languages. The last challenge was few guidelines were different from what they have learned. So they have also given that example. But as I mentioned earlier, the app is entirely, you know, on the GUI protocols. And whatever they have learned during, during their pre-service education may be a little bit different from what GUI protocols are there, because those are the updated one. So they have also, you know, shared this that, you know, few guidelines were different from the pre-service education. These are the key findings. We have seen a broad shift in the knowledge, domain-wise and overall at the end line. And it improved significantly as opposed to baseline, both in the absolute scores and also a broad shifting, individual knowledge level from, we can, we have seen the shift from poor category to average category and from average category to the good category. Still domain-wise and overall means scores remain quite less than the maximum achievable or what we were expecting. But it is an unfinished agenda, which we are going to cover in coming studies and coming, you know, programs in India. For a specific module, Neonatal Resuscitation, we have found that it, participants have performed, you know, poorly as compared to the S3 modules, which is again, area where we need to explore more through research studies. For all the participants, you know, to better know the app, you just need to go on Google Play Store or the iTunes Store and just write safe delivery app and you can see the icon of, you know, the nurse and you can download the app and connect the right language version as per the region and you will be able to access all the modules of safe delivery app along with the features. Again, I want to thank VIDM and Catherine for supporting us in this whole process. And also I want to thank Dr. Tandem Mahapatra, Team Lead CML, Team Care India Bihar and Dr. Aboli Gauri, Team Lead Capacity Building Care India Bihar and Government of Bihar. Without them, it was not possible to conduct this research in the difficult, one of the difficult part of India. So we are very thankful to Care India as well. And now over to you Catherine. Now we can move forward with the questions. Thank you, thank you, thank you very much. Dr. Tamai and Dr. Tarun for that insightful presentation. I am tempted to download this app immediately, though I'm not in the midwifery fraternity. And it's interesting to know that the knowledge of the people using the app has gone up. The user experience also is good and it's still work in progress. You're ready to take back the feedback that you've received so far. Before we go to taking questions, there's a question that was asked in the chat box by Begum. He said, is the application suitable for iOS and Android? And the question was answered by Dr. Tamai. He said yes. At times with some specifications updates, flexibility is also there to adapt to other platforms. And two of the participants, Jane and Linda, confirmed that they've already downloaded the app and they're using it. Since Linda is with us, may she tell us how high experience is using the app? Linda, are you there? Maybe she's not there, so we can continue. Let me turn off the recordings now. I ask a question, this is Jane. Yes. Thank you, and yeah, I've had experience with the app and it works very nicely. Thank you, I've been using it in Florida, USA with the residency program that I currently teach in. So I'm going to... My question, gentlemen, was really a lot of work. Thank you, it's very user-friendly. But as an educator, I agree that sometimes the operationalisation of these apps can be the issue. And I am very interested in that you specifically identified NeoNatal. And as a person that does a lot of simulation, have you tried to integrate this within schools of nursing and midwifery where they do their learnings, maybe on the app, and then they go to the models, the technical simulation models of the moms and the babies to facilitate their learning? Thank you. Thank you so much, Jane, for asking the question. It is very relevant and thank you for asking this question. Yes, so we have integrated the app in many ways. So in India, it is being used as a job bit. It is used in the skill-based training and also it is being used as a learning tool for the educators and the students. So we have recently integrated the app, safe delivery app in the formal BSE nursing curriculum where the app is mandatory for all the students and they need to complete the certification to get their degrees. Apart from that, we also sharing the teaching methodologies which we are having from the global experience which we are having. In that, we are integrating the app into the skill-based training where we are replacing the PowerPoint presentation with the videos. And also during the skill-based training, we are asking them to use the app as a reference tool because all the OSCE related steps are already there in a step-by-step manner in the app. So we are also using it as a buddy kind of thing where they can see the app and perform the OSCEs at the field level, at the facility level. We have the experiences from the Ethiopia and also in the India, we have just started the skill-based training and in June, July, we will start those teaching methodologies in India as well. Hope I am able to answer your question, Jay. All right, thank you. We are going to take these two last questions from Sheila. Sheila asks, for an international audience, please explain what education preparations nurses have. Have they studied midwifery obstetrics on your technology? Yes, Sheila, so thank you for the question. Yes, in India, we have a BSE nursing program where they go through all these specific skills, the curriculum you have mentioned. For the midwifery, so earlier in India, midwifery was not there. It was just a BSE nursing program, but recently, Government of India in 2019 launched a midwifery initiative where they have created on the international ICM standards, we have created the midwifery curriculum. And through that curriculum, we are creating NPM educator, national professional midwives and also midwifery educators in the field. So midwifery is a new thing for India in terms of the, you know, in terms of the cadre, but all these practices were covered in the BSE and BSE nursing already by the NPM nursing council and Government of India. Thank you very much for that response. Sheila, I hope you're satisfied. The next question is from Anne-Marie. She's asking, was the poorer response to neo-nazal resuscitation due to the resuscitation being more practical skills-based, e.g., getting the chest to rise? Yes, absolutely. I guess Anne-Marie has, you know, explained it very well. Definitely neo-natal resuscitation is a little bit complex process. And sometimes, you know, the staff who is working in the labour room is not, you know, that skilled enough because they are not exposed to neo-natal resuscitation on frequent basis because in India, we have a vertical training around that. But yes, it is a more skilled work and more, you know, a complex than other complication managements. All right. Thank you very much for answering those questions. I think it's time for us to move on.


Now the last question you must ask yourself is are you willing to go without pay for a little bit at a time? When it comes to business, oftentimes it would take years before you recoup back your investment. So if you need that cash flow to sustain livelihood, then most likely you would probably need to run your business while having a part-time job. Now on the other hand, I've also seen other businesses make back their investment and being the profits in the first three to six months, being able to make back their initial investment within such a short period of time really comes back down to knowing what they're doing. And majority of the time, it really comes back down to choosing the right product, knowing your numbers, knowing what menu item to set, knowing your customer profile and creating the experience specifically for them and then marketing it to them at the right channels.