Yeah, hey, maybe we could, people can take our seats and if I could ask someone to step outside for those who are having coffee and tell them we're getting started here. Yeah, thanks everyone for coming. I'm going to turn it over to my colleague here. We have Ed Kelly. I'm director for service delivery and safety and since I work in this building and Fin does not, I'll officially welcome you to Sal A of WHO and thanks for getting through our almost activated new security system this morning and we're very pleased to be with you for the launch of this WHO bulletin theme issue. But yeah, before Lord, you want to say anything before we get started? Oh, all right. Thank you. Yeah, so welcome. Thank you all very much for coming. This is a new collaboration for us with UNewider. We haven't worked with this UN-affiliated Institute before, but it's been a very preferable collaboration and we have a really nice selection of papers on quality of care. The reason that we're interested in highlighting quality of care is an issue is that in moving from the MDGs to the sustainable development goals, countries really did a lot to improve coverage. So we had a lot of progress on quantitative targets, but the outcomes haven't always followed. And where we see that gap between quantitative improvements and outcomes not really following, we know there's a quality problem. And every country has not solved, no country has solved the overall health care quality problem because countries move very quickly from a situation of too little too late in terms of care.  And both of these are quality issues. So if you look at the papers in this issue, you'll find from a wide range of countries in all regions, authors trying to investigate the reasons behind the quality chasm and really looking carefully at how patients should be involved in determining quality, how we measure those outcomes, what the indicators are, and how we're going to make progress in that area. So thank you very much for coming. For all the interns here today, I'd really like to encourage you to consider this as a focus of your further studies because this issue is not going to be solved easily. So please consider it as you're moving on in your qualifications in public health. Thanks very much. Great. Thanks, Laura. And just maybe I'll turn it over to Finn to kick us off here. Okay. Thank you very much. Good morning, everyone. My name is Finn Taub. I'm the director of the UNU Institute called Wider Word Institute for Development Economics Research. It's a great pleasure to be here to contribute to the opening of this launch event of this WHO Bulletin Special Issue. I mean, if you would allow me, I'll just say a couple of words about UNU Wider and then a few words about the issue. And then I'd like to add a few words of thanks to colleagues who have played a key role in this process. UNU Wider, the United Nations University World Institute for Development Economics Research was established a bit more than 30 years ago in Helsinki, Finland as the first research center of the UN University. The university has its headquarters in Tokyo and has about 14 institutes around the world. Wider is the institute that focuses on development economics research in an interdisciplinary way. You might want to take note of the fact that one of the founding parents was a Marcia Sen, so of course we have long roots in terms of human development indicators, human development issues going all the way back to when we were created. Today, Wider is a unique blend, if you wish, of a think tank research institute and UN agency and we basically work both on sort of global development issues, taking up themes that we believe are coming up and are going to be important, but then we also do provide policy advice to a range of governments from South Africa to Mozambique to Tanzania to Ghana to Myanmar to Vietnam and so on. So we sort of try as we sometimes formulated to deal both with the bigger development issues but also trying to engage in specific country context. I would encourage everyone to take a look at our website for access, free access, free downloadable access to both books, journal articles and so on. I'm only to about, well I guess it's about 8,000 publications on key development economics issues that I believe are worthwhile digging into. Now how could, how did sort of this particular WSU Billeting theme issue come about? Well this was already referred to. At UNU Wider we take data and evidence serious. We do believe that data and careful analysis will have to play a crucial role in achieving the SDGs. On behalf of UNU I was one of the 68 agency representatives who were involved deeply in the process of formulating the SDGs. So I sort of, how can you say, got a keen awareness of the fact that the call for a data revolution is quite unique in some ways but it has to be followed up. This is particularly the case in relation to health. It's very clear as has just been mentioned that coverage has indeed expanded a lot both in education but also in particular in health and I think it's important to say and reiterate what was just mentioned that this has not always been followed in a way which would have or imply high impact. At the same time I don't think one should overlook the very distinct advances that have been made. At Wider we did a major program on research in relation to foreign aid and aid to social sectors was one of them. Now one of the key messages that came out of that research was exactly this focus on we need to address quality issues not just expand, expand, expand because as economists will put it there are diminishing returns to some interventions. You need to think about the quality if you want to have impact. Now the papers of the theme issue they do provide illuminating insights into the current landscape of the quality of care research in both low and middle income countries. They make a strong case for investing in more and better data on health system quality an area where countries must lead but where development partners add critical value through the production of public goods as well as data collection tools and technologies and analytical methods. Now if you would allow me I just step out of my sort of slightly more formal situation right now when I woke up this morning I realized it's the 14th of June 2017. 37 years ago in Switzerland our first child was born. I can attest to the fact of understanding very much at the personal level what quality of care means because the quality of care was not there. So it's just to sort of illustrate that these things are both at the global development level but they also concern human beings and they concern the very fundamental aspects of development which is to increase people's welfare and well-being. So this is a core issue both to development and to welfare and to the lives of our children and their children. I would like to thank our partners in realizing this theme issue from WHO at Kelly and Shamside from the service delivery and safety department for being a great partner in co-editing the issue. The Bulletin Editorial team with a special thanks to Kittis Bartolomeos for being editor. We were really lucky indeed to have you. We also thank Larry Golligli and Kaleen Selig for their outstanding vision and support. It has been just such a pleasure. Now Professor Margaret Crook well what should I say the star expert in the quality of caring low and middle income countries we have thoroughly enjoyed interacting and have appreciated very much our collaboration. Now authors some of you are here we were indeed impressed with the quality of the selected papers. I mean wider producers about a hundred academic papers a year that appear in refereed journals this particular theme issue will be a core element when we look back to what was achieved in 2016 and 17. I am delighted to be with you here today both personally but also on behalf of you and your wider and I'd like to thank everybody for being here today and for being ready to contribute to a process of learning both from the research papers and from our interactions. I look forward to our discussions and appreciate the possibility for being with you. Thank you very much. Thanks for that and just before I hand it over to Margaret a couple of thoughts just from WHO's standpoint. As someone who's worked in this field for a long time the the progress that we made has been notable I think but I think we should also be humble about the lack thereof as well. If you look at some of the work I started my public health career working doing research on IMCI the integrated management of childhood illness and the quality of care delivered to kids in Niger West Africa so it's based in this tiny town on the edge of the Sahara Desert Tawa Niger as part of a USAID funded project are some USAID colleagues here today and some of the results we got in terms of you know how pardon me we traditionally measure sort of quality of care which is how many times when you have the opportunity to do something right how many times did you do it right the results we got were around sort of generally 60% of the time whether it's sort of diagnostic work therapeutic work or communication work that the health workers were doing in these very overcrowded health centers a number of years later in the US publications by colleagues like Beth McGlynn and others showed that actually the US did only as well and often less well around 50% of the time did we get it right and you can see in some of the baseline data that are in this particular issue in different countries presented it were kind of at the same level and so I think there is there's some lessons to be learned there around some sort of universal universality of this issue that that we should get that I'll just come back to we've also we made some progress on the issue of measurements which is the other big question within quality of care but not enough there's still a multiplicity of measures and not enough unification around measures we when I first was at the OECD we were doing the OECD quality indicators I had good friend of mine I can use the name Jan Mainz is from Denmark he's thought of very well thought of in the quality world told me flat out we will never be able to report on quality of care across countries because the systems are too different it's you won't be able to talk about it now 10 years later quality of care indicators are regularly reported by the OECD they were reported at their ministerial meeting just earlier this year so we have made some some progress but the number of indicators on which we can agree is very scanty it's around 10, 11, 12 even for the OECD and on improvement yeah what can I say there are as many different methods of quality improvement out there as there are sort of practitioners in the world on it and I think looking at how we can come together on that is another is another area but you know I think the the prospect and the and the possibilities are very high Laura referred to this and also Finn our new pardon me leadership here at WHO Dr. Tedros is taken over as the director general has picked up the mantle from our current director general on the issue of universal health coverage as being a top priority if you look at how WHO has formulated universal health coverage quality is not part of the UHCQ but it's not it's not something we've thought about it is we have basically thought in a supply side orientation around universal coverage sort of build more things get more health workers and push out more drugs and we will solve the world's health problems clearly that is not happening in most parts of the world and some of the some of the evidence here will show that there's a big love at the current time in at least in WHO but in other circles too on conceptual models and how can we frame sort of health systems strengthening in UHC for me this is not so interested in all of that and and if anyone in this room before the end of the day says raise at their hand and says can I just ask what do we really mean when we're talking about quality I'm sorry you're going to have to get out of the room because we're we're not going to get into those sorts of questions but I think we spent too much time talking about the definitions of quality and framing quality and not enough about learning from the tons of experiences that are out there and and building from the bottom up on that so I think we do have those possibilities and that this could be a big unifier across not just for health systems certainly but also with our colleagues in maternal and child health a number of them are here and you've just had a really positive and energizing meeting this week on a new network that's active in this space but also with our disease programs each of the big disease programs here and then also neglected tropical tropical diseases all have different programs looking at quality of care whether it's indicators community engagement or sort of improvement activities I think there are a number of very promising initiatives one of which is led by Margaret and the new Lancet commission which maybe she'll say a quick word on and there's also work within WHO and our team on this that's led by shams and looking at trying to bring together within the house a task force on this as well as a group of external team looking at this and a learning laboratory that's looking at how we can advance learning within this space and the learning agenda around it as well as policy questions which we have a meeting on national quality policy issues which has been a missing a big gap for some of the champions locally they haven't had a policy environment in which to work so I think all of those are very positive but I think you know the bottom line is we should be pushing this agenda because for me it's where the supply side of our work actually meets the demand side and as a supply oriented institution as WHO is we miss this piece too often we have a program our patients for patient safety program some folks are in the room helped us start that where we had patient safety champions who they themselves or their family members were harmed by care and they told us their stories and it's amazing if you take I frequently start my presentations with an example from a woman from Uganda business woman from Uganda and who saw her child die in hospital and a man from New York City who saw his mother pass away and the experience they had of the medical teams insensitivity and the lack of proper protocols and treatment and communication there's some universal way we are constructing our systems that a person in Uganda and a person in New York City can have such similar experiences of poor care that we need to take a look at and I think this again highlights how what a unifying issue this could be for us going forward in a new era so I thank you and wider in particular for pushing us forward on this issue and Laura and the team for for making space and a very crowded publication schedule I have a new appreciation for how solicited they are but also for Margaret and the team for for the organization around the papers and all the authors who are here we really pulled together something I think that is that is really very interesting so without any further comments I thank you for that and I'll turn it over to Margaret thanks very much there we go okay good morning everyone it's wonderful to be here with you at what is I think been over a year's worth of work actually probably 15 or 16 months so a wonderful culmination of the work of many of the people sitting around this table and I couldn't be happier actually with what we have in front of you today which is the special issue of the bulletin of the WHO so let me go back to the MDGs the era that's just passed the MDGs have had indeed many successes I would say that chief among them is the much greater coverage of lifesaving interventions that they ushered forth that means something very specific I think some of us are bean counters and love to count numbers but really it is meaningfully different now to be a pregnant woman in many countries and know that you're going to be able to get delivery care out of facility that there is a path forward on that that you have somewhere to take your sixth child so I think those are those are incredibly important achievements and those achievements in coverage actually were accompanied in parallel by an incredible explosion of great research on coverage understanding how to measure access at the individual level the population level and even the equity of access which is such an important dimension who's getting care where are they getting care how hard is it to get to care many colleagues around the world have contributed to building the evidence base on coverage and measuring it systematically across countries and what that allows is for comparison and peer pressure and and for progress so I think the the research part of this can't be understated and the importance of that in moving health for people improving health and so we know that the MDGs have also had notable successes in actually improving health improving survival and they've done that undoubtedly in several areas they have not succeeded in other areas and it is I think instructive to reflect on where the MDGs have struggled actually where we have struggled in achieving the MDGs and often that has been in some of the more complex conditions maternal mortality which requires a chain of events to happen to happen on time to happen with care pieces to be there of the entire health system we haven't done very well for newborns actually globally we've made it some improvement but really not nearly as much as we need to and now today in 2017 we're actually past the dawn of the sustainable development goal era was reminded by your wonderful huge circle that health you know is doing its usual takeover of the rest of the the goals being in the middle but then what do the SDGs really mean for us those of us who think about health systems and people's right to health well the SDGs mean more conditions that we should be thinking about it's no longer maternal child health and a few infectious diseases it's a wide range of conditions it means more complexity the people who are not saved by current interventions are sicker people they're the sicker newborns they're the septic newborns they're the moms with severe complications they're people with chronic disease now that will need lifelong care also we know multimorbidity is a growing issue as populations survive some of the early years and grow on to be older which is a goal that we all share and want they're getting sicker and have often two or three conditions and then let's not forget this while health is changing and the profile of health conditions is changing people's demand for good care is also changing we are no longer in a world where people don't know what good care looks like they can easily text their cousin in Dar es Salaam if they're in a small village and find out what happened when she went to clinic people see on tv the way that good doctors behave with with patients and and worse doctors behave with patients the connectivity and by the way empirically we're seeing it in our research people with cell phones have different demands from from health care even adjusting frankly for their education and we are seeing greater education of course and we are seeing fortunately growing wealth so demand for care quality is growing as well so all of these all of these circumstances i think bring us to the question of quality it is difficult to imagine making progress on the sdg agenda and it is certainly impossible to think of how we can satisfy the needs of populations without thinking carefully about quality and so actually ed i will define quality here because i do think that one of the the challenges has been that it has become a splintered and very multifaceted issue i don't think it's that complicated quality to me has at least two key features it is health care that can either improve or maintain health of course a lot of health care is preventive so we don't have to always be improving health but we should be helping maintain it number one and number two it is health care that brings value to people above and beyond the the improvement in health remember many conditions are self-limiting people want to come out of a visit feeling respected so they know that they can go back when then when the real urgent need arises so this notion of value to people is something that i think we haven't been thinking enough about in thinking about quality so that's that's what i would say a definition is i i think that there are several elements that are particularly worth paying attention to and here now i'm taking the perspective of the patient the user of the system so what do i see when i enter a clinic i've gotten coverage i know where to go i come into clinic and i'm really thinking of two fundamental elements when i'm judging quality i'm thinking about the competence of that provider in fact is my perception of a competence that may have led me to this clinic in the first place is this person going to have the skills to help my baby the the reference you made to your son's birth we all everyone in this room who's had a child knows how hard we all look for the right exact provider with the right support system that's going to help that delivery happen or whether when our child as my daughter did fractured her arm where is the right exact place to go that will that will give me the technical competence i need so we need competent care everyone has the same demand for that but the second part is we want a good experience at the at the fundamental level this is a personal experience of health care this is i am also a customer i am not just a patient begging for your time and so i expect respect i expect ease of use i expect a good consumer interface frankly with the system and so the patient experience and the competence are two fundamental parts of of quality in terms of the process of that care and then if i do get that what are the benefits to me if i do get that hopefully i will have my health will be better now hopefully that can be objectively measured but sometimes what matters most is actually how i feel and whether my symptoms are improved we've seen increasing wealth of evidence including recently a very prominent article in in the oncology field showing that patient reported outcomes what people tell us they're feeling are actually tracking very very well with survival and with other hard clinical outcomes so we've got to be paying attention to how people are doing that goes for a post part of mothers whether they're depressed whether they're looking after themselves or they have symptoms as well as people with chronic and other illness so we care about health certainly but also good quality brings other benefits those include greater confidence in the health system for the next time i get sick or the next time my child gets sick or god forbid we have an outbreak of ebola or some other condition where now i think okay do i have a base of trust with the local clinic with the health system at large i assure you if you don't have that base of trust no amount of government radio messages at the height of the crisis are going to drive me to the health system the third benefit is greater and better utilization those of you in the fields of hiv tb in chronic care know very well how difficult it is to retain people in care and yet when we go around the world and we ask a health program manager what do you how do you know your your program is working the first thing they say is well we can retain people and they're right the idea that people stay in care is a vote of loyalty to that system and so retention actually an appropriate utilization timely utilization avoidance of over utilization by the way is also really important and also efficient utilization so thinking about economics you're not always going to one center which is overcrowded but actually the quality is good enough at your nearest facility that you could go there all right so utilization is another one and finally a benefit of high quality of course is finances so populations who get good quality care can go once instead of getting the three second opinions that they might need the second the third the fourth and you might think i'm an exaggerating or you maybe you think i'm talking about high-income countries where we're very picky no actually it's Liberian one of the poorest countries in the world where we found on average for every formal health care visit there were two additional visits to informal providers to just make sure that this was on track okay so this matters and this has an effect on people's pocketbooks and their economies so i hope that i persuaded you this matters i don't think i've had to persuade you if it came to this to this last equality is a critical element as we move forward on the sdgs so then the question arises what do we know where is the great research just like on coverage research was fundamental in in making progress happen what about quality and there i agree with ed i agree with many of the comments that were made there's a lot of work going on in many countries and yet i would say that today in 2017 our talk outstrips our knowledge and our assumptions outnumber our facts and out our evidence there are extremely few systematic comparisons of quality indicators across countries oecd accepted i'm really thinking now about low and middle income countries indicators where we can say with assurance this is nationally representative right this is measured in the same way across countries and we were at an extremely basic step in terms of being able to do the sort of benchmarking but let's forget about the international comparison what about countries themselves what do they know about how their health systems are working well actually very often they don't know much or they know pieces and that's perhaps more more accurate they'll know how a tv program is working here because those partners are actually reporting and doing a great job they have a patchwork they don't have an overall sense of how the health system is performing and so i am extremely grateful to the bulletin and colleagues at u and u idar and the world health organization for working with us to put together this issue on quality what we try to do here and quality measurement more specifically what we try to do in this issue is to to far and wide reach out to the best researchers working in this area of quality measurement to say bring us your best work bring us your best evidence talk to us about measurement how do we do it across all conditions it wasn't just one set of conditions or another tell us tell us what we need to do better and i think we have a wonderful set of papers more importantly even i would say than the empirical findings which we'll hear about in a moment is the stimulus that we hope this issue brings to the field of quality research i'm talking about research and measurement i'm talking about description i'm talking about analysis i'm talking about better evaluation of improvement the full range of research needs to get bigger and better and soon um finally let me just say a word about the lancet global health commission on high quality health systems which i have a great privilege of chairing together with dr. mohammed pati and 30 amazing commissioners many of whom are actually in the room and advisory council members this is a in 18 months effort to try to define quality in a clear and simple way that health ministers and and line managers of health systems can understand not just experts at who or in academia and also to describe the best that we can quality across conditions sdg conditions talk about improvement what's working and and very critically about the equity of quality who's really getting good care in countries because we don't think it's necessarily the poor uh so that work is is ongoing and we hope to publish a final report in october of 2018 so just over a year from now uh the time is running fast um so we look forward to all of your input on that through our website um which is hqsscommission.org um happy to talk about that on the side but i want to return now to this uh incredible group of authors to tell us about the findings in in the special issue thank you so much again thank you margaret so we have a selection of these fantastic authors sitting up here and we're going to hear from them in turn looping from the world bank is going to start off telling us about infection prevention and control practices and primary care in kenya um if you could walk us through your findings thanks who is managing the presentation okay because it has a lot of animation so i may need to okay you may need to good morning and i'm going to talk about our results uh from uh analysis of compliance with infection prevention and control practices in kenya uh this is a part of a bigger project that is the kenya patient safety impact evaluation uh that uh i am uh part of with a group of colleagues from the world bank but this is only one of the two that we are using to measure quality of care okay so first um why is this important i don't think i i have to motivate a lot for this audience healthcare associated infections are a priority a global priority and IPC practices are relatively easy and cost effective ways of preventing a health related infections um but the problem and the challenge as as the um the speakers have emphasized now is that there is very little research on the extent of the problem and particularly in low and middle income countries and for primary care settings most of the studies are focused on small samples focused on only one dimension or one of the sites and in many occasions on cell reports so to address these limitations we um build or uh use an observational patient tracking tool to measure uh if IPC compliance in different domains and sites and we conducted the largest patient safety survey across low and middle income countries that we are aware of with more than 1000 health facilities are in 1700 healthcare workers and more than 14 000 patients so what did we do first we decided uh we selected three procedures that we were going to observe they are examinations injections and lab tests we decided on five domains that we were going to track is they are hand hygiene protective gloves injections and blood sampling reusable equipment which is basically disinfection of esthetoscopes and uh thermometers and waste segregation and the idea is that what we wanted to track is the the average experience of a patient throughout patient services in Kenya then we developed no goody back when we developed a tool and basically what we did is that we built on a WHO tool so we're already available and validated but we put them all in one checklist it's a two-pager where we actually track all the indications which are the cases in which uh there is risk of pathogen transmission in the in these uh of procedures and also the actions that the healthcare workers took and the idea was to fill in this checklist in uh on average five minutes that uh an outpatient consultation uh last then what we did is that we train assessors and very importantly we use tags so all the patients that we track have adhesive tags and the healthcare workers as well so that we could actually follow them through the healthcare facility or through different sites for the healthcare workers we also measure knowledge at the end of the observation and we also measure whether they had available supplies so the main results first I don't think you can see it very well but I'm going to try to describe it first is that we found very low compliance with the 20 IPC practices that we analyzed 32% on average and this figure that is showing there the the top of the figure shows you the indications which are the cases in which an action should have taken because of the risk of transmission of pathogens and the line below tells you the violations the safety violations and in summary just to to give you a point of the analysis that the first one the 3.1 and 2.9 is if a patient only goes through examination which was 50% of the patients usually all the all the indications are safety violations so it's three indications and three safety violations the last one is if an a patient goes to examination and has at least one lab test and at least one injection they are exposed to 23 indications and 13 violations so it gives you an idea of the main results the second one is the second important result is that there is significant variation across all domains so hand hygiene which is the corn store on IPC the compliance was 2% which was very low and then very high in anything that was related with infection injections and blood loss so 87% on the practices that we measure and 82% of the waste aggregation of needles and syringes the rest of the of the practices were to needle to very low compliance the next in the next one and we found very significant no-do gaps so in the first one just to give you an example that the compliance knowledge and supplies for hand hygiene so what you find in the first one is that in 83 of the cases healthcare workers actually knew what to do in which cases to wash their hands in 70 70% of the cases they had the supplies that were required for the practice but in only two point four of the cases actually you they practice right and what happens is that when we condition that and do that average only for the healthcare workers to actually knew and also had the supplies that only jumps to 4% so 2% on average for all of them but when you only do them the average for when they had supplies and knowledge it only jumps to 4% so what are the the other element is that there is a very weak association with anything that we measure at the healthcare worker or health facility the level the age the location the gender where the facility was big or or large many of the things were weekly associated where are the policy implications on this one is that there is a progress in in some areas so what we found is that all the campaigns that have been conducted for injection and blood draw safety actually have been effective because for instance in one of the indicators where they were using new needles and syringes it was 100 percent compliance and that was a surprise for everybody because every time that I have presented these people don't expect these results however one of the things that we found is that this weak association between knowledge and supplies and all the other characteristics means that is more about behavioral elements or constraints and that remains the biggest challenge on what to do how to affect behavior you can move on okay some caveats and takeaways to finalize caveats are that we cannot currently link these compliance indicators to help outcomes we don't have a research that actually indicates how to weight them and we are centering this in the interaction between health care worker and the patient so we are missing a lot of things that happen outside of that interaction for instance the rest of the waste management and health care workers may change their behavior while they are absurd we actually tested this and we didn't find evidence for this and the main takeaways first this observational tool was effective for assessing compliance with these practices across every type of health facility in very short period of time because on average it took five minutes for each interaction 99 percent of the patients and 100 percent of the health care workers consented to be absurd which was another thing that was concerning at the beginning for us there is high variance but overall low compliance and the conclusion is that improved improvements will require a broad focus on behavioral change that's it thank you what I'd like to do is suggest that we go through all the summaries of the papers and that we take your questions at the end is that if that's all right I just I'm sorry to say that Marcia Lazarini who is going to present the next study on improving the quality of the part of hospital care for children in Kyrgyzstan is unable to join us there's no other author present from that paper who would like to speak about it anyone involved with that study and see there's no one from your group no okay so we'll skip that one but and we'll go directly to Guarav Sharma over here on my right from the London School of Hygiene and Tropical Medicine who's going to speak about the quality of routine essential care during childbirth clinical observations of uncomplicated births in out of Pradesh in India and if you're finding it difficult to see the contrast on the slides please note that most of the figures and tables of course are in the journals which have been distributed to you Garva thanks okay second on our paper which is in this special supplement and I'm presenting on behalf of the other co-authors and would also like to acknowledge our funders so we chose to focus on routine care because that seems to be neglected in programs and also in research and we did this in three districts in Uttar Pradesh next slide so just to I was only given five minutes so I mean it's a lot to back in the five minutes but all the details are in the paper but we conducted an observational study using clinical observations using 14 enumerators who visited admissions emergency wards labor room postnatal wards wherever pregnant women who are likely to undergo normal vaginal births were and we did this across 26 health facilities eight in the private sector and 18 in the public sector and to calculate the required number of observations at each facility we had done power calculations for all of the key indicators and we used W. Chogailan's and available tools to develop a data collection instrument and we conducted extensive pre-testing of the tool itself and also did a thorough pilot study for three days in the public sector and four days in the private sector to finalize logistics during data collection so we realized that in this setting there are not many normal births that happen in the private sector so we needed to spend a longer time at those facilities and data was collected between May and July so overall we collected data on 42 items of care using a very comprehensive framework that included adherence to evidence-based practices harmful care practices harmful behaviors by the health worker and harmful practices but also aspects of women-centered respectful maternity care and what we find is overall across the board quality of care was very poor with a mean score of 35.7 across all of the facilities so women on average only received about 35 percent of the recommended practices and so neonatal care was slightly so this is only immediate newborn care which was slightly better than obstetric care which were which were both poorer but we found a significant difference between the quality in the private sector versus the quality in the public sector and so 40 so women on average for overall essential care at the time of birth received about 45 percent of the recommended practices in the private sector but only 33 in the public sector and you can see the rest there and those were significant significantly difference across the sector but these are aggregate scores so obviously they hide a lot of variation so maybe if we go to the next one so if we look at certain key indicators the quality was as you can see in these graphs the quality was really poor so for example and and the indicators with a star in them are the ones that were significant across the sectors so regular monitoring of labor using a part of graph less than one percent screening for preeclampsia less than three percent infection prevention was better in the private sector assessment of maternal blood loss better in the private sector provision of women-centered respectful care practices things like explaining the process of the labor allowing a companion to be present offering a choice of position were really poor less than four percent and also for newborn care this was overall poor quality obviously I mean there are problems with many of these indicators if we look at individual indicators and the validity of many of these indicators is also questionable and I mean research work is ongoing to look at some of this but so overall patograph use infection prevention maternal blood loss and assessment of fetal artery seem to be done in the majority of private sector cases compared to the public sector and the other interesting can we go to the next one the other interesting thing that we found is that so unqualified personnel frequently provide institutional maternity care in this setting and in both the public and the private sector but if we if we look at this graph a larger proportion of doctors provided care in the private sector compared to the public sector but greater proportions of staff nurses auxiliary nurse midwives traditional birth attendance dies other unqualified personnel provided care in the public sector um so this was an interesting finding that we found if we go to the next one so uh so that was on the descriptive analysis and then we conducted a multivariate analysis using a mixed effects regression model with a random effect at the level of the health facility to account for the clustering and after adjusting for all of the confounders related to the women's characteristics health workers characteristics and caseload at facilities day time of admission we find that there was no difference no significant statistical difference between the care care provided between qualified and unqualified birth attendance we found that private sector provided a six percentage point a small six percentage point higher overall care at the time of birth we didn't find any association between facility size or the volume and the quality of care at birth and but um but uh we found that quality of care during weekdays as compared to weekends uh during weekends the quality of care was three percentage points poorer so having said that i mean it's it's uh defining that slightly nuanced in the sense that there were problems with sampling the private sector in this part of india there are lots of private sector clinics so maybe every 500 meters there is one private sector clinic and we didn't really have a sampling frame for the or a census for all of the private sector facilities in the district and 13 of the private sector facilities that we contacted refused to participate in the study so their quality may have been different to the ones who agreed to participate in the study and they may also have been observer bias due to the general perception that private sector is better because it has um you know better infrastructure or shinier facilities or better better qualified health workers but so in summary these are key findings and if there are any questions we can discuss thank you back to you margaret for your study and several countries on primary health care services and quality thank you laura uh okay so um so together with with our team um that uh are the co-authors um in this work which i'm also having trouble seeing but i can see them in my uh printed program including some wonderful doctoral students postdocs and colleagues in uh tanzania colleagues uh in tanzania we um ask this question um why does quality vary how much does it vary and and why uh why does it vary uh and and particularly looking at the range of countries that are listed here seven low and middle income countries go to the next slide please um so let me say a little bit about a variation and why is this interesting i think for a very long time in health care in particular i would say in uh wealthier countries with extremely well developed and expensive health systems there's been a lot of interest in understanding variation with the idea that variation that is not accounted for by patient status right and patient need is what's called unwarranted variation and may reflect bad quality may reflect deviation from evidence because if you think about it controlling for what the patient needs uh the evidence base should be pointing everyone in the same direction for the care of conditions certain conditions um and so uh and particularly this is the case for primary care sensitive conditions that require pretty algorithmic set of uh things to be done and so we thought why not use that same approach of variation analysis to the systems of lower income countries and to analyze the the variation levels but also what factors might explain it um so next slide please um so this study is based on a nationally representative uh set of health systems surveys incredibly rich data set from the service provision assessment surveys um from Kenya Malawi Namibia Rwanda Senegal Uganda in Tanzania these were done between 2006 and 2015 so relatively recent data and these surveys contain an audit of all the facility stuff infrastructure equipment medicines staff personnel um as well as i think this is a more exciting part provider interviews direct observations of care and even patient exit interviews so we do have a multi-dimensional view of quality from these from these studies in this study we particularly focused on two critical services antenatal care and sick child visits and the way sick child visits work is basically the observers have served any sick child any child that was in um for any illness uh to be seen by a clinician in in a clinic i should also note just to the point about private and public these surveys include both public and private facilities in all the study countries and the weighting actually makes these representatives so they do tell us about what's going on across the health system next please um so the way that we define clinical quality here was the proportion of essential clinical actions completed out of a very uh i would say parsimonious very small set of what we consider the most essential antenatal care actions and sick child actions so eight for antenatal care nine for sick child care these are taken from global guidelines i m c i guidelines a nc guidelines but we wanted to just really focus in on the fundamentals where the fundamentals done you know was the mom asked about past the for example birth complications this kind of thing um and these were in the domains of history exam diagnostics and counseling and management so again this fans span the range of care um what we then did is we used multi-level random intercept models which is a way to um understand data that are that are clustered so these the way to think about this is one provider might do multiple visits right so the visits were clustered within providers of course one provider several providers are within one clinic so that's another level of clustering and then of course provider clinics are within countries and so we applied multi-level modeling but also included a country fixed effect to make sure that we captured inter-country differences as well next slide please okay so what did we um what did we find first of all there were a lot of data which is a wonderful wonderful thing in terms of illuminating what's going on and i should remind everyone this is data collected by direct observation so somebody standing in the corner of the room with a checklist and going down to see what actually is getting done not not what they said was getting done not what providers that acknowledged us but actually what did they do for these sick kids and the moms um so we have 2600 ANC visits almost 12,006 child visits which is a lot of data in this quality sphere as we've discussed earlier 80 percent of the facilities are public facilities again that's because the majority of provision in these countries is from the public sector um so we also assessed in addition to our key outcome variable was the observed quality but we were very interested to see obviously the factors that might explain variation and one of those factors might be facility equipment and supply so we wanted to um also test that so for example facilities scored on average between 50 and 75 percent of basic inputs and this is based on the service um uh readiness index the WHO has developed uh just to say this facility got the stuff it needs to to give this care um so about 50 to 75 percent stocked um the average quality that was observed to be given was about 62 percent I think as you said 60 percent is roughly what gets done well 60 percent is what got done in anti-natal care and less than that of those eight or nine recommended actions in sick child care I can also tell you these numbers are slightly upward bias when we take the full range of actions that could have been done not eight but really 25 or so that really should be done for a sick child these numbers fall even lower but these are the most essential so this is in a way the best case scenario thanks okay a little hard to see here but um these uh these lines represent the mean and the inner quartile range 25th to 75th percentile of the observed quality of care and this um is for for both actually sorry I don't think you can see very well yeah there are two colors here but if you look at your how's your bulletin bulletin issue um you will see that uh let me let me point you to the page let's let's open it up together um page 412 yes very good okay so all right um so um that's why they're not showing up they're very light bars are for um uh anti-natal care and the darker bars the ones you can actually see on this slide are for sick child care so let me just point a few things here first of all you can see these bars move all around each bar by the way represents one country's worth of clinics so these are across all the different countries um so there are countries that are doing better than others that's very clear from this from this graph it isn't always the obvious things though if you look at Namibia which is the richest country in this group by far actually the richest country it's scoring in terms of sick child care about the same as Kenya um so that's interesting to us why is Kenya a much poorer country doing a better job and then other countries with similar incomes to Kenya on the other hand are doing less well so you do see these inter-country differences the other thing I would like to point your attention to is the size of that bar so this is the the majority of the data are quite scattered there's wide variation uh within each country also of the care delivered what you again can't see but that better in the journal is that um interesting to us was also the fact that countries that did well on sick child care didn't 100 of the time do well as well on anti natal care some did better on anti natal care some did worse I would say that of the two services anti natal care is more formulaic right it's preventive care people are not sick in front of you so it should in some ways be easier to score those eight items but we didn't see those clear um we didn't see that well on average they did do better on a and c but but some countries didn't um all right let's move on to the next slide this you obviously can't read it all um and that's okay because i'm going to summarize what this says on the next slides please this is for the geeks in the room um okay so this is the uh the the quick interpretation of what we found probably the top line is after well there's several several key messages after we adjusted um so we were trying to explain what we're seeing what the quality explained the observed quality um what we did is we tried to use all the factors that we can think of from a well-developed set of conceptual models what could drive quality could it be provider motivation could it be provider codder could it be uh the equipment in the clinic could it be i i think we had 20 variables in this model because the data are large enough to permit that sort of exploration um we also then looked at patient level factors maybe if the if the child was younger maybe if the child was sicker so we included visit level factors as well as provider level factors clinic level factors in finally countries at the end of all that we weren't really able to explain very much of the variation in these data okay so in particular after all of these uh factors were put in uh into explained quality we found that nine only 19 percent of the variation in antinatal care but 41 percent of the variation in sick child care um um sorry let me go to the second bullet the full models explained 40 percent of the variation so 40 percent of the differences in quality in a and c and only half of that in sick child care um what i want to do emphasize in the first bullet though was actually what we found surprising was there was a lot of variation within providers from visit to visit that actually less for antinatal care and more for sick child care the providers clearly could do more and did more for some kids and they they didn't do it again for the next child so this kind of uh intra provider variation visit to visit variation um what we also found though that after all um all our um covariates were taken into account all the explanatory factors were taken into account the single biggest by far explanatory factor for the variation was what country was this visit in 80 percent of the explained variance was due to the country fixed effect in these models um so that's that's a key finding i'll come back to that in a moment um other findings um a and c quality was lower amongst physicians and clinical officers than nurses um was higher in private clinics and was better in facilities with better infrastructure equipment and management um those were some differences um but let me explain the size of these differences being in Uganda was linked to 30 percent versus um the the comparison country which i believe was Malawi in this case the poorest country was the was the comparison so being in Uganda versus Malawi was linked to 30 percent better care while better infrastructure was associated with three percent better care just to show you the size of the country effect versus the size of the uh of the effect of the um of the inputs uh to care so huge huge variations across countries let's go to the next slide please so i think the question is why what what are we seeing and and what does this lead to um so um on this chart what we're showing is each of the set of bars there is showing you the performance of worse and better facilities in each country and um and those of you following this this is on page 415 in the in the journal um what we're showing is that the lightest bar on the right hand side shows you what the top quartile the top quarter of facilities in the country could accomplish so look at that within the same country there is a set of facilities doing substantially better than the poorest facilities which says to me there is knowledge in this country and demonstrated ability to provide some better quality care um and so what i'm i think really eager to do and i would really encourage this is uh uh for all the interns and researchers in the room is an important direction for future research is understanding best performers understanding positive devian analysis understanding what's going on in those clinics that can help guide us um i think the other big questions to my mind are why do countries produce such different levels of quality what is it about those countries that's doing it it's not always income that's not the that's not the single um predictor as you saw was in the maybe a case um and so one of the things that our commission um is looking into in greater depth is whether pre-service education really is all that comparable or sufficiently or sufficient as we look across countries whether that can help explain some of these differences frankly our providers prepared um and then i think the other piece is how do we actually get providers due to the right thing for every child not just for the one child a day but for every child because again we are not asking much in this index we're asking for eight or nine items to be done properly basic uh assessment of the child and so i think that my last slide oh uh no that was just a backup slide for those interested in what exactly was was included in these quality measures so thank you very much thank you margaret i'm sure we're going to come back with questions on this study a number of myself um barbara i'd like to hand over to you now um barbara's going to tell us about some work done with colleagues in who on developing global indicators for quality of maternal and newborn care thank you very much good morning everybody um um sorry um so yes thank you for the introduction so um i come from the liver for school of tropical medicine and i'd like to share with you the findings of a feasibility assessment um run with two so two with four other people but two of the courses are actually here in the in the room today um on quality of care indicators developed through um with WTO so the background is that with the increasing focus on on quality of care but lack of standards there was clearly a need to develop those um as a matter of obviously being able to measure what is actually happening with the quality of care so the discussions had been ongoing um as part of that group since 2010 then in 2013 um 70 um experts got together to discuss what would be the core indicators that need to be included to cover uh quality of care um with regard to mothers newborns and children and they came up with a list of 19 indicators uh six on maternal care five on newborn care um four on general service readiness and uh audits or death reviews and then four uh in which cover children's health which we had not included in our assessments since that's not our area of expertise what we did is uh we did not collect data prospectively for this analysis instead what we did is we um used the uh used data in the repository of the center for maternal newborn house coming from two different programs one um which ran in 10 countries so that's nearly 1000 facilities in Africa and in Asia of different levels so both BMOC and CMOC facilities data collected between 2012 and 2015 as part of a baseline facility assessment so we've used tools um adapted tools from from those um produced by WHO AMAMDD which covered both service readiness as well as outcomes and then we also used um for for one of the indicators we used data that came from a study done post Ebola epidemic again similar tools extended us but adapted to to the to the actual work what we did for the assessment is for each of the indicators we tried to identify um data within our set that that would match the indicator where that was not possible we we tried to use proxies then we extracted the data and then looked at the results um so here what we're presenting is the availability of information per indicator um and then as next step we tried to assess the indicators with regard to the clarity of the actual indicator the wording of it but also the availability of data with within the routine registers and facility records and um where when necessary or where appropriate we try to make some suggestions in terms of how you could possibly make the indicator more feasible for use so this grid gives you an overview of that matrix of clarity versus availability of data I don't necessarily want to go through all the details obviously you can you can read through those in in the actual bulletin but what is maybe interesting to to notice that um out of the 15 indicators we looked at we've actually so 10 of those are clearly defined already though only four are ready to be deployed as is at the moment um we've got um five indicators for which data are already available and then the further 10 need maybe um additional sources of information or certain tweaks to make them to make them usable in terms of then be the major issues for indicators which were not clear not ready to use um for the clarity and adapting of of terminology um so there are examples such as use of terms like prolonged labor which needs to needs to be unpacked it doesn't kind of necessarily follow the same standards across the board it is not necessarily used university across similarly for severe systemic infection terms such as operational again maybe subject to to some interpretation and then use of stockouts so I know that last year there was a discussion led by the by the WHO in terms of what stockouts actually mean and I think the systematic review showed that there were 56 different um definitions of of the terms so obviously that's that's a that's a recognized limitation but one that also affects the the indicators proposed for for the quality of care assessment in terms of time frame so again some of the indicators do say that within a specified time frame it would be useful to actually um already specify this probably for things like um so definitely for things like availability of drugs but also for for other indicators to make sure that the standard of measurement is is same um across the board so you can make comparisons more more easily in terms of information availability so we've got some indicators um whereby information is available but will not be available through routine records you would need to go through patient records for those so um things around uh measurement of of blood pressure amongst women coming for A and C visits but um also for for women being treated for clamptail with magnesium sulfate but then there are others like the one on receiving oxytocin within a minute of of birth now that's that's a very good indicator in terms of being very precise whether that's actually feasible to use in the in the field this is maybe a little bit more questionable so um again something that uh that might require maybe slightly different sources of information such as observation or review of patient notes now some information is just generally not available immediately in the in the routine records so that comprises um essential elements of care for newborns um kangaroo care um availability within facilities and oxygen supply so you would again need to maybe adapt some of the tools to to be able to answer this now the limitation for that is that it's got resource implications so again um the the balance between having a lot of information information that is useful um but if yes if you need to adapt tools if you need to start doing observations it will cost more money you would need more people to to deploy those those indicators in terms of data availability so where indicators were identified as available generally our finding is that you will find information so i think in terms of non-availability up to 10 percent of facilities across the different countries would have missing information which is really um somehow it shows gaps within a particular facility rather than a system as a whole with the exception of obstetric complications which are notoriously difficult to capture through routine records the registers rarely have specific columns that would capture this information instead it will be added somewhere in the in the um in the labor register but that's the probably the the indicator which is most volatile but it affects three of the indicators that we're actually trying to assess quality on so it's an important one to make a note of in terms of missing information similarly for newborn death um information on weight categories not necessarily available throughout um maybe sort of that's an indicator that could be somewhat somewhat simplified before it's deployed and um also in terms of sepsis so women with with sepsis the indicator at the moment suggests that we should look also the readmissions which again a great idea and a kind of a great way of of measuring um quality of care but linking records of women who might have been discharged and then coming back to the facilities is very difficult to capture on uh in the through routine assessments and then there are also some country specific challenges so in terms of stillbirth countries like South Africa do not have records on the macerated versus fresh stillbirth everything is combined together here we're only trying to look at inter interpartum um and then uh similarly for kind of for other countries there are gaps so Asian countries Bangladesh Pakistan not necessarily best record keeping in terms of stillbirth so in terms of the overall assessment um the the the indicators that we looked at so the 15 um that there are that great in terms of the scale of the work so they include um input process and outcome indicators they cover um the the critical elements of care which which need to be included in terms of having a sort of a tester for for for assessing quality um there is a mix of denominators so some of the indicators cover individual patients are this goal to facility level others yet need to be extrapolated to districts which is good on one hand because it gives you a mix of um somehow perspectives but at the same time at the same time it makes it a little bit maybe uh more challenging to make it standardized um and then and therefore usable and um a point that um in terms of kind of the different perspectives um that's only the camp provision perspective that is included in the indicator and in these indicators so the overall conclusion is that further work um to ensure the usability of of indicators on quality is is needed however the the list of indicators proposed in in 2013 and 2014 is is actually is a very valuable contribution to get the discussions going to um start putting a framework together in order to be able to assess quality of camp thank you very much thank you bible thank you so i hope you're all holding on to your questions for the individual authors here we're going to hear the last one in the series now and then we'll take those questions claudia hanson from the carolins Institute is going to present a meta analysis of random mice control trial data on neonatal survival and community approaches to reducing that thank you yes thank you very much giving me the opportunity to present our work and and i present you on behalf of my co-authors for sure and i would like to put it a little bit in the context not also what we heard before so this paper is a little bit written with probably more of the same is not the right thing to do so that's how we actually started to uh to to think about and conceptualize this paper and moreover at the beginning the title was actually what can community approaches actually help for weak health systems so it became something a little bit different what it's now but i think it has quite good messages so we so i was involved in the in the uh in a big trial using a community approach with home visits to improve newborn health which failed and that was also one of our point of departure so we looked a little bit through what has been published and the background is that 2009 WHO said home visits to uh for mothers in in high mortality uh regions can can reduce neonatal mortality probably by something like 40 percent so then there were quite a few meta analysis done then there was our trial and also another trial which we which we identified and we did a new meta analysis and what we did differently looking at chili and trying to ask out so the question why did our trial fail is we sorted them by neonatal mortality and saw that there was an effect actually when you looked at the trials which were can you continue no no one more yeah so there was an effect if you looked at the trial which were really done in very high neonatal mortality setting of something like 25 percent when it was lower still high mortality it was only 10 percent and then when you go down and the trials which were done in neonatal mortality setting of below 30 there had no effect at all anymore so the next next slide yeah so then we also looked a little bit at the health system factors and we saw we looked also what did these trials actually do not all trials actually reported whether they were able to improve facility delivery which is a key thing because you want to improve the uh the uptake of care and it's clear in settings where facility delivery was already high these community approaches didn't do actually much then we also looked at the different context these trials took place and we realized that one of the trial which was very effective in reducing neonatal mortality had only two facilities per 100 000 population but our trial which failed had 17 so there was much more uptake of care already going on in our settings than in other settings so we saw really a trend that these community approaches worked less so in lower neonatal mortality higher uptake of care and settings with a with a actually better established health system the next one so what does it have to do with measuring quality quality of care so what's the implication now for measuring so I think it's important to know what you you have to know your context where you do what what you want to do and I would like to really strongly express that we have very different context and we have very rapidly changing environments and I like this picture a lot which is from a small hospital in southern Tanzania so I worked there 2001 and I came back to do the trial and this small facility had 500 deliveries in 2001 and had more than 2000 deliveries today and access completely changed their mobile phones their motorcycles as you see here a completely changed environment and 80 percent of the women today deliver in in facilities so why why we have been good now really to see if you press one more really to see how many women are coming we have the demographic and health service we really know this we have a pretty good idea we also get better if she's up with the readiness of facilities we have the health provision assessments but we are not really very clear about the clinical practice what is really the effective care delivered and we assume that our trial got really stuck in the health facility quality of care so improving actually that women are coming and that they bring their six children doesn't mean that mortality declines if the quality of care is not good enough and quality of care is something really about giving the right woman the right child at the same at the right time really the right measures which is pretty difficult me as an obstetrician I have been struggling for this to learn actually five years so the next so how can community actions now support the delivery and probably also the measurement of good quality of care so I think these approaches which we have been looking at have much complemented actually weak health system so the question is now can we design other ways which are more supportive better strengthening and can improve some accountability so we would see this as a way forward actually what community approaches which I really needed could do in the context of the weak health systems thank you very much okay thank you Claudia the floor is now open so who would like to have questions either about the conduct of this study the results that the authors have shared with you here areas for future research or difficulty with with indicators um take your questions answering please thank you very much just to comment on the last paper because um you metro analyzed a lot of the trials I was involved involved with I'm slightly confused as to why you would metro analyze what's called community-based interventions I mean if I said to you metro analyze hospital-based interventions you you laugh at me because hospital-based interventions cover everything from surgery to medicine to pediatrics to obstetrics um in the community these were totally different kinds of interventions one you know the women's groups were very largely working in rural extremely remote poor deprived marginalized populations with high mortality rates and the mechanism of reduction of mortality was much more to do with changing behavior social support and if it did affect choices about care it was perhaps improving the selection of care at a time of a crisis where you're faced with very limited options so I'm finding that I absolutely agree with some of your conclusions by the way about the importance of context but I don't see how you can metro analyze such different interventions yeah anyway you're you're fully right and we have turned around this paper many times since thinking too right so we have the criticism ourselves so that's that's clear but still all of them actually wanted to improve certain healthy newborn behaviors yeah like breast feeding and they all wanted that women seek care so in that they were actually so the pathway was on so the approach was different but the pathway how they could affect neonatal mortality was probably not so different because all of them wanted to have more breast feeding less warm care not putting the baby in cold water so they all wanted to achieve this and care seeking when the baby was was sick so the pathway was probably a similar thing but the approach was different and I also think in this I even think that the in in in the new context we have now because I think it's really so much different than 10 years 15 years ago in my for me it's I have been in places where I've been 15 years ago it's completely different situation I assume that this accountability and the women approaches actually have more potential than really the home visits yeah because they can now open up and actually support the support the health system so I agree fully with your concern and this paper actually drove me mad because I have done all this meta analysis at least 50 times who's next yes please very much for all the presentations they were really stimulating and I think it's unpacking an area where much more needs to be done as Margaret said many questions I guess but to start with Margaret when you described the variants you saw across quality of care and systems you then mentioned zooming in on pre-service training as a potential factor to say my dad explained some of this variation when you looked at the systems that countries have are you thinking also of looking at other drivers that may look at quality such as incentives for performance insurance could you speak a bit to that and then for the last presentation on the community interventions you mentioned in settings with very high mortality you could see a positive effect of home based care and could you analyze could you describe in more detail what was actually required to get that intervention on place was that done in a study context with all the support or had that dimensions that were more comparable to routine health systems because where the system is weak and you put an intervention that's not supported then that results that one may see in the study that is fully designed to kind of compare the intervention with the control that and that effect may not be replicable Margaret would you like to start and then Claudia on the next one thank you so much Bernadette you can see Bernadette is one of our commissioners it is getting right to the heart of the matter the absolutely so pre-service education was one possible explanation for the very large country effects that we were seeing but I think the broader point here is that there are factors above the clinic that's really maybe the main point to make they're all baked into this word you know Uganda or maybe that we that we essentially use in the model as a stand-in for everything that's going on in the broader environment for these providers delivering care so I think going down your list absolutely so pre-service education but even before that the selection of providers and the secondary education that they receive and are they ready to become a health professional is a harder perhaps and an incredibly important issue and then once they graduate what is the what is the payment environment right how is the motivation contributing to this to this performance absolutely insurance system what about regulation and how how do they feel do they feel the obligation really to to do their very best because we do know they do less than they know so absolutely all of those factors would be critical to examine and the reason we were not able to is because actually there aren't good systematic measures of them that are available for us and that's going to require much more in-depth studies within countries rather than so much across countries so thank you for that question so I will add on a bit on the on the context the first two the papers mostly really reported on too little also on the context they were all done quite program in programmatic to context so all in in in normal health systems and many of the differences between the health systems we actually couldn't analyze because this data are not available so like for example health workers and how fires the service but it was very clear if you looked a little bit more in detail that the trials which were done in lower neonatal mortality settings had higher health in newborn behavior and that we also would expect here they had higher breastfeeding higher facility delivery better warmth care so and then it's actually logic that the effect of of counseling the women better can't be high anymore because how how should it how should it work if you increase facility delivery by five percent what should be the effect on neonatal mortality it can't have yeah because it's not it's not logic and that brings us back that we have to know this and we do not have to and we have to know it for certain settings because they are huge variations also in the countries so not only in India but even in Tanzania you have a facility delivery between 30 and 90 percent you have to know this yeah and one approach of community might work in that setting but might not work in the other one and we don't have the measurement data for that we cannot even if if development aid goes some somewhere they don't know what is right for that setting because the data are not there yeah we have them we have something on the national level but we have seen with a large last paper also clearly at the subnational level at the facility level the data are not there there's a lot collected but it's it's I could now report for hours why it's not usable you're running off the hook yes please over here all right thank you I'm Andrew from Malawi I'm an interested party if the presentations congratulations of the presenters the question I have is how can we translate all the recommendations and findings into action thank you we'll go we'll go along left to right Barbara one of your findings translated into a recommendation please and we're going to ask every author to give us one thank okay so thank you that's actually yeah it's not an easy question it's not easy for the first one to have to answer it and in terms of the recommendation for the indicators I think the work is already ongoing in terms of updating those because by all means we need to have measures that are standardized that can be usable in terms of action for actually changing quality it's then persevering and measuring not only for the sake of measuring but actually doing something with the findings so it's the whole quality of care loop that's probably not a very specific answer but in terms of the the indicators yes they need to be updated and they need to be so we need to have a set that would be usable across and at different levels okay you got that update indicators right and okay for for these and Kenya measurement basically the next step is what interventions to try in terms of behavioral change for hand hygiene that is the cornerstone of IPC so I have been thinking already on some interventions to do the government to try to test because that's the most important is the knowledge is not and a supply so how do we change that so that's one but there are other ones that you can think of there the million dollar question and I hope our our second panel will actually get exactly the heart of your question because there will be policy makers who will tell us some of their perspectives but to me it is the combination of two things passionate leadership and country that doesn't come from research that comes from something else and but secondly that passion and that commitment meeting relevant timely research on topics that matter and that part is within the realm of the research community so are you wide open please tell us no well I I'm not going to give you the the full answer but I mean I might want to just make a reference back to somebody called Cain's he was a key economist and when he was thinking about how do you influence policy making he did say well first of all we engage but he did also say that what goes on in policy mine policy makers minds often reflects finding from previous research and academic insights maybe the link is not direct but the link is there so bringing evidence to the table bringing evidence forward to policy makers in a clear and useful way is an important element in this process and I would say that that's sort of part of what we are trying both with this issue but also of course with the policy relevant work that is carried out the country level thank you okay over to Claudia I'm not sure was any question still addressed to me I'm not sure the question is for each author what is one implication one action from your findings one implication of your finding so are you suggesting that community approaches should no longer be funded once countries reach a certain development level they're not going to have any effect I think they we should we should rethink the role yes accountability yes yeah but just supporting and preaching facility delivery when facility delivery is 90 percent and the quality of care is crap doesn't have anything here here all right over to grab and that's one so from our work I think the main recommendations were that measuring quality of care is important and we have presented an overall essential care at birth index but if people are interested to look specifically at newborn care or obstetric care they can also do that so the index can be disaggregated into different domains so we are advocating that there's more measurements a systematic measurement effort including in the private sector which provides a substantial market share of maternity services and there should also be further investigation into the high prevalence of unqualified attendance working providing institutional services so what we found is in the public sector because human resources shortages are so immense doctors are not available they tend to rely on unqualified personnel in the public sector but whereas in the private sector they're much cheaper to hire than say you know hiring a qualified nurse or and you just hire them train them on the job so they're much cheaper so there needs to be further work looking at why unqualified personnel are being used and lastly I think we need quality improvement initiatives in hospitals on a large scale and link those two functioning accountability mechanisms so those are the three so me as an obstetrician I always think there's really a huge danger with this facility delivery if the care release not good we had the highest ever maternal mortality in French and and a few other European hospitals some 200 years ago we don't have them that high in the low income settings that was because of all this issue of sepsis so if if hospitals don't provide good antisepsis techniques they actually could kill mothers instead of helping mothers and babies and that's my my really big concern so and we have done a lot with really getting the women now in and that's nice but it's it's more or less now really our safety obligation to to get this now also really safe otherwise we are doing the opposite of what we really want and I'm there always a little bit worried yeah crap is maybe a little bit too harsh yeah but I'm really worried as an obstetrician and worried because of the over so of the sepsis and also using too much interventions and influencing too much deliberate care also then in a way which is is not really helpful for the mothers and the babies thank you so so I just have a comment a question really a bit of a reflection maybe especially for Margaret and Gaurav but it's really for anyone who's been working with an index so I came along here thinking measuring quality of care I'm going to hear lots of geeky presentations about psychometric properties of indices and all that stuff which has a lot of baggage attached to it and I haven't heard anything about it at all so I just wondered if you had any reflections on that I'm quite happy not to but but actually then maybe there are some issues there that are quite serious about measuring measuring things and what do they mean sure I can start and then Gaurav well Joanna be careful what you ask for we can certainly delve into that as well yeah you know we've spent a lot of time thinking about this this issue which is really how do you aggregate quality is a multi-dimensional construct so let's start with that we I said this many of us said this many pieces matter right and we get it glimpses of it really coming forward so in our group at Harvard and within the commission we're actually spending a fair amount of time thinking about the psychometric properties and we've examined different ways of combining because they're really the key questions how you combine these indicators into a whole you know principle component analysis which ones matter is it you know how do we weight them what's more important and I have to say having explored and this is a very partial answer and it's about these particular observed care elements so this may not hold for every indicator in the book but one thing we're finding is that it's actually pretty simple additive methods do a very good job and are very clear and I think this is about also important and to Andrew's point the clarity of explanation and being able to say what is it we're doing there's no hocus pocus we're doing the following the one thing we are thinking hard about in in the group is obviously there are sometimes many more kind of microscopic actions that sometimes accrue in a history taking right whereas there are fewer available physical exam items and so to the extent that both history and physical are important sometimes what we're doing is we're taking averages across those elements rather than weight you know because if you don't do that then of course the item with the most elements wins in terms of its weight so we sometimes are doing equal weighting of the components of care based on the basic medical notion the good medical practice requires completion of steps of care that's about the only adjustment I would say that we're making but I've seen others do you know much more and we've tried and don't find it illuminates go everything wanted to say something so I mean we had a lot of we had a lot of data so we thought carefully about how to make sense of all of this and for the 42 practices we essentially map them so for the 42 items we map them on to clinical practices so based on their logic clinical logic as well as purpose so for example for let's say something like active management of third stage of labor so if everything was done only then were they given a one otherwise they were all given a zero where some other indicators so for example monitoring regular monitoring of patograph use that was a single indicator versus some of them were composite indicators I mean I know other people that have kind of undertaken Delphi type methods or use even advanced modeling and all of that but we tried to keep it very transparent so that you know people could use it as it is and we said at the very minimum these are the essential practices that must be done for every case so we've tried to keep it quite simple but obviously I think although it's useful to communicate a large amount of information when used for quality improvement purposes people will need to really look deeply into what is driving the index so for example let's say skin-to-skin care might be really poor which brings the entire newborn care index down or respectful maternity care might be poor which is bringing the offset to care index down so I think it'll be important to look at you know with specific components and further I mean why does why does a particular thing happen so for example if rates of active management are poor is it because injections are not available or is it because of training is it you know what is the issue so it's like so although they're useful to communicate in a concise way I think there's lots of limitations thank you very much I mean that is obviously an extremely important topic let me just reflect not specifically with reference to the health sector but but but more broadly within the development field I think one of the key lessons that is emerging is first of all the clarity and transparency in both the definition and the use of indices is an area that requires more focus more attention than it has in the past there's been a tendency to put forward interesting fascinating indices just to sort of provoke the debate did you know that the multi-dimensional poverty index used by the UNDP does not adhere to the declaration of human rights that is actually not respecting the fundamental rules of the game according to that so I mean what I'm trying to say is that as researchers as policymakers we really need to try to insist on clarity that we do understand what goes into a particular index and we actually understand what are the assumptions lying behind now that would often lead to simpler methods simpler more transparent indices and then in all cases make absolutely sure that they're unpacked that you do try to think about robustness that you do try to think about what the individual elements of a particular index actually is doing and then finally there are methods being developed now where instead of relying on indices that you can think about how the whole distribution is actually moving this is an area that's called first-order dominance but there is actually work there that can push our insights forward which do not depend on the exact waiting of specific indices thanks yes just a quick point to build on that because I think Senator Margaret's points on on waiting has always come through and I'm glad it's been brought up although probably better that it wasn't sort of a feature of our own discussions here yeah someone who spent a lot of time on on indices both in the US and then on for work at WHO we are working there's continuing work in a number of different areas there's there's a tension between colleagues of which I sort of would consider myself from from the data side who look at these indices and sort of roll their eyes because everyone knows just based on the points with Finn's comments on new science accepting however you wait it countries will end up as first on the list or 13th on the list or 15th on the list or communities or whatever the group is and but for instance when I when we did the first US national health care reports I sat in the back of the room on while my boss presented my work that's a strong tradition in big bureaucracies and the sort of the committee staffers were there and some of the representatives and senate people were there and they would hold the books out there and they would flip flip flip stop at a map flip flip flip stop at a map flip flip flip stuff so they were looking at these composite pieces we put together for the states based on state sort of how they were doing on prevention or how they were doing so they matter because they grab you can sum up very complex things and you get people's attention but immediately you need to explain then to say well here's the reason why and personally I think if we could move more particularly in the quality field because one of the issues we also have on the data side with indices is that often these studies were looking at specific populations so that's one solution to it but often when you're talking about bigger groups or policymakers they're concerned with different groups of populations and right there the statisticians heads explode because you've got elderly people and that you're trying to put in one index with perhaps care for newborns and it just doesn't work and so for me one of the things I think we should move more on is this whole idea of how what's the rate of travel how are we improving and rate of improvement and percent improvement and in there you can start looking across different measures and how quickly things are improving so personally that would be my we had to come with come up with one index in the future that we measured it would be how it's on a subject you'd approve of. Could you just explain the Uganda findings being 30% better and the appalling Indian findings on quality of care by simply the quality of training of midwives because it just strikes me that Uganda has probably got better qualified midwives that's why this is just a hypothesis and we know that in India there is no midwifery cadre and that auxiliary nurse midwives most of them when they are passed out and given their certificate of training have never actually delivered a baby and this is some new stuff that's emerging and actually we're working with the ministry and with others at the moment to see if we can come up with some solutions to this to try and develop midwifery within India which is a major issue because one of the major obstacles are other cadres of health workers that actually obstetricians and nurses are not that keen on having a separate cadre of properly trained midwives which doesn't seem to be a common thing in Uttar Pradesh and finally WHO needs to get its act together because the definition of an SBA is all over the place and everyone is calling I think on your data you'd probably classify the doctor the nurse and the A&M as an SBA but actually most of them don't have it I think I'm an SBA you know I'm qualified I'm a doctor I've actually done lots of deliveries I'd be a disaster at a delivery so Anthony quick response and then a great question and I was just referring back to our table four in the paper where we show that for anti-natal care compared to a nurse or midwife physicians had eight percent lower performance actually and nursing assistants three percent lower so this is now adjusting for all the country variable what would be very interesting to see how those nurses midwives performed in say Uganda where the training might be better but these are precisely the kinds of conversations that need to be happening we need to be looking at the structures the basics I say no I completely agree that's that was one of the reasons that we decided to use the term qualified versus unqualified because we didn't really assess the competencies but I know like Japaigo did a study recently where they looked at the training curriculum of auxiliary nurse midwives and said they don't fulfill the you know all the 21 competencies outlined there I think I mean in addition to measuring who an SBA is I think there's not much research on the enabling environment for SBAs which is the main issue so I think maybe in measurement efforts trying to measure the enabling environment at health facilities rather than I mean and focusing on the content of care rather than you know is it a doctor or a TVA or whatever that's okay one last comment from Claudia and then we're going to have coffee I just would like to make two comments first with the midwifery and why it is important so I'm also always when I'm teaching I'm saying just look at the two books from WHO the integrated management of childhood diseases and the integrated management of pregnancy and childbirth the child houses speak like this the pregnancy is big like this so you can't do it as an in-service training you cannot put it on lowly trained health worker the second point is the is the retention of of of practices most of the nurse midwives are rotated every half a year so even if they are a little bit getting into it after half a year they go to urologist or wherever because it's not a protected card if you would have midwifery we also would have a protected card which stays in the maternity we're going to take a coffee break and come back at 10 past please coffee's outside thank you thank you lively we will be discussing some of the critical issues around measuring quality that's been raised in the last session as you may have already seen in the editorial this issue highlighted one of the reasons for the lack of data of quality of health care services in limited income countries was this emphasis during the mdg era that margaret mentioned on coverage rather than the challenge of providing high quality services and then as highlighted by the authors earlier and presented in places where studies have been done these quality measures that currently exist and used have their own limitations some are not sufficiently validated to be recommended for use in other limited income countries for various reasons measurements of qualities were not done consistently and and therefore in such cases were not able to generalize the findings be beyond the steady settings and compare these findings for other stations so what we want to do in this next session is to give our presenters an opportunity to comment specifically on the validity and the practicality of some of their studies so we'll pose two questions what do they think are the main challenges with measuring quality of care and what do they see are the greatest gaps with measuring quality of care since we don't have a lot of time they don't need to answer both they can pick one and and be able to and comment we'll give about few minutes about three four minutes five minutes for comments and then we open it up for discussion so we have again our authors for this panel i'll start in the same order okay okay thank you i have chosen the question what are the greatest gaps with measuring quality of care i'm going to focus on two main ones the first one for me is a tools data gap in my specific example we chose or we built on tools that were already developed validated by the WHO and they are available in their webpage and also in the CDC's webpage and we were very surprised that there was no data even for a few health facilities in Kenya and in many countries in Africa some of the data that was available was like one decade or 15 years ago so it was very difficult to find you know like information about what is the status what so i think one of the things even though we need more data and more different type of data we are not even we haven't cut up even in terms of for the tools that are existing having like basic data from countries especially in low and middle income countries and nobody knew for instance that you know the compliance with a new syringes and nails was 100 when we presented these results in the government they were happy but it was a shock people couldn't like for two minutes they couldn't talk right like you i couldn't follow for for two for for two minutes the other things measuring for instance knowledge and practice and supplies at the same time gave the government a lot of thought because of course you know like the first thing that you think is like we have to train healthcare workers and that wasn't the main constraint so it allows you also to focus more on what is the next intervention or what is the next thing that you would like to test and another example is that our main intervention is inspections of health facilities in Kenya so we have two different treatment arms and when the government was designing the new regulations for actually for random for the random control trial they have no information on how the health facilities in the country or even a sub sample would face or would would be scoring according to these regulations so we did a pilot and based on that they actually defined that they were they couldn't close all the health facilities because 97 of them didn't comply with the minimum so they changed their regulation so to give people the health facilities to improve in a year so that's the power of data that actually is not there and my second gap is the link to outcome gap and again the main issue that we have found is that it's very difficult to determine what matters the most even in this checklist that I was telling you about it when they were trying to put a warning sanctions into this new regulation they just decided to put an equal scoring to each of the 300 items because there is not a lot of evidence and they didn't dare to actually put more scores here and there so there is very little evidence in a lot of these indicators that we are testing how strong is the link to the outcomes to see which ones matter the most in the example that I show in the paper for instance it's two percent of hand hygiene but does that matter more is that more important or is equally important that is in the examination or in the lab or the injection room or in other settings of the health facility is that more important that other you know waste aggregation when we cannot actually follow so try to understand where to put more weight because that will define in a setting that is very constrained in terms of resources it's very difficult to go to the government and say focus in these 500 items equally and we could have more information and again we try to find evidence we couldn't find a lot of evidence that allows to actually link all these indicators to outcome those are my two okay next would be great thank you Kitas you asked first about challenges I think they fall into two categories the what should we be measuring and secondly how are we measuring it I think I know what there are legitimate questions about what elements of quality matter we said there are very many elements I don't think there is sufficient clarity we even saw it when the papers were submitted for the special issue there are many people doing many things some of which are peripherally related to quality it's difficult to say something is completely unrelated to quality actually because it's such a large construct but but some clarity about common definitions I like the point I think we're made several times about even for very narrow sounding indicators there's just not agreement on how are we supposed to be measuring that I think what to measure matters at the micro level having the same definition for indicators but I would even say at the kind of deeper level like where should we be investing our resources in measurement measurement is expensive it's hard to do so of the thousand things that we can identify that are related to quality which 10 27 should we be measuring and one pitch I will put forward and this is work that's ongoing is that while inputs are important to to quality for sure you need stethoscopes and medicines they are not themselves actually very descriptive of the quality of care that people receive and so one push for what matters is further along the continuum toward what actually happens to people in the care context and also their outcomes so that's that's my vote for what in terms of how I think we have a couple of problems that that we're seeing what is survey crowding I think we keep complaining there's not enough data but I actually see in some ways the opposite there is a multiplicity of surveys when I sit when I think from the outside sometimes it seems like oh there's not a lot going on or datasets are not available that we could look at in the way that every for example wealthy country looks at its own data those datasets are public and people can can can examine them that doesn't happen as much it's improving for some of the surveys health facilities surveys for example and others but when you actually sit down with a health ministry official in charge of data they say oh yeah we've done 10 surveys of facilities in our last in the last five years I'm thinking about our friends and colleagues in Ethiopia and I know we're going to have the former minister with us but I mean that is a data rich country the service provision assessment was done a very large one including a census of facilities just a few years ago a gigantic census of basic emergency etc care was just done a new spot is going to the field and I think that's maybe an extreme example but there's a lot of surveys now the flip side of that is that often the only people who know the results are the people who wrote the report and I do not see a direct link between many of these surveys and action in fact I don't see reports and strategies citing these surveys even though they're maybe very well done and nationally representative so there's a big disconnect and a lot of what I'm thinking of is dormant data in countries so on the gaps I think there are a couple of gaps I want to emphasize I think there are many but I want to emphasize too one is the people's voice the user voice in this entire enterprise you know my entire research career did not start with assessing health system quality it actually started by asking people why do you or do you not want to use the health system I was interested in utilization and I was interested in people's preferences because I was struck by the very different levels of utilization of essential services in sub-Saharan Africa so working closely with colleagues there over the past decade it became clear that quality was a driving force in people's decisions and so now we're shifting our attention to understanding okay well what quality are people getting but I'm not forgetting the user I think what's really incredibly important is understanding people's experience I think the maternal health community has really grabbed this agenda with respectful care and measuring that and developing ways to think about that but also the outcomes of care that mattered to people already mentioned this before patient reported outcomes perceived quality what do people think about the competence of this provider because we may or may not agree that they can tell all the elements of course they cannot in every case but it's certainly their perception certainly will drive what they do right so knowing that perception is important and by the way that is linked with another complete gap which is understanding what is the demand for quality in any one setting I think why many of you will be familiar with the observation that many people are suspiciously satisfied with care that some of us I think would probably many of us in the room would consider completely terrible in many countries why aren't satisfaction ratings so high 90 percent 85 percent 94 percent you know and and of course social desirability bias and surveys people wanting to be nice and kind are all the driving things but another factor might be that there's actually frankly just a very low expectation that people have and as we think about this issue of improving quality it seems impossible to move that mountain without people demanding better care and so thinking about demand generation in the population for better care just like we drove demand for utilization right is absolutely critical as a gap and I think the other final gap to mention is that and this relates to your question about how do we aggregate things is it's one thing to understand how how people fare for particular need or service but for the ministry of health they're often more interested in how is my primary care system performing or how are my hospitals performing is referral working so rather than taking a disease by disease view or condition by condition we're interested in a system assessment of quality is the system performing as it should and I think that's still going to require quite some thinking about what's the right set of things that convey a system quality performance yeah I would like to add on a on I agree with all what has been said and these are the known challenges I would like to put now the health worker as a key person which has not been put in the center of quality of care but these are the ones actually providing the quality of care and we meet with the data we collect we also need to trust the health worker like we have done actually in the in the high-income countries in a health worker reports active management of labor is done within a minute then we cannot come up we better do observation to check whether it was really done in one minute yeah so I think I find this ridiculous so we have to trust them really to some extent because these are the people in the end really providing the quality of care and we have to listen also to to what they say we have done a qualitative study and the health worker said it's all like rain sometimes you get supervision sometimes not sometimes we have drugs sometimes not so that is also not a good environment for really providing providing quality quality of care so this is a this is a challenge the other gap which I see is that we have been under utilizing a health management information system so we have a lot of service they are somewhere they're even not on repository so districts can't use them districts don't know what is there whereas the health management information system is actually in their hunt but it not in a way that they can easily actually use them and it's also very difficult to use them for for more comparison I just said a PhD student who tried from Cambodia and it took a whole month to compile from three provinces data over over two years this cannot be so they must be somehow readily available readily like you have it actually with the demographic and health survey so we have to we think with our technology we have how can you utilize health management information system to drive quality of care I think I mean so much has already been said but just to add to what Margaret was saying I think linking information from different sources different data sets that are already available for example I mean data from clinical observations with EMOC assessments or service provision assessment results and actually seeing what is being done or that is important and also I think it's it's useful to try and get as much data as possible so for example there is some evidence that provider effort is the key determinant which determines like quality of care so information on their motivation or the incentives that exist or management structures within the facility I think getting that sort of information is also quite important apart from that I think I think having a strong research team locally who knows the in that community and ensuring that high quality training program is given to the research assistance is very important because some of these concepts so for example around respectful care or other things you know you really need to pay emphasis to these things and like observations are a good method but they also have many limitations by themselves and there are also many other methods to assess processes of care but I think one thing to keep in mind is the possibility of any observer bias or author and effects that can occur and if you plan for these things from the start if you if you I mean make sure that you can order the observations by the time that they started by every individual research researcher you can kind of account for all of these things so I think those are some of the challenges okay thank you very much so I think I'd like to echo a lot of the comments that have already been made but just maybe to add to the to the discussion on challenges in measuring quality of care so as I mean as already has been mentioned the topic itself is really broad so looking at the example of the special edition of the bulletin I mean the diversity of topics covered is broad therefore kind of it's quite difficult to maybe find ways of capturing things in a uniform matter so one of the aspects is the multiplicity of stakeholders involved in the work so at global level you want to have trace markers of quality that will be applicable throughout the lower down you go in terms of facilities there are many more that that are important that need to be measured and reviewed locally to improve the quality at the level of the facility in terms of the user perspective so that's that's kind of somehow almost a different type of discussion in terms of what Margaret has already alluded to in terms of what people's expectations are which sometimes are dictated by lack of experience and kind of but sometimes it's kind of its cultural aspect as well where you don't challenge what you receive obviously having some sort of care is better than not having any care which then links to benchmarking what kind of how do you define quality is receiving any sort of care and improvement on no care of course you want to strive for the best by kind of then you also need to take the limitations of whatever is available on the ground as a limitation of the kind of the level of quality you can aim for and then the big issue is then is really around data so we've I think kind of it's it's it's nothing new to kind of to say that there are quality issues in terms of data available within the facilities to help measure things but it's also an issue about timing so sometimes so data should be recorded at the time but by the time they are available very often they're out of date so again it's already been mentioned the context changes very rapidly things change very fast so how do you actually tap into it in so sufficient quickly to be able to act and actually improve quality in terms of source of information so within kind of our work we've looked at facility information so the registers and records do not cover everything that would be relevant the systems that are used like the HMIS will not cover all of the aspects of quality that are important it should somehow be measured but then there's then this issue of the alternative sources of information like exit interviews like observations which I've already mentioned in my presentation have further kind of costs and resource implications which need to be taken into consideration but that's a challenge for measuring quality but then another issue is the difference between the records that you will come across and what the reality actually is so there are so the poor recording systems is maybe one aspect but there's also sometimes deliberate action on behalf of the people capturing the information who are under pressure or for kind of alternative reasons put in information that isn't necessarily reflecting of what is happening so the maybe kind of find out a standard example is filling of paragraphs post delivery because you need to meet certain numbers so kind of the quality standards you're trying to fill but filling a paragraph once the baby is out doesn't help anybody apart from maybe the record-keeping staff but then also the fact that if kind of the quality improvement really is based on a punitive system so if something goes wrong the only thing you'll get is scolding from your supervisors from people from the district level chances are you maybe you will not want to share information if you don't want to share negative outcomes chances are you will not be able to improve it because then nobody realizes exactly how how big an issue is and then the final point maybe in terms of linking of of coverage and then kind of measuring quality versus outcomes so the high rates of of coverage we've already discussed doesn't that doesn't necessarily mean that the quality will improve so an example here would be the WHO multi-country survey on maternal newborn health from 2013 where out of the 29 countries assessed the ones with highest coverage rate for essential care were also the ones with highest mortality so there is obviously kind of a quality gap but how you tap into this again maybe it's not so clear so kind of again the safe childbirth data analysis which again shows that coverage is improved adherence to essential practices during childbirth is actually improved if you look at the standards being followed but the outcomes don't necessarily follow this so then the question is how do you tap into it so these are some I hope kind of um again not kind of to paint to to better picture of where things are because obviously it's a complex issue it's one that requires discussions it requires kind of coming up with some sort of consensus on how you measure it but challenges exist and I think kind of it's an exciting challenge to have to maybe try to input into how to how to overcome them thank you okay thank you I think our panelists have touched upon many things availability lack of data quality of the actual data linking and measuring linking them to measuring to measuring outcomes how to increase demand how to make the data relevant for policymakers with aggregation so many things so we can open up now we have about 20 minutes for questions and comments so if you can just say who you are because we have remote users so use the microphone and comments questions general yes yes look sorry small intervention I know there are people in this room with tremendous expertise actually in this area not only the authors here I just wonder just to really encourage comments and your own perspective please okay I mean thanks a lot for some extremely insightful observations maybe I can try and provoke just a little bit I'll be sort of a little bit dented it doesn't mean that I don't appreciate a lot of the progress that has been made but when I as an economist try to look a lot of the data or a lot of the surveys and a lot of the dormant information that's there I can't use it take Ginea Bissau as an example I have a very good friend medical doctor who has collected information for a long time and he said there's this fantastic database but there was no information of economic social and so on variables in that fantastic dataset so when he came and said to me can we collaborate I said well I would be I would be happy to but what it would take is that I would have to start from scratch because I need in order to generate answers to a lot of the kind of question we're discussing I need to have the economic social and other variables in the datasets there are of course some datasets that do have that but I would like to submit that a lot of the information that's out there doesn't have it so maybe one of the challenges that I can see is how to somehow get some communication around that and if you wish maybe clean up a bit and try to ask the question up front what is actually required in order for the various surveys to respond to the actual questions that you want answers to because it seems to me that that's not happening right now thank you thank you I wanted to just follow up on a couple of issues and Gaurav raised this Margaret when he challenged you a little bit about the observation method which seems to be the dominant method that was used in the surveys I think what we heard from the Liverpool group was how few actual indicators are available for use at the front line that relate to quality and I want to go back to Andrew's question which is what's the practical value of all of this measurement I think it's enormously important that you have exposed through all of these papers the state of quality but dynamic improvement is going to require real-time data and there were none of the papers the closest that it came was the paper from Ethiopia but even there it was a survey that was a team went in every few months to spend three days on site that's just not real life for the front line providers who are going to be expected to be at the front line of improving care so I think there's a huge gap that is revealed by the lack of good tools to measure quality at the front line and I just be interested to know the thoughts of the panel on how we're going to solve that problem because if you combine the lack of available indicators that reflect real issues around quality the complete absence of any measurement of patient experience we have just a giant way to go in trying to bring real-time measurement to bear which is what's going to be required for real-time improvement I mean anybody can comment on it but I thought that it might create some reaction and that was what I was hoping for the different measures of the front line and the timing etc so we we faced a similar problem when we were designing our impact evaluation in Kenya and when the government wanted only to measure this checklist and we were like but what if those are not the set ones that matter right so now we are collecting data with covering for cover standardized patients with direct observation with this one we're trying to have like a wider set of tools to measure with the idea that later we can also say okay where are the ones that were highest color related with the ones that are closest to a patient we are measuring also patient experience but with the idea that is a big story but try to understand where the key things so that we can contribute somewhat to understand where the key tools that the government could measure with more frequency lower cost etc but it's a big question we have very little data on that that connects as it's comprehensive and representative so the idea is that is the whole census of three different states so hopefully we can contribute somewhat to that later I think we do not need surveys for example to assess whether a health facility has all the things which is needs to provide quality of care so we can ask the nurses to report this once a month and in most countries actually they do it only that nobody notice this it's just lying lying there and then I also feel one can maybe look at innovative ways also we can we can have nurse midwife registers and they say I did that many deliveries and I got this many in-service training and I have done this and this and this is my key challenge so that they are maybe re-registered every year so we can think of a little bit something out of out of the normal box of doing surveys which might then be much more sustainable quickly I'm on since maybe is data linkage a way for like this economic you know having economic data and wanted database and if it hasn't been structured from the beginning haven't thought about that will also create problems so I'm just a prompt so Andy I think economic data and the absolutely critical need to understand what quality costs what facility you're spending how are in resources invested actually generating or not generating quality care I think there has been much more work for example and and even that hasn't been sufficient in understanding whether resources are you know correlated with volumes in terms of productivity assessment I think what we are saying this needs to move to is not productivity in terms of patient scene but in terms of patients treated well and patients getting better and those the linkage is absolutely critical because countries we're talking about in this room primarily are have incredibly constrained budgets they can't do everything that we're measuring and would love to do so that so this is an absolutely critical linkage to be very concrete for example in some of the service provision assessment surveys or Sarah surveys they're basic basic economic factors completely missing so actually half our surveys don't reveal whether the facility is urban or rural that kind of matters you know and another one is that we don't know anything about the wealth of the neighborhood or the or the the context in which it's in we do that by doing data linkage with combining data that were themselves highly imputed to try to figure out whether the facility is operating in a resource rich or a very resource poor neighborhood these these things obviously matter these are the very things Bernadette and others that are above the level of the clinic that actually make a massive difference to what's going on there none of that's captured right now and I don't all have to be in one survey but if it isn't then there has to be a clear way to bring those variables in I think on the second point about peer your point about what are these surveys contributing given the need for real time data I don't think it's interesting right you're getting lamppost bias here a little bit the data that we have are what we have and what we obviously what was featured in this in this issue and that has to this point been coming and the best quality data have been coming from nationally represented these nationally represented facility surveys is that the only source of data that's relevant absolutely not what we want to start with with all data actually is asking what's it for who is it for and I've already highlighted and I think others nodded vigorously that actually most of these shares are not even used in country so what's the point I do actually think that the measurement the minimum measurement package around quality has to include facility surveys much leaner than the current version as a way of tracking quality across the country from ministry of health to know where's it moving to see whether people are are getting the care they should be getting including some mix of these tools some of them can be simplified and made more efficient including using cell phones to reach patients getting patient voice in but the piece you're emphasizing is what's HMIS doing for us what are other methods doing for us and I can tell you just look at this quality issue they're under delivering in terms of comparable robust data right now and that has to be an urgent point of action can I just add something so one thing that I just wanted to highlight and Dr. Matthews is also here I know that I know that like the ENAP group is doing a very big validation study on indicators around the time of birth including for sick babies maybe you want to say something I think 12,000 12,000 births across multiple okay okay but I think the their results will also be very useful to you know have the which in the let us know which indicators are the valid ones which can be integrated into the HMIS system and so on so okay I'm Teresa Diaz I'm I took over for Matthews position coordinator for modern evaluation epidemiology in the MCA department there's a lot going on with routine systems and and specifically with DHIS too getting not only interoperability but apps that that provide printouts and analyses and so forth but there's still this problem with aggregate data and I think and I'm really always say this I think we have to have a strong voice about individual patient records I think that you can't provide quality of care without individual patient records as a clinician myself if I don't know what happened the last visit how can I provide quality of care the next visit and being able to have those individual patient records in a routine system you would have a lot more information than you have now and being able to have these more ro