Good afternoon and welcome to the Green Mountain Care Board on this historic day. This is actually the first official board meeting of 2021. And the first item on the agenda will be the Executive Director's Report, Susan Barrett. And just because it's 2021, I have to start the meeting out by saying, I'm on mute. Just also to agree with you, it's been a while since we've all met a couple of weeks and our world is very different today. And I just want to mark today by, you know, in so many ways, having Kamala Harris as the first female Black and of Asian descent vice president. I know it means a lot to a lot of people and it's pretty exciting for me. So, but back to the board business, we have a few meetings coming up that I just wanted to make sure folks were aware of. Today, we're going to hear from our data team and our hospital folks on the federal price transparency update. And then tonight, we have a meeting with our primary care advisory group that starts at five. And then next Monday, we continue to have meetings with our prescription drug technical advisory group. I want to thank board member Lunge and Christina Gloffland for leading those. And of course, all the stakeholders participating in those. And then next week, we're going to hear an update from our data team. So that's going to start in the morning. We have a full day of meetings and starting at 10 a.m. And then in the afternoon, we'll have an all pair of model update on some reports, and we'll have the all pair of model differential reports. And that is all I have to report, Mr. Chair. Happy 2021. Did you want to mention a few reports that are now available on the website? I think that's a great idea. So we did submit our annual report to the legislature, our statute, and that is on our website. And there are some also other all pair of model reports on our website. And I may be forgetting something that you're referring to, chair Mullen, but that was the big one, the annual report. Okay. Thank you, Susan. The next item on the agenda are the minutes of Wednesday, December 30th. Who would like to put 2020 behind us? So moved. Second. It's been moved and seconded to accept the minutes of Wednesday, December 30th, without any additions, deletions, or corrections. Is there any discussion? You're hearing none. All those in favor signify by saying aye. Aye. Those opposed, signify by saying nay. Motion carries unanimously. And now we're going to move over to the subject of today's meeting. And as Susan said, this meeting has been somewhat split up because of all the circumstances of today. We've focused strictly this afternoon on federal price transparency and we'll have the rest of the data team updates next week. But at this time, I'm going to turn it over to Sarah Lindberg for a discussion on the new rule that went into effect January 1st regarding price transparency from our hospital. So Sarah. Good afternoon. This is Sarah Lindberg from the Green Mountain Care Board. I had our data team and I will start by asking if everyone can see the slides that I'm presenting. We can. All right, that's a good start. So I'm here to offer a pretty cursory update on the federal price transparency rule. A lot of information was dropped and so we're still digesting it. And I've been working with both our legal and hospital finance teams as we tried to do that. So as a review, this is a federal requirement for hospitals according to a CMS rule that said as of January 1st of this year, they would have to post two new resources publicly. One is a machine readable file of all the standard charges for their items and services and then a subset of those services that are dictated by CMS in a consumer friendly list. And they refer to these as shoppable services. It will apply to all of Vermont's general hospitals, those for which you do the hospital budget review. But it does not apply to facilities other than hospitals, even if they might perform similar services. The requirement is limited to those rates negotiated with commercial insurers. So there's nothing that would prevent a hospital from including it with other information from their public payers and they could integrate that. But the rule does not require that. As I said, we're actively reviewing the information that has been posted by the hospital so far. Right off the bat, we notice a lot of... Is that like a Casio swatch from the 80s? This is a great sound. So it's right off the bat that I look like there was a great deal of variability particularly in the way that they approach the shoppable services and that was allowed in the rule. So some people posted spreadsheets or comma delimited files. Other people used price estimator tools and other hospitals went with the interactive web form approach. So you pick a service, you pick a type of insurance, you pick a policy number and it will show you the information related to that shoppable service. So here's an example of a price estimator tool on the Dartmouth Hitchcock website. So you look for a keyword or CPT code or you can browse by kind of categories of service and then it prompts you for information about your insurance and then you would see a range of negotiated rates including the one for your particular information you provided. That's an important caveat. And then another example, Rutland has a series of downloadable files. So there's the patient-friendly pricing file, the full scope of their services and then that machine readable charge master which has been required for a longer period of time. So a few important notes that this rule is consumer focused. So the real audience intended was for consumers to get some information about expected expenses for their specific health care services at a hospital. But what it doesn't do is like talk about the cost sharing or the expected patient share of that expense. This is the full negotiated amount. So that's one thing that limits its usefulness if a consumer is trying to figure out exactly what they would be on the hook for. And given the variety of different formats and organizations and approaches to the information, I think that this highlights how complex this area is and that there's just a lot of variation in billing practices among our hospitals. An important thing for anyone seeking services is that the care that you think you're going for might not be what you ultimately get. One classic example is you think you're going for a colonoscopy to screen and they end up removing five polyps. So what you are expecting to spend is going to be a lot different than what the actual care that you needed was. And so that's just we want to make sure that whatever we do with this information we're very responsible and don't make any false promises because some of this is very difficult to predict ahead of time. And to that end, there may be programs or other need-based services that a patient might be eligible for that if they were to reach out and contact the hospital ahead of time, they may be able to get set up and avoid some of these insurance-based reimbursements and that might end up being more beneficial to the consumer's pocketbook. And so I think there's going to be a lot of stress in that it's always a good idea to contact a hospital proactively to talk about what might be on the table before seeking service. And as I already mentioned that these are the full negotiated amounts or allowed amounts and is not reflecting the patient's expected share. So without that kind of insurer side of the equation, that's very difficult if not impossible for a hospital to necessarily understand ahead of time again. So in thinking through the regulatory applications, again, I can't emphasize enough that this is just an incredibly complex set of information. So we're still reviewing and assessing the scope of what we're going to want to do. But as I alluded to earlier, there's some things that are going to make apples-apples comparisons challenging. So the guidance provided by CMS did allow some room for interpretation. So that is something that we would definitely want to make sure that we were careful in investigating before we tried to use it in any public-facing or regulatory context. Again, that variability and billing practices is something that can be hard to iron out even if you have kind of a code-level negotiated rate. The limitation to commercial rates, so the reason that's important to think about is there can be a relationship between hospitals reimbursements in other public programs and how that flows to their commercial negotiated rates. So I think that it would be as a regulator that it would be pretty important to consider the full breadth of the patients at a hospital seeing and try to incorporate that. And there's one major area in how the rule directed hospitals to account for non-employed physicians. That might be a real a real significant issue to consider if we're going to try to do any comparisons across hospitals, particularly tertiary care hospitals and the way they might handle admitting rights for non-employed physicians. So that's just some examples of the things that we're thinking through at our first blush. And I can't say enough that all of this is really based on a fee-for-service mindset and isn't really looking at it from the value-based approach that things like the all-payor model are intended to look at where maybe per-service prices is less of a consideration. Some other things to keep in mind. I asked a staff member Jeffrey to do literature search. Thank you very much, Jeff. And I did this years ago and I think there's a lot of fear that this might actually increase prices by adding some negotiating power for maybe hospitals and making that negotiation unbalanced. We didn't see evidence of that in the literature, but there is not a lot of research in this area, particularly for samples of any significant size. So this is going to be a very interesting experiment. But to date, you know, we found that very few consumers seem to adopt these tools, you know, ranging from two and a half to five percent. And because the negotiated rate is not closely associated with the consumer's expected share, there really isn't a lot of incentive to use them. With the counter example being those with high deductible health plans where, you know, maybe they do have more skin in the game and are more motivated to do this kind of shopping. We also know that, as I alluded to earlier, sometimes because of the way your benefits are structured, shopping is less of possibility, particularly if there's a, you know, HMO or some sort of limitation on your network. You might be limited in what you are able to access. And yeah, but we do think that according to the literature that these kind of actual negotiated rates or should say allowed amounts will probably be more likely to spur any changes if they're there than the charge master, which we know is not necessarily very closely associated with what the hospitals actually get reimbursed. And just, you know, down the pike, there is a related requirement that was adopted for most health plans and issuers to post machine readable files for their in network, out of network and prescription drug rates, which would apply for plan year 2022 onward. That's still early days and much like this requirement, I would not be surprised if there were some legal action taken in that area. So we'll certainly be keeping our eyes out for anything that develops on that front as well. So in the meantime, we will be working on gathering and synthesizing this complex data set. And this will be one of the things that in our enhanced data validation workgroup, we will be considering for its utility in trying to validate actual reimbursements that we see in our all claims payer database of vCures. So we're going to kind of try to kick the tires on this with providers directly as part of that process and figure out, you know, what information can be, this can be used for and how to do that in a responsible way and how that might differ depending on if the audience so how we might package it for a consumer, for instance, might be different than say a board member or yeah. And you know, I think the main thing is just really it's going to make sure we understand this fully so that we're using it appropriately. But I do think it's a, you know, it's a new data point. It's never happened before, so it's hard to predict exactly how it's going to go. But we are going to be following it and keeping you posted. So those are the slides I have today. Any questions I can address? Thank you, Sarah. That's very, very helpful. I'll start out with a couple. I do agree with you that I think that for the most part, price tools have not been effective. And it all gets back to the fact that what the patient looks at is what they would be paying. And in that, that's therein lies the real problem with healthcare, because you never get a true price equilibrium with supply and demand, because if someone else is paying the majority of the expense, all you really care about is convenience and quality. And so you end up usually going to your hometown providers. But I do think that we've seen tremendously poor uptake from the actual tools that, for example, Blue Cross Blue Shield has a tool for any of their members to access. And it just doesn't result in a lot of healthcare migration. But on the flip side of that, on the shoppable services line anyways, basically what the federal government did was 75 of the ones that they picked. And then the hospital had some leeway on the other 225, plus they had the ability to do the calculator instead of the shoppable services. But what I see there is there are a couple of scenarios where it could be quite useful and quite utilized. So for example, you mentioned a high deductible plan. It fits, say for example, the beginning of December. And all of a sudden you need to have something done. It probably doesn't matter anything other than trying to get the best price that you can get for that. And I think in that situation, if somebody all of a sudden needed to have a medical procedure done, they would be looking for the lowest price alternative. And the other situation that I point to is where employers could become major players in this if they created a system where the employee had some benefit. So say for example, your local hospital charges $1,000 more than one that's 45 minutes away for a procedure. Let's pick something, a colonoscopy. If the employer gave the employee a few hundred dollars as an incentive to go to the lowest cost alternative, they would still see a huge savings themselves in their self-insured plans. So I think that there is some usefulness. It's not going to be what everybody had hoped for. But do you see those types of utilitarian uses as well? Yeah, I think those are all totally valid use cases. And yeah, to your point, I think if there's a way we could say, even if you're not paying for it at the time of service in a deductible or a copay, that there's a reason premiums keep increasing. So trying to remind people that even if it's a little down the line that it does have an impact on their health insurance. And I think we've heard a lot through our work with the all-payer claims database that providing more meaningful tools for purchasers, such as employers, is really an important thing to focus on. And we're happy to help provide any information that might help in that domain. So on the larger posting that's supposed to be in a machine-readable format of all the services provided by a hospital, what is the definition of machine-readable? And is there an easy way to input that data from their website to some type of a program that would create a comparison tool? I know that as a board member, one of the more frustrating things is that a 4% increase from one hospital as compared to a 3% increase, it doesn't necessarily mean that that 4% increase makes them more expensive than the hospital that's only asking for the 3% increase. So is that something that is easily extrapolated from and put into some type of usable format? So I would say that machine-readable had a few... So it just means kind of like it's something you could put into some sort of program to read. So they specify a few different formats in which you can do that. However, there wasn't a consistent layout, which means that some of these files might have 300 rows or columns across, some maybe have 150 and there's not necessarily consistency. There's certain columns we know need to be there for every hospital, like the minimum and maximum for instance. But aside from that, there's just a lot of variation. And so figuring out a way to clean it up and make it as apples to apples as possible is not trivial and something that we'll be certainly working on and figuring out the best way to do that. And then of course, because hospitals offer different services, it might actually require a little clinical insight in where comparisons are appropriate and where they might not be. With just the 14 hospitals, would we be able to selectively go to each one and call out the fields and then put it into a comparable field? On our own database? Yeah, and I haven't spent enough time with every hospital file to answer that today, but that's the type of thing that we're exploring to figure and validating. I think that validation will be an important step. Super. Thank you, Sarah. Questions from the board? I have one. Well, I'm just, as this moves forward, it could get pretty complicated because this is kind of public data that's out there broadly in the public on a hospital basis, and it has the CMS label on it, which gives it some high credibility, I think. And I just wonder for the board, as we go forward, how we position ourselves to use the data for our work, but also to maybe be a referee on how the data can be used or misused. And I think Sarah's presentation clearly reflects that concern and complexity that it's a lot of data. I took a, I'm going to make Sarah nervous here a little bit. So, Sarah, I will admit from the beginning of this comment, this is not an analysis, but I took a look at the UVM Medical Centers and they had, it's a big spreadsheet, an Excel spreadsheet, and they have 14,237 rows of procedure codes. And of those, 13,290 have a Medicaid amount associated with that code. And of that, and I kept in the spreadsheet the minimum and the maximum just to kind of get a sense as to what percent of all the codes did Medicaid have the minimum allowed amount. And obviously, those amounts are kind of put on the Medical Center because Medicaid doesn't really negotiate with each hospital individually. But of those codes that had a Medicaid amount associated with them, 8,279 or 62 percent were the lowest and the spreadsheet has a minimum and maximum amount. So I don't know if that's good data or bad data. It certainly has a high probability of being misused. But I worry as we go through our regulatory processes that this brand new source of data that is very complex is going to be a source of information that we're going to have to tangle with one way or the other because I think people are going to use it. It's public data and dozens of different stories that it can tell. So that's something I'm thinking about is just how as an organization we get our arms around this in a fair and thoughtful way using the data for our purposes where we really scrub it hard and we feel that we can rely on it, but also working with our stakeholders elsewhere, whether it's the healthcare advocate or whoever to make sure that the data doesn't get used in a way that's misleading. So that's not a question. It's just an observation. Thank you, Tommy. Other board members? Yeah, I would just echo. I went on to a couple of hospital websites to look at what's out there and there's a lot of information. I mean, when you go in for a code, I agree with what Sarah was saying at the beginning. You may think you're going in for one thing, but it could be something different. If you go to a dermatology visit and then you could look at all the different things they could do and the different costs it would be to remove something by code, I think the average consumer wouldn't really know how to do that. But I also saw in some hospitals they are listing every single insurer and what they're being reimbursed by that insurer and other hospitals that maybe I didn't find the right places, but just kind of gave a range that said it went from X to Y. So it's harder to compare from that front. But just a comment on something that you brought up, Kevin, I mean, I think from a self-insured world, yes, there could be incentive to have people look around for pricing. But being on the corporate world side, I'd say there's a big but which is we're a pretty litigious society and should something happen because even though it may have nothing to do with that they went to a different hospital that costs less, but if something happened, you could imagine someone trying to come back at their employer because they recommended going to hospital X versus hospital Y potentially because of price. So I agree it's something from a self-insured world, you only pay what you spend and you want it to be less. I think employers tend to stick out of the recommending where people go because of HIPAA and other things. But that'll be interesting to see how that plays out and if employers go there. But it would be great to be able to benchmark against the hospitals for some of the procedures. But the other caution there is without looking at the entire universe, a hospital could be high in one area, low in another area. And I would caution jumping that we could jump to a wrong conclusion if we looked across one code and said, oh my gosh, this hospital is much more expensive than another without looking at all of the codes. There may certainly be outliers. There may be a hospital that's higher on every single one and one that's lower on every single one, but those would be things. I think as we look forward into this, how could we use it? It'll be interesting. But it's a new thing we'll be looking at starting in 21 for sure. Yeah, and just to piggyback on that, for a member of you, this certainly doesn't incorporate any quality directly in it, nor does sometimes the thing that's cheapest on a unit cost basis doesn't end up being the least expensive when you think about readmissions or things getting unintended complications. So without the quality or kind of the population health perspective, sometimes it can be short-sighted to kind of do just a procedure level comparison. Any other members of the board? Sure, I'll happen just a quick question for you, Sarah. Have you, and this is probably too soon to tell because I know you're just starting to dig into the data, but I'm curious about, I know the federal government mandated the first, was it 75 shoppable services, but then there was a lot of latitude for the other 225 or so hospital choice? And I'm just wondering if you've looked at the variation there in terms of is it all across the map, or was there sort of some consistency? And this is probably too soon to tell, or have you read about other people who have analyzed this? I'm curious about the 225 that were not mandated. Is there anything information that we're sending yet? Yeah, and I think what makes it challenging is that's kind of the set that could be shown to consumers in such a variety of ways. So I'm trying to figure out, I might have to do some web scraping to get at the data feeding the web tool, for instance, to even get at some of those comparisons. And then I think an important first step is trying to see if those tools and the information presented in the consumer-friendly way are actually jiving with the machine-readable entire manifest. And I think that's a really important place to start because it's much easier to work with a machine-readable file from a data perspective. But if it's not jiving with the consumer-facing one, that's a really important place to start about why that might not be true, which I could think of lots of reasons just given some of the latitude. But yeah, so I think that's kind of making it really difficult to get at that really basic question, unfortunately. Yeah, and I'm just thinking from an entrepreneurial standpoint, this is a treasure trove of data, and I'm just wondering if there's any third parties out there that are all over this in creating some kind of, you know... Yeah, I predict a lot of resales. They're going to do all the web scraping and they're going to pull this together. They're going to create tools for lots of states and consumers to use. I mean, I'm thinking there are entities out there that are going to be prepared to do this and have all the resources available to them. Is there anything that you've heard of of some third party that's doing this? I'm not aware of anything off the top of my head, but yeah, I fully expect a lot of resellers and data companies to be doing just that. Unfortunately, if it's garbage, then I don't know. I was more like an healthcare proctor institute or something like that. My last question was... So we've talked about consumers using it potentially, although it'll be complicated. We thought about how the board might be able to use it, caveats and all. And I'm wondering, is there any been conversation about how insurers are going to use this to do some comparisons of how they're doing in bargaining with hospitals? And just curious. Yeah, I haven't been privy to those conversations. So yeah, that's something else we can certainly monitor. Well, thanks. And I can't wait to see what we all do with this. This is interesting, although I do think to Tom's point earlier, a user's guide or some sort of... Here are the caveats before the data is misused or misinterpreted, I think is really important. And to the degree that the Green Mountain Care Board and the data team have some insights into that, I think it would be helpful to push that out there so that it isn't misused, because I do worry about that. Okay. Are there other questions from the board? Hearing none, I'm going to open it up for public comment. And I'm going to call on Eric Schulteis from the Healthcare Advocates Office first and Mort Wasserman, you're on deck. Thank you, Chair Mullen. I just... I mean, so I echo concerns expressed by the board and by Sarah. I think in response to the 70 mandated shoppable services of mandated 70 services that are required, it gets a bit tricky looking across the hospitals, just because we have a large number of small hospitals. So they are allowed, if they don't provide the service, to add additional services to get up to the minimum required disclosure of 300. So it's not like every hospital in Vermont is going to disclose these 70 services, because quite a few of them don't provide them all. I just wanted to hit on another topic, this idea that Sarah already hit on, kind of what a big task, working with the data is in addition to the different sources and whether it's a web tool that you have this website that you have to scrape or these machine readable files, it's not in a structure that is particularly useful for data analysis. It's pretty clear that different hospitals are structuring it different ways. So kind of restructuring the data and so it allows for comparisons across hospitals is a very non-trivial task and in my opinion, it's going to take a lot of work to do that. Kind of on that end, I think, you know, at least in my limited experience with an MC was that they were very interested in working with me and helping me understand, for instance, you know, was a no value for a specific procedure? What did that mean? Was it a zero? What does it mean? They're not allowed. They don't provide it for that carrier. So this seems to be a great opportunity to maybe work with VAAS and the individual hospitals and the carriers to kind of attempt to make cross system comparisons, the whole project of creating a database to do that more tractable. Because I think for any individual entities, just going to be too much work. And yeah. And then lastly, I mean, this data is going to be coming out every year. So as we're thinking about how to design this and set this up, I think we should think we're going to need to think a little bit past the immediate term and structure the data so that, you know, when this new data comes out every year, we're still able to implement processes to allow the analysis to be done. And I think it does require a little forethought to try to think about what might be done in future years with releases. Thank you. Thank you, Eric. We're going to move to Mort Wasserman and Mike Deltrecco is on deck. Mort. Hi. I have a question for Sarah and then a comment. What proportion of Vermonters have high deductible plans? The sorts of plans that in theory, price transparency is they're supposed to help those consumers. Oh, boy. You put me on the spot here, Dr. Washington. Sorry. Not that many. You know. Yeah. It precisely it's growing, but it's still not the majority. All right. No, it's nowhere in your majority. If I understand it correctly. So I was glad, Sarah, that you pointed out that this whole idea of shopping around for services flies in the face of a value based care all payer model as far as I understand it. And it really depends on economic theories, market theories that apply somewhere between poorly and not at all to individual consumers in terms of the way they make decisions about what to do in healthcare. So I would, although this is a lot of data and it's very tempting, it's also in typical fashion, not it is not structured uniformly. There'll be a tremendous amount of work involved to try to allow one to compare it from one hospital to another. So I don't I think if there's a limited amount of time for the data team to be doing stuff in 2021, I'm not sure this is where I would invest resources. Thanks. Thank you more. Mike Del Trucco. Good afternoon, everybody. Can you hear me and see me? Yes. Perfect. So first of all, I wanted to thank Sarah and the and the board for their thoughtful comments. And additionally, the public other public commenters. You know, this is a new data set. You've heard all of the challenges and from the get go, our members have really stressed call your hospital. And it's not only call your hospital about what code you're going to or what procedure or how do I estimate out of pocket. It's also about accessing cherry care policies. It's also accessing other programs that may be available, getting insured if you're not insured. So it's a whole host of things that we want to attach to aside from this transparency. So if there are any patients listening to this or they do listen and you use these tools, please reach out, look at your hospital's websites. It's pretty clear who you call and how you get help. So reach out and call. And to the to Eric and the other patient care advocates on the phone. If there are patients that have issues, challenges that you hear of, I am available to make sure you get the right connection to the CFO or whoever it might be at the hospital that there may be an issue. Thank you. Thank you, Mike. Is there other public comment hearing none? Thank you, Sarah. As more has said that it may be fraught with the inability of use, but I have tasked you and Patrick to take a look at this to see if there are possible constructive uses that make sense for the board. And so we look forward to at least conducting that analysis and not spending too much time, but the right amount of time to determine if there are uses that would be helpful and constructive. So thank you for your work today, and I thank you in advance for your work in the future on this. Sure. And we look forward to updating you next week about all the other stuff we're up to, so we can have a conversation in context. Perfect. All right. Thank you. Thanks. Is there any old business to come before the board? Is there any new business to come before the board? Hearing none, is there a motion to adjourn? So moved. Second. Is there a second? Second. Thank you. All those in favor of adjourning signify by saying aye. Aye. Aye. Those opposed signify by saying nay. Thank you, everyone. Have a great rest of the day.


Hi, I'm Carl Stutzman, Director of Library Services here at Anabaptist Menonite Biblical Seminary. And I'm Andeo Santoso. I'm a student in Master of Arts in Theology and Peace Studies. We just want to welcome you here to the AMBS Library. This is the premier Anabaptist Theological Library. It's also the first theological library that was built to the Green Building Standards of the U.S. Green Building Council achieving the LEED Gold Certification. We're proud that our environmental commitments are reflected in our space. Our main collection here consists of about 100,000 volumes in Biblical studies, theology, and ministry with an accent on Anabaptist perspectives. Over here we have a display of our newest items in the collection. These have been chosen by our faculty to be tailored to the needs of our students in their programs. If you'll follow me, we'll go up to the north wing and see some more of the library. Here in our north wing we have the main library collection. We have the variety of seating in this area, study tables. And up here we have our north reading area. This area has soft seating, exercise balls, a hammock, and bike desks. And we have views of our rain gardens and restored native prairie. So this is a really nice variety of places for you to read and enjoy. Our library takes very seriously the needs of students at a distance. We scan from our print collections. We also have extensive digital collections that are available from our website. In fact, the number of books and journals that we have available online has exceeded the numbers that we have available here in the building. I'm going to take you on back to show you some more features of the library. Here we have one of our group study rooms. This is available to be reserved by students. Students are able to reserve these carols for the block of the semester, enabling them to read, write, and research here in the library in their own space. Back here we have some special collections available from our Bible collection and more views outside. I'm just going to take you over to Antio's who's going to complete the tour. And here is the center of the library where you can find many Bible commentaries. And here is the place where you can find help if you need any assistance. There is also library offices for writing service with our talented and professional full-timers. We are already here to serve you. And I'm going to take you to the bookstore. This is a small bookstore. There is some snacks and apparel. And also there's cards. And this is some of the new books published by Anabapis and Mennonite publishers. And here is the library computers where you can find a spot to work. If you don't have your own computer, you can use these library computers. And here is the collections of the periodicals from many different places. And I'm going to show you the group study room. This is the place where you can maximize your time with your friends to study together. And here is the collections of video, also audio. And now I'm going to show you the special collections. This is the special and rare collections where you can find many old bibles and many other books from various years and then also from around the world. So this is our library. If you have any questions with AMBS library, you can go to ambs.idu.slashlibrary. And if you're interested to be a student here, we will come you to go to ambs.edu. And we hope to see you soon. Thank you.


What's going on everyone Sam here with late in sports cards ripping two boxes 22 top scrims sonic baseball late for William A. It's good luck William. Let's do it buddy some other one It's like I just remembered that that for some reason spark this family guy. What do you say though?  He's like, I'm just kidding.  Uh cheapest football product trying to think probably That we have yeah Yeah, thonny's gonna check for you burger. It's probably gotta be like for some jake mires Leaf products. Yeah, I'd imagine it's a leaf for sure like pro set Yeah, probably Ray wave hobby bias for the Tigers I'm gonna drop a link for the football In chat, but the cheapest one we have is a leaf ultimate giraffes. Was that like 90? 80 for do go refractor. There's a Royce Lewis youth quake for the uh, rookie for the twins Yeah, I mean the next one's like 130 for a pro set metal pro set metal. Gotcha The cheapest license when we have is the panini prison t-ball But even that's kind of Ryan Anderson Not really Yeah Ray wave JD Martinez That one's just a gamble personally probably score Don Russ would be your best bet Uh, George springer refractor. Yeah, I'd agree with that probably just score you at least get what was it for us? Yeah Ton of numbered stuff You only get like one auto of men, but you're also getting like a lot of rated rupees Lube up that is two 199 on the teal ray with Yeah, you get a lot of rookies and you get a lot of inserts Yeah, a lot of like those short prints to Yeah, they have a lot of Ozzy albis and Adolis Garcia Yeah, it's not bad Two autos one line getting a box elite. Yeah, because I don't think I've ever opened an elite box before refractor is matt olson Jt real me so For an elite box. Yeah, I think bad actually. I mean there were at 350. Yeah, it's not bad And base. All right box number two here. Uh, William. Good luck buddy. Maybe someday My we're gonna get married Wow Yeah, Don Russ you get what at least like four numbered cards a box to three four numbered cards I'd say Maybe not as that many but I'm trying to think besides the auto one man My open mind. I got the one sauce that was numbered and then An insert that was numbered Like you got any of the jersey numbers of the purples of the Like press proofs to 100 or to 75 on the die coast. I did get a press proof Rogers, I think. Oh, that's cool. Drew. I'll let's refract a rookie. Nice one there So I I think I probably like five or six numbered cards. Yeah, I would say yeah Including auto jersey That sounds about right Lou bob on the ray wave Burger in my honest opinion. I I want to do it There's vladimir goro jr. You tweak That's just me rare refractor rookie I would think the majority of people would say if it's not licensed, it's probably Yeah, depends on who you're I would say I would say the only exception would be like baseball and tee Max Kepler refractor or something like that Uh, jeff mcneil prism and ray wave read deptmer's rookie But I could be wrong. I don't know. Yeah Just my opinion There's karsten kelly on the purple yellow sonic pulse the 299 for the diamond bags, uh brandy built Yeah, baseball, uh, nts. Awesome. Yeah, it's so good. The patches and everything they have just so fun to look at There's freddy freeman. Matt Chapman. Yeah, tender's optic 2020 is nice That's obviously burrow and herbert, right Uh max freed refractor. We got green ray wave francisco lindor to 99 21 of 99 nice one there for the mess youth quake spencer torkelson rookie ray wave andre jackson rookie for the dodgers and last pack here william. Good luck buddy Face that'll do it for your break. Congrats again on all the hits there william. Thanks for the support Get them right out to you, buddy


Hello and welcome to the session in which we will discuss organizational expenditure and startup costs. Two different topics, very similar, but we're going to break them into two separate components. So what about organizational expenditure? What did we learn about this topic before? Well, organizational expenditure we learned about this topic in-gap. And what is organizational expenditure? It's when the company is forming. When the company is forming, there are legal costs, there are consulting costs, there are accounting costs. And as a result, those costs we need to be accounted for. Hello and welcome to the session in which we will discuss organizational expenditure and startup costs. Well, we did learn about organizational expenditure for gap. And what is the idea of organizational expenditure? Well, the overall idea is the same. Before a company is formed legally, before a corporation is incorporated, well, you are going to incur certain costs, legal costs, consulting costs, costs to put the legal paper together for the company to start. Now, from a gap perspective, from a financial accounting perspective, what do we do with those costs? Well, we assume those costs will have no future economic benefits. From a gap perspective, we will expense them, whatever we do, we will expense them, and we credit cash. And we're going to have organizational costs for an organizational expense debit, and we credit cash. Now, we need to learn from a tax perspective, what are our options? And since gap will be treated differently than IRS, then we're going to have a difference which we'll have to account for on schedule M1. Notice in these sessions, every time there's a difference between gap versus IRS or gap versus tax, I always mention eventually we're going to have to reconcile all these items on schedule M1. So let's go back and dive into organizational expenses. What did they include? Well, as I mentioned, legal services connected to the establishment of the organization. An example of it will be drafting the corporate charters and bylaws. Well, you may need a lawyer or hire a firm to do that for you. Essential accounting services before the organization comes to life. Cost associated with temporary directors and meetings held for the organizational purposes by directors and shareholders, you're going to be incurring cost in those meetings. Well, also that's organizational cost. Fees paid to the state where the corporation is incorporated, you're going to have to pay a fee. It's not that much, $300 or $600 for the state of Delaware or whatever state you are, you are incorporating in. Well, that's also considered organizational cost. So those are typical example of organizational cost. We're going to see what startup cost later. Before we proceed any further, I have a public announcement about my company, farhatlectures.com. Farhat accounting lectures is a supplemental educational tool that's going to help you with your CPA exam preparation, as well as your accounting courses. My CPA material is aligned with your CPA review course such as Becker, Roger, Wiley, Gleam, Myles. My accounting courses are aligned with your accounting courses, broken down by chapter and topics. My resources consist of lectures, multiple choice questions, true-false questions, as well as exercises. Go ahead, start your free trial today. So when the corporation initiates its operation, it means when they start their business, they have the option, notice, it's an election to amortize organizational expense over a period of 150 months commencing from the month the business begins. So simply put, from the time you start your operation, you're going to be incurring some cost. What you do once you start your business, you are going to amortize this expense. What is amortization? It means treated as an asset, treated as an asset, then expense it, amortize it as an intangible asset over 180 months. Well, is that all? No, there's a special provision that allowed the immediate expensing of the first $5,000 of these expenses. So guess what? Well, they will tell you the general rule is treat those as an asset for tax purposes. You treat them as an asset and you amortize them. That's the general rule. Now for GAAP, you'd say expense them, right? And I hope this makes sense because IRS, IRS here, they're basically, they don't want you to take the expense because if you take the expense, you're going to lower your taxable income. Does that make sense? That's why they don't want you to do that. However, there is a special provision where you can expense immediately $5,000. However, not everyone can expense this $5,000. You could expense that $5,000 and you will start to lose that $5,000 of immediate expenses, dollar to dollar. Once your organizational expense exceeds $50,000.  If you have $48,000 of organizational expense, you'll get the $5,000 immediate expensing, then you can amortize the rest. If you have $51,000 of organizational expense, guess what? Of that $5,000, now you lost $1,000 of immediate expensing. What's left is $4,000. So you can immediately expense $4,000. If you have organizational expense of $55,000, well, if that's the case, you lost $5,000. You lost the whole $5,000 that you can use because you're already at $5,000 above $50,000. So the limit starts at $50,000. The limit starts at $50,000. So that's what you need to know. Don't worry. We're going to work examples illustrating this concept, but that's basically the special provision. So they do allow you to expense $5,000 as long as you don't have too much of organizational expense. What's too much? Too much starts at $50,000 and too much, way, way too much, $55,000. You no longer, you can no longer take a penny of immediate expense and you can still amortize. Now, we have to know that expenditure relating to issuance and sales of shares or other securities. Now, notice the difference here. Those are costs related to the stock itself, to issue the stock, as well as the transfer of asset to a corporation. Any costs related to that are not eligible for the qualification. These, in quote, expenses because they're not expenses, diminishes the amount of capital raised and are not deductible in any way. Hold on a second. If they're not deductible, if they're not expenses, if they're not asset, if I cannot amortize them, what do I do with those expenses? Expenses such as those, what we do is we debit, paid in capital and we credit whatever we do, credit cash. So what they do is they reduce your paid in capital account, your equity account. So they're not expense, kind of they reduce your equity. Let's illustrate the organizational expenditure. Let's assume Farhad Corporation, the C Corporation operating on a calendar year basis, and we're going to see why it's important to mention the year commenced operation on September 1st of the present year and accumulated 53,000 in organizational expense. What's special about 53,000? Well, 53,000 is 3,000 above too much organizational expense. Remember, you could immediately expense five. Guess what? I already lost of that five, 3,000. All what I can do is if I want to, if I want to maximize my expenses, I can expense two. And let's assume Farhad's objective is to, is to maximize the expenses and amortize anything that's left. Well, here's what's going to happen. If I can do that, if I want to do that, I can only take immediate expensing of 2,000 because I am 3,000 above 50,000. Well, 3,000 above 50,000, that additional 3,000 above 50, made me lose 3,000 of my immediate expense. So I'm left with two. So if I'm left with two, I have 51,000 left. The remaining 51,000, I can amortize it over 180 month, whatever that amount is, if I take 51,000 divided by 180 month, my monthly amortization is $283 and I'm have four month for that year, which is September, October, November, and December. And that's going to give me, if my math is right, $1,133 in some change. So my total expensing for that year is $3,133, the immediate $2,000 of expenses plus the amount that I amortize for the remaining. Let's take a look at another example. Farhad Corporation was formed and started, started business October 1st. Remember, you can start the amortization only when you start the business. Farhad incurred the following expenditure from October to December. Fees paid to the state in order to incorporate $2,000, cost to print and sell stock certificate $15,000, quite an expensive deal, accrued legal services for drafting the corporate charter $21,000, notice those are accrued not paid, expenditure incurred for temporary director $23,000. So the first thing is here I have it accrued to mention it's not cash. Well, if it's not cash, can I still, if I'm a cash basis taxpayer and I accrue those expenses, can I still count them? And the answer is yes, whether you paid them in cash or accrued, you could still expense them. Now, however, how many of these expenses are considered organizational expense is fee is fee paid to the state and organizational expense in order to incorporate. Yes, cost to print and sell stock certificate. No, remember what I told you about that money? If you incur any money related to the issuance of stock, which is cost to print and sell the stock certificate, you reduce your paid in capital. Legal services, sure they are whether accrued or paid in cash, that's organizational expense and expenditure incurred for temporary directors before the organizational start $23,000. That's also organizational expense. If we add them all up, they will add up to $46,000. Guess what? We are in good shape. In what sense? If we want to maximize our expenses, you don't have to. But if you want to, you could immediately expense $5,000. What's left is $41,000. $41,000, which is $46,000 because I immediately expense $5,000 out of the 46. What's left is 41. And if I take this 41 and I divided by 180, I'm going to be able to expense $227.77 per month. And I have three months from October, October, November and December. If I multiply this by three, I'm going to get $683. Therefore, if I want to maximize my expense, it's going to be $5,683 for that year, which is 2024. Let's assume the legal expenses, the legal accrued expenses were $30,000. So what I'm trying to say is, let's instead of $20,000, I incur $30,000. Well, let's see. If I take $2,000 plus, let's make it $29,000. That's rather than $30,000. Let's make it $29,000 plus $29,000 plus $23,000. Now my accrued expenses, let's make sure the math is right, $231,000, $54,000. That's correct, $54,000. What happens now? Well, if my organizational expense were $54,000, now I can only immediately expense $1,000. So if I'm going to have $54,000, I'm only going to expense $1,000 immediately. What's left is $53,000. I divide this by $180,000. Whatever that monthly cost is multiplied by $3,000. It will be $1,000 plus this answer. What if my accrued legal expenses were $35,000 now? Well, if my accrued legal expenses were $35,000, let me add them up.  So $35,000 plus $23,000 plus $23,000, that's $58,000. Just those two will put me over $55,000 plus the two is $60,000. So what happens is I'm going to take all the amounts, the $2,000 plus the $35,000 plus the $23,000 and amortize them. That's all what I can do and deduct $3,000 in 2024. Now, organizational planning, organizational expenditure, I would involve some proper tax planning. Why? Because in order to be eligible for the election to take that immediate expensing, the expenditure must be accrued prior to the conclusion of the tax year in which the corporation starts its operation. Simply put, it's the year when you start your operation. So the corporation choosing accounting method, as I mentioned, is irrelevant, whether it's cash or accrual. Even if it's a cash basis, as we saw earlier, and you incur the expense, but you don't pay it until the subsequent year, it's still qualified as just we saw earlier when it comes to the accrued legal services. However, it's important to note that this rule could present a potential pitfall for corporation formed later in their initial taxable year. What does that mean? It means sometime what's going to happen is this. You might be incurring organizational cost. Let's assume this is year one and this is year two. And the organizational cost, some of it may fall in year one, some of it may fall in year two. Now, what happens is your organizational cost is split. So if you do proper planning, you might be able maybe it's much better to combine those two, or maybe keep them split or move the expense from year one to year two or move the expense from year two to year one. Okay, so let's take a look at an example to see how this work if that happens. Let's assume Farhat Corporation is established December 29 at a incur qualified organizational expense as follows 60,000 in year 2024 and year one and 25,000 in the following year. So notice in the two years, 60 plus 25, they incur a cost of 85,000. Now assuming Farhat follows the calendar year for tax purposes, what would happen? Well, it can only deduct 60,000 of the organizational expenditure over 180 month period. Why? Because you can only do that because the 60,000 fell in the first year. Whatever you incur in the next year, you already started your operation. Okay, so you are established and incurred those expenses in the year that you started so that 25,000 cannot be combined. So tax planning, what would you do? The first option if you're into tax planning is to ensure all the expenditure incurred in December. So try to make those 25,000 spend them in December of 2024. So the entire amount now can be amortized over 180 month period. The second option is to do what? The second option since the remaining 25,000 incur in January, what I would do, I will start my year. I will rather than choose a calendar year starting January 1. I will start a physical year after January 31. And this way I will be able to include those because now they are considered in the same year. And this way I can amortize the full amount of 85,000. What about startup expenditure? Because this lesson is about organizational expenditure as well as startup expenditure. How do we treat startup expenditure when it comes to taxes? Well, guess what? They are similar in form of tax, in terms of tax handling. However, they should not be confused with organizational expenditure. Startup expenditure include a range of investigation costs associated with entering a new business. So when you enter a new business, you're going to study the market. You're going to have to travel. You're going to have to look at financial statements. You're going to have to audit financial statement. Maybe if you're buying a company, you're going to incur legal fees. But that's in the process of investigating the business. All these costs are called startup costs, which are different than organizational expenditure where you already know you have the corporation and you are basically putting the business together from a legal perspective. Here you're investigating whether the business, whether you should go into this business or not. They also include operating costs like rent, payroll that are incurred by the corporation before generating any gross profit. Sometimes you might need employees to help you in this process. You might have to, for example, if you're traveling to South America to start the business, you may incur an office rent. All this process is to investigate whether you want to go into this business or not. Now, generally speaking, startup expenditure are required to be capitalized. Simply put, they are treated the same as organizational expenditure. Taxpayers have the option to elect to deduct startup expenditure in the same manner as organizational expenditure. Under this election, again, 5,000 of startup costs can be immediately expensed. Again, subject to phase out limit. Any remaining will be amortized over 180 months. Again, the rules governing the election and waiver of the election for organizational expenditure applies also to startup expenditure. So now it's a different type of expenditure, but the same rules apply. What should you do now? Well, organizational expenditure, startup expenditure, why are they important? Because the way they are treated for GAP is different the way they are treated for IRS. That's important and we need to know the rules. Go to far hat lectures, look at additional lectures, MCQs, true, false, additional notes. That's going to help you because this is basically we're setting the ground, we're setting the seed to work on schedule M1. Good luck, study hard, and of course, stay safe.


decisions. And so we want to highlight some of this. So we have two abstracts, just two. Unfortunately, our third presenter is not able to join us. So you guys have a little bit more time, which is good. So we have two presenters. We first have Stray from South Africa. Tony will be presenting from his South Africa on how we're using a super set to analyze and do some advanced analytics with DHS2. And then after that, we have a John Panger from CDC PMI. He'll be coming up and we're sending on how he's using some art to do some very advanced analytics as well. I'm going to glue some interesting insights from the malaria data he's working with. So I think with that, Tony, are you ready? Okay, I'll hand it over. Thank you. Can you guys hear me? Great. Good afternoon, everyone. And thank you for joining the session. I'm Tanya Governor. I'm the project's portfolio manager from his South Africa. And my colleague Comfort Manga, who is probably still having lunch as usual, will join us soon, but he will cover any technical aspects or any technical queries that you may have following the presentation. Right. So for today's presentation, I will be taking you through the use case. I will introduce to you the super set solution. We will talk a little bit about the data analysis from the multiple data sources. I will introduce the DHS super set portal app. And I will take you through the next steps that we envision for the future. Before I continue, I would also like to get to know you a little bit better. So by the show of hands, how many of you are developers? Great. Cool. And project managers, program managers, product implementers? Awesome. And how about people that actually is utilizing the data? So data analysts here? Awesome. Well, that's really nice. I see some of you have your hand up for all three, which is super cool. So welcome to the session. I hope that you will find it quite interesting. So for the use case, while we were implementing a project in South Africa, we were actually faced with a challenge regarding reporting limitations. We knew what we wanted, right? We wanted to have reporting from multiple data sources with different data models. We needed to expand on the visualization capabilities, and we knew that we needed a business intelligence tool to do that. We wanted to analyze existing DHIS2 data together with non-DHIS2 data. We were quite aware of the licensing costs and the cost of ownership. We also still wanted to utilize the existing DHIS2 systems, but we needed a tool that would complement that technology. So what did we do? And you're also aware that we collect all sorts of different data from different sources, and the need to actually bring that together, analyze and present that from a central point without having to reinvent the wheel actually presented this interesting opportunity to the team. And of course, all of this needed to be done in the most cost effective way, and the tools needed to be super user-friendly. And also what we were considering and prioritizing was the cost of ownership to the client to make sure that that cost of ownership is as low as possible to promote the sustainability and the institutionalization of the product. So what did we do? We pushed on. We were looking for a solution and we knew we were going to find it. Our vision was clear of combining data from multiple data sources. We knew that business intelligence tools would provide the technology to combine multiple data sources and enable a single point of advanced analytics. And we were aware that proprietary business intelligence tools does carry quite a heavy licensing fees. So we had to make it cost effective and sustainable for implementation. With all of this information in mind, here's South Africa then embarked on a process of setting up the Apache SuperSet as a business intelligence tool. So why Apache SuperSet? And apologies for the headings of my slides. I think it doesn't recognize this point. So why is Apache SuperSet? One, it's free and open source. So we didn't have to be too concerned about the licensing fees. Two, it has an awesome interactive filtering capability so the user could actually interact with the data. It has wide SQL database support. So this enables us to handle and integrate multiple source configurations from different sources. It has its own user and role management component which then allows us to manage the access that the user has, the users have to different reporting with the aim of replicating the user management system from the user management app in DHIS2 and align that with the SuperSet software. The reports are easy to share and that they can be downloadable and they can be shared on the multiple communication platforms. And then lastly it does have a community of practice to support us during this implementation. So how did we implement this? So so far HIST South Africa is utilizing the SuperSet in two projects. Firstly it is the South African human resource sorry the South African health management information system where we have set up a SuperSet dashboard for the Minister of Health. The second project is the human resources information system where we are combining human resources information with routine health data. Now there are several ways in which SuperSet can actually connect with the source data but for the purpose of this use case I will be talking more towards the human resources implementation. So the Apache drill is an open source database connection tool that we use to use that we use as a connector between SuperSet and the source data. A Apache SuperSet can still speak to your DHIS2 APIs and other systems such as Citus data, Fire, it also integrates with Python and Streamlet which we are using for machine learning in the country. This diagram actually shows how SuperSet is pulling data from DHIS2 systems and other source data through the Apache drill connector. So all of the data that you are looking for is actually pulled into the SuperSet platform. You set up your dashboards in SuperSet and what is really cool and what we're so excited to share with you today is actually the DHIS2 SuperSet portal app. This app allows you to communicate and to interact with the dashboards that have been set up in the SuperSet. So now that we've implemented what does the solution now allow us to do? What are the benefits that we are seeing and how are we utilizing it? So firstly it does allow us to analyze data from different source data in a single dashboard. It provides more visualization capabilities that is currently not possible in the DHIS2. It is fast, it is lightweight, it is intuitive, it has loads of analysis and visualization capabilities. It is easy for the users to utilize and it has capabilities from simple pie charts or graphs to your detailed geospatial graphs as well. Overall we found that there's not really a need to pull all sorts of data into one huge warehouse but rather have the analytics that can be displayed through the DHIS dashboards using the linkage of SuperSet connecting to your source data. This is the SuperSet dashboard and why we would recommend for you to use it. It expands on the visualization options of DHIS2. It allows the user to actually interact with the SuperSet dashboards that have been created through the DHIS platform. You only need a super user to be able to set up the dashboards in SuperSet and then share those dashboards with the user through the user management roles and groups in DHIS. It allows for advanced SuperSet reporting capabilities from the DHIS2 interface and the generic SuperSet functionality makes it available to any DHIS2 implementation. So what are the next steps that we envision for the future? DHIS2 has a user management app and SuperSet has a module for user management as well. So we are currently looking at aligning the user access in the DHIS2 and SuperSet for the seamless integration of user management. Secondly, the SuperSet app is generic and can be plugged into a SuperSet installation with the documentation available for the implementation of the Apache drill. The portal can benefit lots of DHIS communities and the app will be submitted to the DHIS2 app hub for your consumption. We would also like to expand the SuperSet app to different projects that we are implementing in South Africa. The successful implementation of SuperSet will be expanded to other projects and bring business intelligence and DHIS combining business analytics, data mining, data visualizations, best practices and more data-driven decision making, creating better lives for all. Obviously, this work would not be possible without our funders, our clients and all the contributors and the HISP teams. So we're really grateful for that and we thank all of the contributors. Thank you so much. A few minutes for questions? Yep. I'll invite my colleague, Confit. Yep, please. I don't mind sharing the center stage. Let me hand this over to you, Confit. So if someone asks a question, just repeat the question for everybody online. Okay. Thank you. You know, one of the really cool things about this is everybody is always saying, we can't make this chart in DHIS2 because you want something like Fox and Whisker, Winrose, like these really advanced charts. But now, look, you can. You just got to go through, use the app. Subset. Okay. Any questions yet, Nora? One comment. When did we last have a woman developer standing up there? Thank you. Thank you, Nora. Sorry, you have to say why we were applauded for everyone online. I've just got a wonderful applause for being a woman representing software development teams. Thank you. Any other questions? Are you excited about the app? I see a question at the top. Okay. The question is, I mentioned that we're using Python for machine learning and how does that fit into the presentation. So I was saying that the superset has capabilities to connect with DHIS2 API, as well as other systems. So other systems, including FHIR, other systems, including the Citus data, also it can connect with applications such as Python and Streamlet, which we are using for machine learning. Is that correct? Yes. In addition, superset was developed in Python. So for configuration and other things, I mean, you need a bit of Python to do that. The question is, why did we use Streamlet for the visualizations? Oh, why did we not use Streamlet, but rather use superset? Yeah. I mean, when we started, like, with this work, we looked at Power BI, but we had an issue, I mean, with licensing, right? And then even publishing the dashboards. And then so we wanted Sanya's, like one of the slides, I mean, she was explaining the reasons why we went for supersets like costs and all of those things, right? And we're looking for BI2. So we found superset, and then we tested it out. And yeah, I think Streamlet is not really, I have a little knowledge of it, but I don't think it's a BI2, right? You can do like visualizations and things for your machine learning stuff, but it's not really a BI2. So you need a BI2 to expand on what you already had on DHS2. And for the internal use case, we needed data from multiple sources and just allowing us to, superset was allowing us to make those connections and also to make the same visualizations available on DHS2. So yeah. There was a question online, when will your app be on the app? Okay, we are aiming to have it on the app app, by the before end of the year, because we've just sort of put everything together now and it's coming together very well. So hopefully by the end of the year, we'll have it in the app app. Yeah. If not sooner. Thank you. There was another question. Oh, okay. Go ahead. An example of, oh, don't have a online question. Yeah, sure. Amri, you can take that. The question is, can we show an example or speak to an example of the superset dashboard? Yeah, but please join the universal health coverage session tomorrow to actually see that integration. Is the question is, did superset, did the platform itself have sufficient visualization options or did we need to further customize? Okay. Okay. In terms of visualizations, we didn't have to add in new visualizations. It's more on like pulling the right data and then we have a wide variety of visualizations that we could pick from. So we didn't have to customize or add anything new. Just more on your data sources, making sure that you have all the data you need and then just like in DHS, then you can pick the right visualizations for your data. Yeah. At this time, the visualizations are quite extensive. We haven't come across a situation where we needed to customize as yet. There is quite extensive capabilities. Question? Go ahead, ma'am. Yeah. So with superset, it's low cost and open storage is great, but you mentioned that we need some coding experience. Are there any opportunities that people can take into account when they're comparing the other folks? Yeah. I mean, for us, it was more like, for the ministers of health or for the clients that we work with, right? Not to be restricted by, like I said, we started with Power BI and then we had challenges with licenses. So we needed something that we could implement for them without restrictions on licenses and all those things. So yeah, and supersets came in handy for us. But in terms of programming, it was just for the app to make sure that the visualizations are also available on DHS too. So we had to build the app and also be able to interact with the superset API. But once it's set up, I mean, like for example, the app that we built, once it's set up, I mean, it's generic, then you'll be able to pull like anything from superset. So you need any programming. So yes, for configuration, configuring superset and allowing permissions and all of those things, you will need your system admins to assist you with that. But once that is done, then you just use it. And yeah, and then you'll manage permissions. I think that's the only thing that additional thing that you would need to do. Okay, there's another question from the gentleman. Yep. Okay, so are you saying that as we start pulling lots and lots of data into superset for analysis, is it able to still function well with that load of information? Okay, okay. Yeah. Okay, the first one I think it's around superset is not actually holding all the data, but it's actually pulling when needed. Yes, for example, we have a centralized data analysis of the data of the city and we have 64 data which is 64. We need only 5% frequency in the centralized database for work, but 80 to 90% is located in local database. So we need to hope that when data is based on vertically or horizontally and it will be distributed in different locations. So it will be very abstract. Yeah. Okay, now I think I understand. In one of the slides that Tanya presented, so in terms of connections, I mean, you can go via the connector, right? Like, for example, if you want to map the DHS-3 API, but you can also add the different connections to your other databases and then you'll be able to query them within the databases. So, yeah, it can do that as well. Not so if like that answers your question, but maybe you can look at your particular use case and then discuss. Yeah. We can definitely have a side discussion to further unpack. I'm sorry, I'm going to have to... Okay. Oh, okay. That's it. Okay, thank you. Thanks. Last question. We do have a tea break and lunch and we're all here together. Okay, if it's fast. It's fast. You have one minute. First of all, I really want to appreciate this expression of which I use to... And I really like the idea that you are using the open sources to go down the path of... So this is really appreciable. So actually, we have a product which is similar to that, but that product has a fee and a Dica, which is running on different platforms. So my question is, are you using the same database and same API, running on the same DHS2 platform, or is it running on different platforms and different databases? Okay, we are connected differently, right? Like for DHS2, then we'll never use our account, right? That's created on DHS2. And then we map the DHS2 API to superset, right? So basically in terms of the queries, superset queries DHS2 via the connector, right? Using the API. Yes, but it's through the API. Like, you know, how the query happens. So, yeah, like for example, say if you want to query organizational units, like for example, right? That happens through the connector. And then you're able to put... Yeah, yeah, yes. Yes, we use that. Then when we, for the portal app, so for the portal app to pull the visualizations in, then you interact with the superset API. Yeah, okay. Thank you very much. Okay, of course. Yep. That works. If anybody hiding in the back wants to actually sit on the front row and get a good view, that's a great time to come down. Calla, no, you've never hidden from anything in your life, so it's okay. There's a couple of slides where it's a little hard to see. So, yeah, you really would benefit from being down front. Hi, good afternoon, everybody. I'm John Painter. I'm from the Center for Disease Control in the Malaria Branch. I work with Presence, Malaria Initiative, largely in the Sub-Saharan African countries, where we use DHIS-2 to monitor our malaria programs. Reef outline, I'm going to give a moment of appreciation for DHIS-2 before I go on to tell you why DHIS-2 is not enough. This is particularly for the analysts in the group. But the other folks, I hope you'll appreciate it as well. And in order to provide what is at least maybe enough, we made a web-based app to help throw in some of the gaps that we see. Now, I won't be going through a live demo because at least one day-wise, programmers told me that that's a bad idea. So I've got some screenshots to show you as I talk through why we do what we do. So, a moment of appreciation. We have the most far-to-reach health facilities all over the world. And DHIS-2 has been really impressive. It's transformed our surveillance from very unmanageable paper to analyzable digital form. It's been a game changer. Now, so it's really easy to access health facility data. But does it tell the whole story? No, that's the problem. And to go from data to really good action, we'll need more than just the raw data that's in DHIS-2. And I hope to show you some examples of why that's true. Ideally, when we think about an analytic plan, it should involve documenting what data we're going to use, controlling for reporting bias. We'll move them outliers and then doing some statistical models to get the best idea of whether there is change, whether that change is real or just some random variation. So we've gone through a lot of code to do this, but I don't want to make it sort of more available to most users. We created an external app that has been dubbed by one of our partner's magic classes. Data is an intervention-effective risk and trend analysis tool. It starts, of course, with a DHIS-2 instance and then uses the API to connect to an instance of R that we run with RStudio. That instance then hosts the app that makes it visible on a browser through a Shiny application. We use the API to access the metadata, to access the actual data, do the data cleaning and munging, various visualizations, and then, honestly, the advanced analytics, the time series models. So here's a quick screenshot of the app. It is set up with some tabs across the top so you can go from setup to analysis and six pages. It starts by putting in your credentials for the instance you want to get to and a place where you want to store the data, all the data is stored locally. So the first issues that we wanted to solve was finding and documenting the data that's in the DHIS-2. We found a couple of countries we work with. Metadata is unavailable to most of the DHIS-2 end users. Now, if you go to the DHIS-2 demo, you see all the things in admin would say it's great. But when you're ready to end users, you don't see all that. Only three of the other countries that I reviewed had a metadata app that was available where they could see some of the elements. The funny thing is, it's all actually available through the API. The problem is, it comes in in a format for most people, it's not readable, but we do the business of translating it into a readable form. Though part of the reason for this is that we frequently find that different numbers are published for things like confirmed cases of malaria. If you go to the World Malaria Report, per country you see one number. If you go to the country's annual report, you see one number. If you go to PMI's report, you end up with a different number because each time a different analyst is probe a number. And naively, girls are like, oh, well, I want to see what are your number of confirmed cases. But the DHIS-2 instance doesn't have available called confirmed cases. What it has is a case positive for malaria RDT, where it's a case that's positive for bionicroscopy, where it has outpatient cases, inpatient cases, community health worker cases. And we don't know which ones were pulled. So it's very hard often to compare where the sets are. We want to have data that's reproducible. So this first step is being able to expose to everyone what is available. For most of our countries, there is many as 10,000 data elements. We're not all malaria data elements, but there's hundreds of them. And if you've ever tried to go like the visualizer app or the pivot table app, you've got to box this size to try and find your data element. So this is exposed to everyone. You can search by data elements, by knowing the data set, the period type, et cetera, so that we don't confuse monthly data elements with weekly data elements. And then we expose the indicators as well, importantly, showing what is in the denominator and the numerator. So it's spelled out so that we can see. One country I went to, they were really aggravated that their indicator for confirmed cases did not match what they would have pulled by themselves. And they had no idea why, whereas they came to say, we can't trust the HIS-2, we should go back to our old system. We looked it up here, and we found that when the HMIS folks built the indicator, they'd accidentally written in one of the data elements twice. So therefore, all the numbers didn't quite add up, but they had no way of seeing that themselves, just with their access to the HIS-2. After browsing the all the data elements, we're asked people to make a data dictionary, define exactly what you want to include, then you can say, I want to download this data for a set time period. You get a progress bar, the API is called, and after a bit of time, you have all of that data. And once it's downloaded, you actually don't need any more internet connectivity, which is a great help for some folks. So at that point, you can work with the data offline. All right, so now we have the data, let's analyze it. We're going to check for completeness and the potential for reporting bias, and we're going to check for outliers. So completeness and outliers, we all know that when you go from a register, like you see on the left, over to the digit number, the errors are going to happen. I mean, that's not as well, it should happen if you're doing that much of it. And a huge investment is made in making sure that facilities do this correctly. That's their data quality checks. But we found that even when the data quality is high, there's actually potentially a much greater problem when we don't know how many facilities provided the data. And I'll give you an example of what I mean. So that involves something we call reporting bias. It's potentially a huge problem with aggregated data and if there's one thing you take away from this talk, it's this. So an example, this is one of our countries where there's high levels of malaria. So I selected a county from August 2021 and I pulled out four data elements that I thought would be the typical thing someone would want to review when looking at data from this county. What's the first attendance at the outpatient clinics? Total number of suspected malaria cases. The total number that were tested for malaria and numbers that were confirmed with malaria. And now typically what we do is we'll take those first two values and say, okay, what percentage of patients coming to the facility were suspected with malaria? 13% in a highly endemic month and a highly endemic country is low. It calls into question whether or not providers were thinking about malaria or were dismissing fevers as something else. Okay, among those suspected of malaria, how many were tested? We have a, what is our program, roughly $750 million per year to ensure that every fever in a more endemic country is tested. And here in this country, we received a ton of money. Only 35% of the cases were tested, but it was very low. And yet all of those were at 126% that were positive. Now yesterday there was a great talk and we had this line on screen, I thought that's a great line. I thought that HMIS is the cornerstone of information policy and planning country. Yes, that's why we have it, so I would want to use DHS too. But if this is the data, it's a cornerstone for a new project. What are people going to conclude? Even though this program was seriously underperforming, but the data is bad? They're both. But this gets to what the really underlying problem is, is how many facilities provided the data for these numbers? So what I've put down in the second column here is the number of facilities that provided the number in the first column. And you'll notice that in this county there was 219 facilities that provided the number for first attendance. And yet only 76 facilities provided a number for testing for malaria. And twice as many facilities gave a number for a confirmed malaria, which you may remember we had confirmed many cases then tested. I think that's pretty obvious why that is now. So it wasn't that the data was bad or this malaria program was seriously deficient, it's just that we only had partial data. And interestingly enough, you notice how there's different numbers for each of these elements. So if I had gone to the program officer and said, well, what's the recording rate here in this county? Let's say, oh, it's fantastic. It's actually there was interesting enough that the expected reports are 261 and 253 reports were received or a reporting rate of 97 percent. Oddly that didn't include, all of them didn't even put down first attendance because only 219 facilities filled that in. So this reporting rate for the data set just tells you that the form was sent in. So technically they're reported, but it doesn't mean that all elements were thrown down. So what we need is not the data set reporting but the data element reporting. So that percent reporting rate is unfortunately very deceptive. So one of the things that the Magic Glasses app does is it identifies the facilities that are consistently reporting so that we can look at the data from those facilities and not mix up those that are reporting this month and not the next month. We find that if we restrict our analysis to these facilities, we effectively eliminate reporting bias. But if you do remove some facilities from your analysis, you can't do things like what's the total number of cases. But what you can do is a better sense of these ratios and trends over time. So when I was putting out the data for one month, obviously my now I guess problem happens when we're working over many months. So on the left hand side, you see a chart of the number of Magic Glasses over many years for the facilities that didn't report every month. And this is actually the majority of the facilities and it gives the impression that malaria is on the rise. Again, given our investment in malaria control, this is very disturbing. But if you restrict the analysis to the facilities that reported every month, you see where it's basically a flat line, also is a little bit disturbing, frankly. But none of us are very different picture. So we're comparing the facilities reported every month to the facilities that didn't report show increase in malaria and that is probably due to just then over time we started reporting more and excused the data. Another thing that skews the data is outliers. We go through several algorithms to try and screen values that don't make sense. And like when someone was supposed to put in 105 cases and it became 1,050, those kind of errors. There are entries that happen because of enormous magnitude. And there's also errors that while they look like they're in kind of the right range, maybe 100 cases, they shouldn't have that during the dryest months. So we can pick up those seasonal and overall outliers. Now, there are a lot of ways of doing outliers. I'm not going to compare the algorithms. But I want to point out is there really a key difference between what we're doing here and what happens in the WHO data quality tool. And that's the after identifying these outliers, I can remove them from the analysis and I can document which ones you do. Whereas in the WHO data quality tool, it's great at flagging the outliers. But the other is the flags or the facility can go back and change them. But if they haven't changed that value, the data remains in the data set and that continues to be used for the analysis. So all those erroneous values will screw up the finely tuned analysis that we want to have later on. All this is to get to this final step of really being able to evaluate are things getting better or worse? How well do our interventions work? The key is that we start now with data from the facilities with the most reliable data, those that report every month or sometimes we can allow a little bit of wiggle room, maybe report 11 or the 12 months and the outliers being censored. Okay. So display here is a time series of malaria cases from one of our countries. And you notice that there was a big drop happening in 2019. And so to investigate that, we set sort of an intervention date. In fact, there was an intervention. There was a large bed net campaign here. And then we apply a number of time series models. I won't go into all the different ones, but there's not just a single one. There's many different ones. And they may fit better or worse. And we use a series of cross-validation techniques to find for each facility what is the best model fit. Then we find the best fitting model. We use that to forecast what we believe probably would have happened had there not been an intervention. But then we can, and that's what's shown here with the dotted line.  So from that best fitted model, we compare the expected values with the actual values, which are shown in red. And in this case, the best fitted model estimates a 39% reduction over 12 months, which is a fantastic impact from that bed net campaign. Turns out that bed net campaign was actually a little bit more complicated. It had a couple of different bed nets and also a few alternatives of vector control strategies. And we were able to split the data and compare the impact of each of those interventions during the same time period. Here's another example now, it's a little bit takes an extra step, and that's investigating an unexplained drop in cases 2021, 2022, actually going through to the present. What you see here is the number of confirmed cases of malaria. It is documented exactly what that one's going at, but for the purpose of this, I'm just going to say confirmed cases of malaria. And this is the raw data. This is what you would see in DHIS2. And you notice some amazing peaks and valleys. The impression that was given to me from the NMCP is that malaria just swings up and down, and it goes that way every couple of years. But around that have a couple of years that are down. Given there was a pandemic during this time period, one of the biggest things that I was like, oh, is it possible that facilities aren't reporting anymore? That would be, that could be very problematic. So why is this decline in cases real? Is it due to a lack of reporting? Could it be due to decreased attendance, perhaps lack of testing? So we did again, applying some statistical techniques from our recreated and adjusted corrected confirmed cases that controls for the potential for rare testing rates or lower attendance. And we then made a second time series that we believe is the closer to the real number of confirmed cases. That said, we saw what there is still a large decline, decline of 51%, that was not explained by lack of reporting, decreased care seeking, or decreased testing. This is extremely useful example for the malaria program that may indicate a real decline in malaria transmission. I'm going to skip ahead to one more example here. And that's about something called minimal detectable change, maybe not the best phrase. But the idea is that with the emphasis that we have on data quality, sometimes you just have to ask, look, when is good, good enough? When can we, we always want to keep making sure facilities do well, but we often find that there's always an example of a facility that had bad data or an outlet or something. And so it's easy to say, we need to double down on data quality analysis. But you really have to ask, but when is it finally going to be good enough? And I'd say that depends on the size of the change you want to detect. So for instance, going back to the data set that I showed you earlier, the best fitting model I could find for the data, the historic data, going up to, this is 2018, had a fit that was only about 29%. You mean that if there was a change that was greater than 29%, or less than 29%, I wouldn't be able to say that that was a real change and not just some random variation in the data. That means that if we had an intervention that was supposed to make a 20% impact, I don't know that I could actually detect it statistically. Similarly, if matter cases swung up by 20%, I may not be able to say that that's really important. And that far in the same country, through the data quality improvement program, they've gotten to a point where they've had a 10% fix, and actually the last time they did not show it here, it's about 5% to 7%. Which means if they want to, if they have an intervention that they're trying to examine, they should be able to see any change that's greater than 5% to 7%. And honestly, that's unlikely to do any intervention that's supposed to have an impact that's less than that. So one could say that probably for this country, the data is now good enough to do what you want to do. So I just summarize data reported to DHS2 provides a fantastic basis for evaluating program implementation and effectiveness. However, the uncorrected data is potentially really misleading. And the data doesn't need to be perfect, but it does need to have a systematic analytic process to use that data. And we hope that to improve program evaluation, we could get external analysis apps like this installed alongside DHS2 so that we could make the best use of the data we have. If anyone's interested in looking at the app, all of the R code is available at the GitHub site that is listed here. Thank you. So we will have a few minutes for questions as well. But one thing I just wanted to re-emphasize is that we've been talking about this for a couple of years now and John made it extremely clear to us that data set reporting rate analysis is not enough. If that's all you're doing, you're not doing enough. And you have to be looking at data element reporting rates because people are cheating. They're not filling in the data sets. And they know that. And they know that you monitor for that. And that's what they're being graded on. Yeah, and it's a perverse incentive because they're also being monitoring and getting it in on time, which means that I'd like to fill that in, but I'd be better off getting credit for getting it in on time than actually having it fill out. So John's presentation, I'm sure you've got a lot to say. John's presentation inspired us to write some guidance documents on how to use standard DHIs to look at data element reporting rates. And I think I'm still probably the only person who's read it, but it is on our website. We have a data quality guidance document on our website. Please have a look at that and utilize some of the tactics. We got the inspiration from John and hoping that countries actually start using it. But I don't think anyone actually has yet. Okay, Colin, you have a question. Yeah, two comments. One is that a lot of what you're talking about is things that you can normally can address through better tools that you're presenting and also the systematic analysis and monitoring of what's happening. But I'm also interested in why do we have so many problems? Because I looked at lots of different countries. Okay, and I see an enormous backlog in metadata quality and data quality, etc. Things that should be done, even the very basic things are not being done. And I think analyzing why, and I would like to mention those two things. One, I think the problem that DHIs do being generally a centralized system very often under the real control of informatics people, computer science people, and not health managers. It means that they simply don't see where your data don't make any sense from a health and a health management perspective. And to take a practical example here, I do a lot of work in Sierra Leone, I've been working for them for six years now setting up their IDSR system, particularly the case-based IDSR system. And what I find there is that the disease surveillance managers, they're thinking is a still paper-based. They're thinking like forms and data being sent from side to side, instead of thinking, oh, we have a central database, etc. They're thinking is like the exercise controls through data forms. And also what they're used to is weekly aggregating data. But the fact is that weekly aggregating data is always suspect. Right? It's not confirmed cases. Now when we have a case-based system which is actually slightly getting better of higher reporting rates, we can see that we didn't have 93 percent, sorry, 93 cases of acute barohemorrhagic fever in 2022. We had 10 confirmed loss of fever cases. All the other APC cases were of the lab analysis found to be non-confirmed or false. So I'm just saying here, a main problem here is actually to change the way health managers think about their data. And just to give you one thing you could do in any country instance, look at data elements and indicate that, see how many of them have proper definitions. I can tell you, you're going to find that most countries, 80 percent has no definition at all. All the definition is just a copy of the name of the data element and indicate all of it. And that it reflects a developmental problem how managers are interacting with data and information. Yeah, those are good comments. Thank you. The name Magic Glasses came from a health manager who said, oh, she could finally see what was in the system. Yeah. Thank you. Yeah, very wise words from the father of DHIS. He wrote the original code for DHIS. Any actual questions? Yes. Like I explained, when you have partially complete data, you can analyze that data. My other question is, can we use machine learning to predict the missing data? You know, so maybe you can see past data and all the like the reactions. Can you say that? Okay, we don't have this data from this, for example, location for this period. Looking at, can you predict that what like that would have to do? Yeah, it's a great question, which was, can we use machine learning or other techniques to impute the missing values rather than just censor them? And yeah, I spent a lot of time looking at that, thinking about it. I don't know, really necessarily came up with the best answer. But what I found was it was very hard to predict what that missing value was. And in so doing, I was imposing my model onto that data, when really I want the data to tell me what the model is first. So I worried about the tail was wagging the dog. That said, there may be some more sophisticated ways of imputing the values from neighboring facilities that could adjust for it. But in the end, I decided the simplest or truest method would be just to look at the facilities that reported every month where there wasn't missing data. Hopefully that's a majority, that's a good representative sample of the facilities. If it's not, that's problematic, but in most instances where I work, it appears to be a very representative sample, both geographically and in terms of small clinics, large clinics, et cetera. John, there's a question online about differentiating between zeros and blanks. Yeah. Yeah. That's another thing that people wouldn't be able to see in Magic Glass as you know risk when they were done is that zeros sometimes are not stored in DHS too, but it's a setting that is done by data element. It's not that or Malawi doesn't store them and Zimbabwe does. It is a question of which data element has it. And that's one of the attributes that is visible when you look at and measure the data elements. It says whether or not it's there. And so it is something to be mindful of. Fortunately, in most instances, for things like malaria cases, I've seen that countries have turned that switch off so that zeros are recorded. We have a couple more questions, maybe. Yeah, it's not that much, but we have more of such problems that are very important. I think there's quite a lot of great data on the issue of things. It's not that much. When you get there last, there's lots of data, lots of content in some cases. Thank you. Thank you very much. How many data elements do you notice that they are very complicated? That's one of the things that we're ready to talk about. But sometimes it might also be that we're very sensitive to data and how are they not coming up. And I think it's a little bit of a challenge for people that they don't know about. So we have this comment on the impact That's good. I mean, it's really looking at it as a thanks. I appreciate that. I wanted to, Scott and I had a conversation a couple of years ago, but the question was, the comment was about looking at the completeness of the form. If, and I remember when people asked me, why don't they just require it to be an entry? You can't submit the form unless there's an entry. That would do it. Scott, I want to, when you looked at that, that becomes a real barrier to getting the form in. And the fact is there are sometimes hundreds, one country, I guess I want them, their monthly form has over 1,000 boxes on it. And these rare things, it might be viral hemorrhagic fever. They're going to be left blank. Another common example is stock data. There's a box for a number of days, stock out. And they often leave that blank because they didn't have any data stocked out. Yeah, it would be nice if they read with zero, but at least they wrote in the other boxes. So there will be some blanks that are implicit zeroes. And if we specify, if we require one to read a zero, I think there'll be a lot of pushback. We're going to have to end it here guys. So if you're already finished over, so let's give it John another.