Welcome, DEF CON travelers!  I'm Liz Wharton, Chief of Staff at Scythe, but today I will merely be your narrator, your guide, helping you through the decisions that have already been made because you've made and created and voted on this adventure. When we talk about airports and airport cybersecurity, and particularly when we talk about the aviation, industry, cybersecurity in general, what we're forgetting is how far airlines and airports have come. That is, we've become more digitized as everything is connected, as air travel has grown, so has the threat landscape. And so has, well, the tax vulnerabilities and the pieces that make up this adventure. Decisions that you've made from the time you leave your home to transportation to the airport, departures, decisions made during flight, arrival, transit to your destination all provide, well, additional players and pieces, but additional threats and opportunities for bad actors. As the GAO noted in a recent report that airlines and their IT infrastructure and their systems also provide opportunities for potential IT outage effects from planning the trip, reservations, flyer, frequent flyer systems to check in, to boarding, to at the airport with the Airlines mobile app, the airport kiosk, the check in, the boarding, the baggage, the planes, the flight planning, the dispatch, all of these come together and have potential outage effects on, in some cases, systems that were not built nor designed for the amount of impact that they have. But when we look at the playing field for our adventure today, the tax surface, it's not bounded like you find in most, in fact, it's quite expansive. As you'll be traveling through our fictitious airport, you'll notice that, well, not everything is in the same place, not every airport is designed differently. So to create and craft policies, procedures, all the different pieces that will go into what I hope you gain from today's conversation. And well, future conversations is that airports and securing airports, be it the physical, the systems, the software, all of the pieces that come into play reports is not easy. In fact, we may not even get through or scratch the surface of all the different parts. Because well, who knows, between cyber squirrels and different choices that are made, you may not even get past well ticketing. Because within the airport attack surface, you also have a lot of players. You have from air traffic control to the many cities that airports truly are to security to your gate. It's a minefield protected and controlled by airlines themselves, each of the airlines plays a part in this. Other authorities, be it the local governments or other authorities to the different, well, local law enforcement to the FAA, DHS, security, concessions, vendors, the electricity, all the different telecommunications services that flow into each airport. Well, those are a different provider. For example, at the Atlanta airport powers provided by Georgia power under different agreements that have been put into place. So whether those provide opportunities for protection or vulnerabilities is open. Also, you have to look at the third party providers, the multi use software and systems. And so Garmin recently provided a great example of, well, when we think of Garmin, what do we really think of, do we think of the watch the directions, you know, are tracking our steps, etc. To the fact that Garmin provides aviation services from flight planning to mapping, and that when a ransomware attack hits Garmin that it's also impacting the avionics, the airports, and a shameless plug for a breakdown of the recent Garmin ransomware attack and attack factors, check out size threat Thursday in the news. And when we're talking and weaving our narrative, our narrative story today, also have to look at what is the intent that when we have incidents such as glitches, breaches and well, agent cyber squirrel hitting airports that the intent and the result matter, but what we sometimes don't are is chaos criminal carnage, or really is it just an oopsie that results in chaos, criminal activities or carnage. And further, what are the incident impacts that a power outage at Delta's operations center in August of 2016 $150 million and two days worth of cancellations that a software glitch at with one vendor provided worldwide check in and booking issues that a data breach results and finds at one airport that a software Southwest computer outage led to $177 million of damages and three days of outages and flight disruptions. All of these potentially could have been worse and all of them. Well, potential threats. So from ticketing to take off. It's going to be quite an adventure. Shall we begin and beware and warning, because this talk is different from other talks other technical talks and other talks in the aerospace village for in this talk, you and you alone. The things you voted in the Twitter polls are in charge of what happens. The mission was to see if you and your data can make it safely to your flight from ticketing to take off. You have to dodge the delays and the data breaches. So shall we begin. It's time to pack your things and head to the airport. And our first decision point.  I mean, you're going to Vegas for a week for Def Con, and all of the other villages, or do you bring a carry on, put it back and see what happens. Well, you chose a carry on wise move, because airline baggage check in and tracking has the potential for delays, not to mention data breaches. So now that you've got your backpack on, and you're about to head out. It's a digital and app world, who even has a printer at home these days. Do you have your boarding pass in your ticket. Do you print it at home. Do you use the airlines app boarding pass, or like me, I do a combination. I also take a screenshot of my boarding pass should something go wrong. It's almost evenly split. But did you choose wisely you chose to print your boarding pass at home. Well, well done, because ticketing frequently leads frequently leads to well delays and data breaches.  You know that sometimes there are long lines at ticketing and check in.  So Def Con safe mode, secure aerospace travelers. What do you decide to do. For the overwhelming majority, you chose to check in online with the 24 hour advanced window, avoiding the kiosks, and well the airline app, because you know the check ins are well the threat there are delays. So much so that when a third party provider covers well most of the airlines, you see a 30 minute outage of their software can lead to three airlines going down. A couple days later, a different airport provides service provider covered five airlines in 40 minutes. Saver was added again on April 29. Just less than a month later, 90 minutes, three airlines. You've checked in online, you've shown up at the airport, and you're a seasoned road warrior. Even if it seems like forever since, well, our last flight, cutting it close to departure time, and it's a big airport, flying on a weekday morning, bold move. How are you feeling lucky. So for decision for the fourth decision point, Def Con safe mode, secure aerospace travelers. What did you decide to do TSA pre check clear or regular TSA security lines. Most of you seem to be, well, hesitant with the facial recognition that comes with clear, but you're okay with providing some data and information because you're TSA pre check. Well, what seems like a safe mood move could lead to delays and a data breach. Well, here's what happens. Is it security theater where you have moving parts and multiple players, because when you have nationwide us customs computer outages, causing gigantic lines at airports. You had JFK LA X F SFO Philadelphia O'Hare Midway CTAC and other airports, all confirming delays. Our airport is probably hit by those. So, not only do you have a delay with pre check. But one of the recent trends has been for one provider to provide some of the software and systems support for TSA. In this case, facial recognition software provider and systems writer in EC. Well, pretty much covers European airports and mostly airports across the US. And while they take a long time to admit it, they've had a breach. The facial recognition data collected by US airline US citizens is stored for 12 hours for between 12 hours and two weeks and 75 years for non US citizens. And that data is stored in several government databases, which border officials can pull up when you're arriving or leaving the US, including at airports. And well, any seat is not very good at confirming their security breaches are giving a lot of detail. So in this case, you didn't avoid the delays, and you didn't avoid avoid the data breach, but you made it through security. Because when you have these data breaches, well, quite frankly, what are you going to do. Well, worrying about that check in and their security lines can be a headache.  It's a long flight ahead of us to Vegas. So Defcon safe mode, secure aerospace travelers and do you. Stop what you're doing because you've got to have your headphones noise canceling headphones and knowing your luck so far.  I don't know people. So do you stop and buy them now, or maybe we'll wait a little bit if there's a place to buy some closer to the gate. We'll go with that or quite frankly, you're feeling lucky, you're going to go for a roll the dice and well, who cares. Well, a slight, slight, slight majority went with who cares. You'll figure something out and you'll find a way to entertain yourself without your headphones, which when it comes to airport vendors and the threats of data breaches probably wasn't a bad idea. So now you've made it to decision point number six. Your departure gate is farther from the main terminal than you originally thought. It's time to move and get in those steps, but quite frankly, who wants to carry our bags afar. I mean, keeping in mind, we've got to carry on and a backpack. And while, while we've got our traveling shoes on, we haven't been exercising quite as much during COVID so Defconn Safe Mode, secure aerospace travelers, what do you want to do? Do you want to take the people mover, take the airport train? Or one two step, let's walk there. And you chose to walk there, which while we're going to get in the steps and we're going to walk this way, we had a double whammy, both delays, as well as the dreaded agent cyber squirrel. Well, in this case, just an agent squirrel, because lax on Thanksgiving Day 2015 had cyber squirrel reports, a squirrel plus a transformer, which in this case, while the power outages weren't severe in the surrounding area. At the airport, you had the moving walkways, the elevators, the escalators, the screening equipment, the baggage screening equipment that just stopped. So while none of the outages completely shut down the airport, well, for those of us trying to get our bags from security checkpoint to terminal, it's going to be a little bit of a long hike. All those speedy ways to get there are no longer at our advantage, thanks to Captain Chaos cyber squirrel. But we finally made it to the gate area. And well, decision point number seven. In this case, you know, beer pairs well with breakfast, right? We have had to deal with all kinds of different challenges. And, well, there are no seats at the bar. The restaurant doesn't look too crowded. And, well, a table it is, if that's what we want to do. So DEF CON safe mode, secure aerospace travelers, what do we want to do? Do we, okay, take time, stop for one and one only? Because, well, again, beer pairs well with breakfast. Or do we decide to wait a little bit? Because quite frankly, we're in first class. Did I mention we only travel first class. And first class has beer, even in the morning. And it looks like the majority of us, well, chose beer first class, not going to stop, which is not a bad thing. Because one of the other hidden hazards are the point of sale systems, and the threats and restaurants, a data breaches. Not only that, there's also a potential for delays. You don't know what systems the airport or restaurants and airports are running off of. Are they bringing in their own Wi-Fi network, or are they piggybacking off of an airport facilities or nearby vendors or hers? Are they piggybacking off of, well, free airport Wi-Fi? That's not secure. But there's no rest for the Wicked. And as we reach decision point eight, well, not only is there no rest for the Wicked, there's no rest for travelers with laptops and deadlines. Because we don't get to fly first class, and we're not heading to DEF CON. Unless we're big shots, and we need to review a draft file and respond ASAP. So we pull out our laptop and DEF CON safe mode, secure airspace travelers. What do we do next? Do we tether to our cell phone to connect for Wi-Fi? Do we use the airport's free Wi-Fi? Or do we carry a cell tower in our backpack? Because, you know, Wi-Fi pineapple and Wi-Fi cactus, they could have made it through airport security. So let's see. And while we do miss the sites of pineapple Wi-Fi cell towers in our backpack, the good thing is, is our data coverage on our cell phone will allow us to tether to it, which tether not to free Wi-Fi, because the threats there are data breaches. So as we've sat, we've reviewed our documents, and we're looking around, we've reached decision point number nine, because storm clouds are gathering on the horizon. And we think, hmm, there may be potential flight delays, locust storms. It's not winter, but still, stranger things have happened. And what happens if we miss our connection? Luckily, the airline we're flying on has an app. We can see over in the distance that there's a counter. There's a little bit of a line at the counter, but not enough to deter us. So what are we going to do? One, are we going to face down the storm? Because as DEF CON Safe Mode Airspace travelers, let's admit it, we are the storm. Do we use the airlines app and switch to a later flight? Or do we go up to the counter and talk to the person from the airline and attempt to rebook? In a slight, slight majority, we decide to go up to the counter and rebook, which turns out to be probably a pretty good idea, because airline ticketing systems and those apps both have delays, because if the app is out or hasn't been developed, you have problems with data breaches. For example, EasyJet had 9 million travel records taken in a data breach. And the British Airways was fined a record $230 million after data breach exposed the booking details of over half a million customers. Hackers had siphoned off thousands of credit card numbers after installing skimming malware on its website. So it's a good thing we didn't go to the website or use the app because that would have been bad. Instead, we just talked to a live person, which is daunting enough, but at least we're still on track for our flight. Yet, because check encounters do have software glitches as well. And, well, unfortunately, as highlighted the Greater Toronto Airports Authority, they had an outage with their airline check-in system that impacted processes at both terminal one, three, and had IBM technicians working with the technology authority to solve the problem. So the fears aren't always just the airlines. You also have to look at, well, what happens with the airline authorities? And are they providing assistance? Can they have those handles? And what happens when their systems go down? In Toronto's case, if you had checked in online ahead of time, which we did, you would have been fine. But then what happens when you're trying to rebook? And if you're having checked baggage and having to move through the airport, it caused additional delays. So unfortunately, in our journey, we've had the potential, we've avoided most of this data breach issues, but we've been delayed. Not enough to miss our flight, but we've been delayed slightly, which leads us to, we finally get to our gate. We show up and it's decision number 10. Surprise. It's a gate change and it's a crowded and noisy terminal. And let's be honest, who can ever truly hear what the airline attendants are saying and when they're announcing where you're supposed to go? Because we didn't catch it. The PA system was terrible. Now, where did they move our flight? What gate? Is it even on the same terminal anymore? We need to find out. So DEF CON safe mode, secure airspace travelers, what do we do? Do we check the nearby display screens? Do we go to the airline app, cross our fingers and hope that it's updated? Or do we ask a stranger nearby? We've already spoken to one person. We spoke at the counter. That's a lot of peopleing for one day. So an overwhelming majority didn't trust the app. Instead, we chose to check the display screens. Well, gate display screens are one of the big areas for delays. And what do we mean? You knew it was going to pop its head somewhere in the story. Well, ransomware. It's not just about taking down data. It takes down the systems as well. And in this case, the, well, Bristol airport got to find out exactly what that means. When in 2018 ransomware took out their signage with their gate information. Staff were left with having to hold up whiteboards directing people for where their flights are going. So in real time, are having to take that information, write it out. And what happens when they start running out of whiteboard space? Well, they found out cause delays. So again, not a data breach, because our data information isn't on those screens. But instead, it did cause a delay. We're cutting it awfully close to departure time, aren't we? Well, it's about time for boarding. And we're, we think we found where the right gate is. But at decision 11, hmm, what happens when the direction that we go? I mean, we can either go right or left. And we're savvy travelers. And while our frustrations are mounting, we're not going to panic. No, instead, DEFCON safe mode, secure aerospace travelers.  We can risk it, go left, if it's wrong, we'll go right. Well, turns out that airport has its own app that, of course, while we're bored, we downloaded it and we can talk about whether to download stuff to our cell phone without properly vetting. But it also has a navigation feature. We want to choose that. And again, we think about the issues with Garmin and all the other, well, where's our app data being sent? Where's our location data being sent? So maybe we're not going to go there. And while we know some of the signage in the airport is out, surely not. All of the signage is out. We can check the digital directory signs because, you know, they're running on a different system, aren't they? Spoiler alert. Most of the time, no. And the other thing is there's hidden dangers with the directory signs as well because let's think of, think back to, oh, I don't know, some of the botnets. The mirror I bought net that liked target IoT devices. Those LG screens you see all around the airport. What are those? But waiting, danger, waiting to happen. In this case, we're going to risk it. We're not going to panic. We're going to check those digital directory signs because surely, surely they're not all it again. Get ready for delays again, because ransomware has hit multiple airports again, impacting the digital of signage around the airport, displaying, again, only back black screens. Cleveland Hopkins International had this happen in April 2019, you know, back when people flew still. And took out their computing systems as well. So like that, you had to worry about whether their email, their internal app, their internal direction, if they're able to get some of the information at it. So, once again, wasn't our data that we were worried about as much here, but our ability to catch our flight. We're getting really delayed here, which leads us to, okay, we finally found the right direction. We're heading there. We're almost there. We skipped that beer, because there was going to be some of that on first class. So we're walking by the newsstand, and we spot the bestseller burning book. And we've been meaning to read it. I mean, on this call, and Peter Swinger did a great job, we heard in bringing and predicting all of these IOT connected smart city. And one of my favorite drone issues. And, well, we heard that lawyers get the short end of the stick. And this, and who doesn't like to see bad things happen to lawyers. So DEF CON safe mode, secure aerospace travelers, what are we going to do? Are we going to stop, get a book and, oh, wait, look, there's a sign. And check out says, if you pay via this payment app, you'll get a free coffee. So we didn't have a beer. Who doesn't want a free coffee? So do we decide to go for the coffee? Or buy the book, skip the coffee. We don't need another frequent shopper card. We don't need another stamp. And again, it's on our business credit card. So who cares? I would never do that. I would never pick that one. But the purposes of our narration cream. Who cares? It's the company's card. The data gets stolen. Not our problem. Well, luckily, you chose to buy the book and skip the coffee. Solid choice and excellent read, especially considering. Well, you didn't pick up the headphones. Because again, the payment systems and the shops are notorious for data breaches, as well as well, best practices. And with those third party payments, apps and different things, even within the airports, you've had currency exchanges go down to different attacks. So, okay. We finally made it to the gate. And, well, we've made its decision lucky 13. We've grabbed a seat in the boarding area, and we noticed that our cell phone battery is really low. I mean, we tethered it so that we could send and review those documents and those files. And, well, how are we going to tweet and text from the flight. So, savvy def con safe mode aerospace travelers. What are we going to do. Use the extra charger you're carrying in your backpack because while you're not carrying a cell phone tower. Well, you do know to pack an extra charger. We wait, risk it, because we are first class. And as we know from flying flying first class frequently. There's chargers, or there's outlets, and there's our free beer, or do we go and find an outfit, or excuse me, an outlet, because quite frankly, you never know. Again, tweeting, how else are we going to show people that we're sitting in first class. If we can't tweet a picture of us sitting in first class, holding our free beer. Well, charging devices. Well, in this case, we're going to fall victim to agent cyber squirrel. Because he just hyper squirrel captain chaos likes to cause brief power outages. And again, in this case, it was the bio buffalo Niagara Falls and Niagara International Airport caused a brief power outage. It affected gates. Well, a select number of gates. It only caused one flight to be delayed for a few minutes. But in this case, luckily, we had our, we had our charger with us. So while agent squirrel calls power outage at our gate. Again, he didn't cause us to have any issues because our backup charger was fully charged. Well, decision time. It's finally the gate agent is calling our flight and don't forget, we're first class, because that's how we roll. And with first class, we're the first ones to board. Oh, lovely. The airline is testing facial recognition for the boarding process.  So what are we going to do. We've made it this far. We have really cut it close. And quite frankly, we're tired. So weary def con safe mode secure aerospace travelers. What do you decide to do. Do you opt out because you've heard all those stories about facial recognition. And you know that a lot of those algorithms are wrong. And that's not to get into you. Well, quite frankly, it's just an upfront. You're not looking picture perfect right now. It might as well if it's going to keep the boarding process moving and if it's going to be convenient. Sure. But we did get that new AI facial recognition defeating tattoo and makeup. And while it was funny going through security. We're so close to boarding or flight. In this case, you decided stand up on principle. And no, we're going to opt out and use the paper boarding pass. Well, not only does facial recognition at the gates and also throughout the airports. But ticketing cause data breaches.  It also causes delays. So you can opt out. But that's often into wait and wait and wait. Because as we've learned through several different approaches that yeah, you can opt out. And as Zach Whitaker has a whole article from May 2019 that walks you through it, but know that opting out means they're going to go manual so that the airline staff will manually check manually, check your passport or boarding pass. Like they would normally do when you're boarding a plane. And that also means you've got to sit to the side and everyone else who's going through the facial recognition is probably going to bump up ahead of you. So what's the point of having our first class perks if we're going to sit to the side. Well, that's assuming the facial recognition technology and equipment is even working. Because according to one of the watchdog groups, the facial recognition systems at airports only worked 85% in some cases. And quite frankly, we've got that new face tattoo. And while all those delays and delays and delays were waiting and waiting and waiting. Well, and that's okay, because quite frankly, we still made it onto our flight with our data breached through several different choices, as well as our delays. But when the weather clears, and our flights cleared from ticketing to take off, we've made it. And really, through this hacking adventure, we've learned several different things we've watched how the airports, the airlines, and well vendors and different service providers all play together. And all some of the pressure points for where cybersecurity has a lot of policy has a lot of room for development. But one of the other things to keep in mind is with each of the choices. There was a lot that we haven't uncovered, or that we didn't get to discuss, much like the choose your own adventures. You chose, well, wisely, but you also chose poorly. You didn't die of dystery.    But it's open the doors and discussions to see how you would do the next time you go through whether you would make the same choices, because from ticketing to take off, airports truly are a hacking choose your own adventure. It's been fun to be your tour guide. I encourage you to check out the rest of the aerospace village and all that DEF CON has to offer as we go into safe mode and go digital. Find me at lawyer Liz and also follow site at site underscore io. Thanks for flying with us.


On immigration in the border, as we discussed here today, Vice President Harris has been put in charge of addressing the root causes of the border crisis. Last Tuesday, she spoke with Guatemala's president, but she hasn't visited the border or Central America or spoken with leaders of El Salvador or Honduras. She was traveling this week, took time to visit a bakery in Chicago. I'm wondering, is she still working on this? And can you address the perception that she's kind of quietly backing off while the secretary of my orcas is pursuing some Trump-era policies, such as potentially building new border barriers and potentially prosecuting people who illegally cross multiple times? Is it just me or is Yamachi Alcindor there giving that New York Post reporter a death stare? If you're not familiar with her, this woman is a far-left activist and mouthpiece of the Biden administration. So it's not surprising that she would be upset that this guy is asking critical questions of Kamala Harris. There's a whole lot packed in there. So let me just see what I can do here. First, I would say the Vice President was visiting Chicago actually to talk about COVID and the importance of communities getting the vaccine when it's available and accessible to them. And so while she was there, like many Americans, she got a snack. I think she's allowed to do that. But she was there to talk about COVID and play a role, as she's playing a significant role on our efforts to address vaccine hesitancy, communicate with the public about how we can do this and it's imperative to get the vaccine when it's available. Second, I would say that yesterday, and this is actually a statement put out by her team, USAID announced the deployment of a DART team. We only have these in a couple of places in the world. So this is apparently significant to respond to urgent humanitarian needs. That's what she's focused on. I don't have an update on when she'll travel. I'm sure it will be soon. But she's still working on this issue very much. Well, they wouldn't be putting out a statement this morning if she wasn't.


So I'm going to proceed here and the first thing is just to point out. We're often in the digital health community running around looking for evidence about what digital interventions to be applying to our health programs and It's worth pointing to in saying that the World Health Organization did a very large systematic review in 2017 2018 Which led to their first set of Recommended digital interventions that were published in 2019 This was based on all of the available evidence at the time about which digital health interventions actually have an impact on things like quality of care health outcomes or or on data quality data use With I think very high levels of endorsement So dr. Tedros here in the opening to this Putting forward the WHO position that a lot of these interventions that are recommended We should no longer be thinking of them just as a luxury But they're rather a necessity in order for us to achieve the sustainable development goals In order for us to really bring Quality of care to every every woman every child every person So from the University of Oslo, we agree with this this stance We also have have done a lot in the last few years looking at these recommendations to make sure that the functionality present in tracker aligns with the kind of recommended interventions in that set of recommendations there were 10 or 11 different Interventions and tracker is set up to cover seven of those at least Depending on how tracker is used and which functionality that you intend to use it for But just to say that this these Set of interventions again, it relies very much on what you choose to do with it and how you set up your tracker configuration But knowing that there's an evidence base behind some of these functionalities So really it should be something that you are considering with your own implementations if you're setting up a tracker program and are going to be able to have access to kind of individual longitudinal data the kinds of Interventions that you can put in place. I also wanted to give just a quick state of tracker use Often those that are familiar with DHS to don't know how far along tracker has come adopted until they become involved in one of the projects DHS to of course began as an aggregate kind of HMIS system for reporting for for analytics For capturing data from the district level and above But it's been over 10 years now that we have had the tracker component in DHS to collecting individual longitudinal data The system itself has progressed quite a bit in that amount of time and at this point at least as far as we've identified We have at least 77 countries using tracker at a large scale Most of those countries are using more than one tracker program. They're covering a variety of health services using tracker Again, we have kind of a minimum of a hundred and twenty thousand regular users Which in in the world of DHS to is is quite a lot of users because we're typically aiming then below the national level users We're talking about the the people at the labs the people that are providing health services the nurses the midwives The doctors that are relying on tracker day to day So the the extent with with which tracker is being used has grown considerably and we assume There's actually quite a bit more than this going on because of course we aren't Implementing these projects ourselves. There's many users that we never contact or that don't have You know a close connection to the University of Oslo, but they still are adapting and using tracker for many different purposes Despite that the fact that there are so many users of tracker with so much functionality that it can be very overwhelming We realize when it comes to adopting tracker and starting out as a new tracker user So in my mind, I liken it to this giant box of just mismatched Lego You can do anything that you want But it often seems a little overwhelming figuring out what all of the functionality is how it best connects together and What you can use tracker for So one of the initiatives that we've had ongoing actually for for years now the University of Oslo is a collaborating center with the World Health Organization and Part of that partnership we have been making preconfigured packages of DHS to that take into account the the global recommendations for given diseases or health areas the Indicators that are recommended the the kinds of you know decision-making that wants to be done And we work closely with not only the World Health Organization But some of their partners on creating these packages So for example, we've worked very closely with Gavi with UNICEF with Global Fund With the HIV team at WHO many different groups to create these different packages of DHS to Which are preconfigured are available to be added or adopted as your first package within DHS to The again, we'll have a link here in the slides that will take you to the the landing point on on DHS to org where you can Get the documentation behind these packages You can get the information about how to adapt them and we'll talk about this a little bit later in the session Some of the the key considerations of what it takes. I did just want to put a list here of the existing tracker preconfigured packages So again, all of these are packages which we have worked on with various global health partners And come up with at least a standardized starting point for your tracker configuration and any of these various topics They're meant to help you to again take advantage of the functionality of tracker Without needing to first become an expert you can see from the the package the way that it's laid out What components we are using what data are being collected? What is the workflow and process? How do we do the sharing amongst users the privacy settings? Pre-configured dashboards that provide the recommended analytics with the different visualization So there's quite a lot available to you if your country or your project is starting out in one of these Topics right now hoping to use tracker or at least to see what tracker could be used for We also have demo sites set up for these packages, which again are in the link So you can take a look before you ever need to install anything and see if it matches the Requirements that you would have for your national system in any of these various topics So in my mind these packages are a bit more like this set of Lego So they are ready tell you what you can be building. They come with instructions They they make it a little bit easier than that giant bucket of Lego to have a vision about where you're going and Know that you will end up with a certain Final state of the tracker configuration So now I would switch over a little bit to talk about why You would use tracker and we'll start looking at kind of the the prerequisites before you begin a tracker Implementation or if you're trying to strengthen a tracker implementation that is maybe giving you troubles The the first thing to point out is that tracker does come with kind of benefits and challenges It's important to be prepared for what those challenges are and try to mitigate them I think that the it's it's not unique to tracker this idea that we in the public health space want to have better Individual level data This has always been the the hope and the kind of cornerstone of proper health decisions health management decisions clinical decisions Is to get the actual individual level data where it is generated? Whether that's coming from the you know a lab sample and the result of a test or whether it's the the patient Providing information about their nutritional choices This is the information that really allows us to to dive much deeper into a given health topic and and allows a ministry of health to make more informed decisions about About the use of medications about the diagnosis processes about the reach of a given disease So there's there's much that can be said for capturing this individual level data a Lot of it can be structured to not only provide that data for decision-making at the higher levels But also provide some benefits to the people collecting the data So if you are a healthcare worker, you're seeing a patient you can enter their information directly into the system and It will generate the reports for you So saving you a lot of the time that you would need in order to do your your standardized reporting But it can do more than that it can provide you with decision support It can produce a working list of those patients that need a follow-up You can standardize some indicators that make sense for a single clinics management of a given disease So there's there's a lot of benefit to trying to set up your individual data system So that it isn't simply a reporting mechanism, but it actually can be used day-to-day operationally while also collecting the data that you want for reporting This data that you're creating because it is individual data It's really useful long-term for conducting kind of ad hoc analysis for diving in when there's a new health complication or New medication is administered You can go back to the individual level data and do new kinds of analytics that you haven't planned for previously You don't have the problem of trying to disaggregate something that was previously aggregated The data are there for you to conduct kind of the research and policy decision analytics that you want to Of course, all of this comes with increased complexity So setting up the configuration of tracker in the first place Because you're thinking about clinical workflows or you're thinking about, you know, structured data coming in from a level where they're not used to using digitization Tracker itself if you're you're building out a national program, for example supporting Antimatal care, that's going to mean that you have a large much larger number of users than you previously ever had with your health platform They're collecting a lot of data. This brings performance considerations hosting infrastructure considerations And the data itself by the very nature of it being individual data is more sensitive It can be a part of a person's health record the data if you're registering people it can be personally Identifiable and so it brings with it a lot of considerations about privacy about The performance and how quickly and routine your system needs to be able to respond to the the various Needs of the clinical worker. It's it's much more catastrophic if your system goes out for a couple of days Than it would if it was just a reporting system. So these are significant challenges. They shouldn't be taken lightly. There should be a lot of planning that goes into ensuring that you have the resources and the team that you need. And that you have the ability to support the number of users. So from our perspective, these are kind of the key considerations in these square blocks and the key prerequisites in the kind of long rectangular box at the bottom That would ensure that your tracker system actually does bring you the benefits that you see at the top. So all of these components are critical. They require that your management team, your operational team, your supporting it support structure, all of these different pieces are fit together well. And that they are supported long term and that they can be kind of a sustainable infrastructure supporting your tracker implementation. So these are the things that will now dive into with a bit more detail. Again, we clearly won't be able to cover every aspect of these but wanted to highlight some of the kind of most important information for any of the given areas to make sure that People are going into the use of tracker with their eyes fully open about the resources and the support that are going to be needed to ensure that it works well. So breaking these down into a couple of the different areas that we can have some some quick points to mention. So the scale of a tracker implementation can be dramatically more than what has been previously done in the in the country or in the national health program. It depends very much on how you utilize tracker and what you want to use it for. So it's very possible, of course, to have a simple tracker program. That you use with a small subset of users and that really all your imprint interested in is setting up maybe some sentinel sites for reporting. And those that kind of scale is much more doable with kind of your existing team, your existing IT support structure, the existing resources that you have. But what we've seen from most countries is that this is not what they want to do actually what they they want to do is to be able to push the use of tracker down to the lowest level. To the health sites to the community health workers and that this brings with it, you know, a vast increase in the number of kind of IP support that you need with the performance requirements that there are for hosting infrastructure. For training, for example, which we have listed here. So we're often asked kind of about the costs of tracker. The cost for configuring your tracker and setting it up and going through an approval process are very similar to what you would have done previously in a country that is using DHS to for their HMIS, for example. But when it comes to training costs. This is where you see a vast increase in the amount of resources needed and the length of time needed. So if you're attempting to train 10,000 midwives, which is something that, for example, Bangladesh has done. The training plan for that it has to have a lot more trainers needs to have a lot more time cost a lot more money. And a lot of thought needs to go into things like the turnover rate. If you're investing this much money in training your entire cadre of midwives. How many of them will no longer even work there by the time the round of training is done. How many new people will have been hired that weren't a part of the training. So there's a lot to be considered there about, you know, the reach and extent of what you're trying to do with tracker. One key aspect of this is the it support unit. Again, we've seen many countries that are Well functioning already using DHS to and other kind of software applications. They maybe have a core team of, you know, seven or eight people at the Ministry of Health. Or working, for example, with one of the his network organizations and they are used to providing the routine IT support to a cadre of maybe 100 users. When you jump from 100 users at, you know, 20 sites to 10,000 users at 3000 sites. This is an exponential increase in the amount of kind of IT support calls that you will have. And so it'd be very important, important ahead of your roll out and ahead of the training plan to have structured an IT support unit that can actually be responsive to their needs. This very often means that you create kind of a new level of IT support that is based at the district or at the region. You're maybe taking advantage of existing, you know, data staff that sit at the district level and you're training them to handle kind of the first line of IT support. So when somebody forgets how to log into the system or they don't understand why they're getting an error when they haven't completed the forum or something that you have a large cadre of IT support staff, which can at least answer those basic questions before referring some of the more complicated things up to the higher levels. That core IT unit sitting at the Ministry of Health should probably be expanded with some additional resources that are well suited to doing things like handling the hosting and support infrastructure for a system that is much larger. So, so again, just ensuring that in the program when there's planning about the resources and the ongoing resources that a big part of that is making sure that you have an appropriate plan for the kinds of IT support requests that are going to come in from a vastly expanded user base. One of the other key considerations about what to do with tracker is to think about the impact of doing real time data entry versus secondary data entry. So if you want to take advantage of some of those key interventions that were recommended in the WHO guidance, you want to be able to provide decision support. For example, then you need to actually have a plan for something like real time data entry where the person that is doing the data entry is the person that also would need to receive the decision support. Which comes based on the data they've entered that a rule is triggered and it provides them with a recommendation. If you're not able to achieve doing real time data entry, then your program can probably be a little less complex. You're doing secondary data entry. You have a data clerk that's one step above the facilities. That's taking all of the paper individual data and entering it into tracker. This still is not the same as the aggregate data entry that they're used to. Rather than getting a monthly report with their summed up numbers, they are now taking individual records of some sort, whether it's the ART treatment file or it's a lab result or whatever it is, and they're entering them in one at a time into an interface that first required them to do a search, identify the person or the tracked entity that they're looking at and enter that data. So the process for entering individual data of course takes more time, even if you're doing the secondary data entry because they're supposed to be assigning each of those data elements to something that's being tracked over time. So again, it's a trade off deciding whether you want to try to incorporate those benefits of real time data entry versus pushing some of that work on to a secondary data entry process. And it's important to mention that you don't need to have a single approach for any given country or any given health program. So it would be very useful beginning your project to look at the reach of your network to see how comfortable your users are with adopting a real time data entry system. To look at where you have data clerks that you can take advantage of them doing secondary data entry. So it might be that in portions of your country, they are using tracker as a real time data entry system and other parts of the country they're doing secondary data entry and still in other parts of the country, they're not even doing tracker they're doing aggregate reporting. This is the most I think kind of responsible way to implement. So rather than making a single decision for the entire country that doesn't take into consideration their kind of current infrastructure or human resources. But you rather do a more in depth look at who your targeted users are and what is likely to benefit them the most. And you can also then of course over time bring some, you know, additional sites into the real time data entry versus secondary entry or move some of the aggregate users finally over to doing an individual level kind of system. So there's there's a lot to be considered there. This also has an impact on whether you're choosing to do your tracker system on Android or in the browser. Android is the solution that we have for offline data entry in tracker. If you're running the Android application it's storing a local copy of the data, even when you're not connected to the server and that data can be synced at a later time. But of course using Android then also brings a lot of complications, you would be most likely providing hardware to the users rather than relying on them having their own. The the syncing process itself, you might need to have individual plans depending on the area and their network connectivity about how they sync and where they sync. So for example, you might want to train the users that at their site, they may not connect to the server, but when they go for a weekly meeting at the district, they need to bring the device and do a sync there. For example, so there's there's some complications with adding in Android there's additional cost associated with it. There's the concept of needing to keep track of all of the hardware that has been put out there. For those that lose or break their device. How did they get a replacement. What is the supply chain for that are there existing, you know, hardware resources that can be shipped to locations as they are needed to replace their Android devices. And even the life cycle of the device itself. Are you planning for you know the the 10,000 devices that you put out into the field. Are they going to last one year two years will you be able to replace those with another trunch is that built into the budgeting process. The web users on the other hand, typically are those that are sitting at a site that has a strong network connectivity, probably they already have at that site computer desktop or a laptop computer which they're using for other purposes. And so, Sorry, it looks like my screen has just decided to change what we're looking at. So yes, the web is more appropriate for that type of user, the web has a constant connection to the server, or at least it only breaks connection very intermittently with for small bursts. And so the system is essentially doing real time data collection at that point, meaning that the data that are being entered would immediately be something that you could analyze. It's also available you know from one room at the clinic to the other. They have just updated you know one part of the record as the nurse has seen the patient and then they are sent to another room for a lab. And the data are available on Android unless both of the devices are connected, then you can't really expect that they're going to have the updated information in the next room over if you haven't been able to sync. So again, a lot to be considered their trade offs. We see that very often for these kind of large scale implementations there's at least some portion of their user base that needs to be on Android in order for it to work well. But that will again carry additional considerations about it support and who is capable with the Android side and able to respond to those users as they have issues and many other considerations. So we are often asked about how to plan for this then we've we've brought up a lot of different scenarios that have an impact on the amount of resources that you need and the the length of time that it will take. And so we have put together several budgeting tools we have information that are in the DHS to implementation guide the tracker implementation guide the Android implementation guide. We also have this spreadsheet that I'm showing a screenshot of here which was meant to give a sense of kind of the different levels of adoption around the packages. So the first yellow area showing you some of the costs for a basic kind of aggregate package, maybe moving forward then to include more sophisticated packages like the electronic immunization registry. What those additional costs might be for things like field testing etc. It may be that the country wants to add some custom apps or build in some interoperability scenarios those have additional costs. So it's it's a little bit difficult to give kind of a standard answer about what is the cost of implementing tracker, because it depends very much on what you want to do with it how many users you're reaching. How long will the training go on for how many trainers do you need what is your hardware costs from from the software side it's making sure that you have a team that is capable and can work with the tracker application. Which is potentially the same cadre of people that you have working on your aggregate system, but maybe not maybe you need additional people who have been trained with tracker maybe you need an external group that is providing support to the ministry. So it's again we have some tools that you can use a big part of what we will do in the level to Academy is to have a planning spreadsheet that every user has and we fill out regularly throughout the 10 day Academy. Adding in more information as you have it about what your implementation is to again give you kind of a better estimate of the resources and the costing that are associated with adopting tracker. So for those of you that that either already have a tracker system or are planning to introduce it the the implementation Academy really is not focused on the software development it's focused on the planning on the management on the day to day operations. And those are the kind of people that would benefit from attending those that are responsible for making those decisions for coordinating teams for recommending timelines. So that's what would be focused in this the focus of this level to Academy. Okay, I'm going to stop my slides on on this one I'll turn it over to Marcus and just a moment I know there have been questions that have been coming up in the community of practice. Please do add your questions there we have some people responding to those and we'll we'll cover some of those questions and answers after Marcus shares. And this topic of interoperability again it's it's often a question that is raised about what DHS to will interoperate with and what tracker interoperates with. This, we have of course standardized approaches for integrating with other systems, DHS to and DHS to tracker have always been open systems with a strong API for connecting to other systems. We have the full documentation behind that there you know every every system that I'm aware of is making use of some aspect of the API to extract data they want to share it with other applications to do outside analytics. So it's it's a very well supported and well used aspect of DHS to that you can use the API to extract the data that you want. On the aggregate side the standard that we follow is the ADX standard. And again, any other system that is using the ADX standard it should be fairly straightforward to do connection with the DHS to aggregate data for tracker we have built a fire interoperability layer in the last couple of years. The fire layer is it comes out of HL7 it's meant to provide the standard approach to dealing with the clinical level data. What I would say right now about the fire interoperability layer is that it requires knowledge of fire as a standard. You would want somebody who is really an expert in in the fire approach and understanding how to map this interoperability layer on to the system you want to connect to and the tracker configuration that you have set up. So there's it's it's not the simple fix it's something that requires some expertise. Excuse me, we are of course ourselves always working on additional connections to any system that would provide kind of a large scale benefit to the user base. So those that are ongoing right now that we're working on. One of the WHO packages is for reporting adverse events following immunization. There's actually a webinar on that that's starting very soon today. This is of course extremely important right now with the COVID vaccine being able to capture if there are adverse events and monitor the situation. And there's a global repository that WHO supports for all of these adverse events, which uses the E2B individual case safety reporting standard. And so we have mapped the package the WHO adverse events package in tracker to be able to submit the information to this global repository. There are some 40 plus DHS to countries that already have a mandate to report to this global repository called Viji base. And so being able to ensure that they can do the reporting using the DHS to application was really important. We've worked closely with the core team behind this global repository in Uppsala Sweden to ensure that we can do this kind of integration. Another one that we're working on right now is called DIVOC. It's a vaccine certification platform that is from non NGO nonprofit organization out of India. They've been doing vaccine certification and other certification for for many years. And what they have created is, you know, an open platform for countries to use the vaccine certification capabilities of DIVOC for the COVID use case. They rely on the W3C verifiable credential standard, which is something that we've been working on to show countries how to take, for example, the the COVID vaccine registry package and be able to link that up to this solution for producing a vaccine certificate based on this global recommendations. This is something again, it's in the works right now. In fact, we've been exchanging information about it today. But we already have, you know, a recommended kind of approach for this countries that want to adopt it soon. We could work closely with them to make sure that the prototype comes out well and that it could support their use case. We also again are closely working with WHO. They've recently published their first digital adaptation kit. These kits are again focused on individual level data and provide kind of non technology agnostic recommendations about the standards and approaches to use for collecting clinical data. They've also published the anti natal care digital adaptation kit. So that's something that we're working with now understanding what's in that kit what the recommendations are and ensuring that tracker can adhere to the recommendations coming out of WHO. We are also asked all the time about some of the other kind of existing digital health solutions in our space, such as ODK com care open MRS. These solutions are software that already have been integrated with DHS to in many countries. Usually this is the result of some specific need for integration and so there's been some custom work involved with linking in those solutions, but it has done many different times and many different ways and done successfully. We know that these can be connected we know that there are various approaches to doing so, but that this DHS to tracker can be mapped up to any of the kind of existing solutions that your country is using to make sure that the data are exchanged to ensure that the data coming out of those systems is also available for aggregation and for use within the DHS to HMS for example. My key takeaway for this point is just to say there's no easy way to do this, none of this is just plug and play. If you have interoperability requirements for your tracker solution. It's important to think those through early on in the process of adopting tracker plan to have a team that provides support for interoperability. This will be a process of mapping which data that you want to exchange with another system or receive from another system, making decisions about whether fire is the standard that will work, or if your other system doesn't use fire is there another way that you're going to be exchanging data, having somebody who can manage working with the DHS to API. So again, it's, it's something that, you know, it's it requires a dedicated stream of work it requires people that know what they're doing. And this really for the foreseeable future this is how interoperability works. It's the same for all other software applications there's no such thing as kind of a free, easy way to interoperate systems that always is going to require somebody making key decisions, and somebody putting in the effort to ensure that those systems remain connected over time, as you make changes to DHS to or as the system you're exchanging data with changes. So with that, I think I will pause my presenting and I'll turn it over to Marcus to carry us through some of the next topics. And again I'll be taking a look at the questions that have come into the COP we have others answering them please feel free to put your, your questions there will come back to those questions once Marcus is done with the next round of slides. So, Marcus over to you. Thanks. Thanks, Mike. The next topic that we're going to cover is some key considerations for designing configuration and in many ways going from from aggregate to to tracker is like, is more like going from zero to one than from going to one to two. And for the design and configuration process you have to go through it's this is also true. The options for building and modeling, especially a tracker program is is vast. There is a lot of different ways to solve a problem. What you do is directly affecting a lot of users, a lot of users that is probably going to use your, your program for much of the day. And making a design for a tracker package or for a tracker instance is is is a very critical process that will require design skills, and it will require interaction with the users. It will require knowledge about the options that is available to the, to the team and configuring and setting up tracker. And so what we have been seeing over the years is that when everyone starts from scratch, then there's a lot of things that can be done very differently and a lot of goals that can be met in very different ways. And that makes it hard for us as product managers to, to make sure we build the best software for you. Because every time we make an assumption we are essentially wrong or someone has done something creative that we didn't know about. So it's very valuable to try to point everyone more or less in the same direction when they're going to build a tracker implementation. And one of the tools we are using to achieve this goal is the tracker packages. So Mike, compare the tracker packages to a box of Lego, or the, to an instruction manual for Lego. And one of the best tools is a, is a, is a box of, of plain Legos and these tracker packages are kind of pre-built sets that will help you this help describe how a Lego house like this might look. Tracker packages are useful in many ways to, as a starting point, as a learning tool, as inspiration. And when you are starting a tracker project, one of the first things you have to look at this is your existing ecosystem. Do you have the address already, or do you have a system already. Do you have, do you have something that you need to integrate with. In the interest, you have, you have an existing DHS instance. You would probably have an existing instance with your HMS potentially, or you might have existing tracker programs. And looking at the existing ecosystem is very important when deciding how to proceed with your tracker project. And it's important to think about when you're, when you're considering starting with one of these packages that has been built. And picking up a package. You essentially have two ways. You can directly use it and, and if you're starting from a blank slate if you don't have an existing ecosystem, installing a package is very, very easy. It's, it's just a few clicks and the package starting point would already be running. If you have an existing ecosystem, then you will either have to look at integrating the package with your existing data with your existing setup. And this can be a pretty involved process. And your existing attributes and existing data and definitions to this package is, is not something to underestimate. So what some instances end up using and doing. And what we think is useful is that instead of taking the package entirely and importing it as it is, and you might kind of pick up the manual for the package to design reference and you might even build the program from scratch, matching your metadata and your, your situation looking at the book that followed your Lego set instead of installing the entire Lego sets. So we're always an adoption process it seems and when, when, when starting a new program or, or a new instance of tracker. There is, there is always, there should be a review with the stakeholders for this, for this, this product that your, your end goal is. You would have to review the, the configuration and, and your goals for the, for the setup. In, in most cases, there would also be national guidelines that would maybe not adhere exactly to the guidelines that was used for building the package. There is, there is some types of packages that is more, where the WHO recommended guidelines are more, more universal and there's other programs where these are not so universal where they're national, where the national guidelines differ more. For example, for modern child health. There, there seems to be quite big differences between national guidelines and WHO recommended guidelines. And therefore, any package even if you start from fresh, you would have to review the underlying guidelines and potentially change to match your country, country guidelines. The, the last point there on the adoption process is that you, it comes a little bit back to the, to the users as well. It is usually a good idea to start small and then scale up. And one of the advantages of this is that you will, you will most likely not hit 100% the first time. Your actual users and actually use cases will, will uncover problems that you need to fix. And fixing these problems with a smaller, smaller group is usually a good idea. So that your big country training and training trainer of trainers doesn't have to be done again if you have changes, significant changes to the, to the setup and design. Then the last part that must not be forgotten is the, the HMS if there is an HMS. And this is comes a little bit back to the first point of the existing ecosystem. If there is an HMS. And you are introducing a tracker. It might very well be that the indicators of the HMS should also be reviewed. Most of the packages for tracker are coming with a, with a sister package for, for aggregate HMS. And, and these are also undergoing reviews from time to time. And we have seen that in many use cases, it is useful and you should go back and review your HMS. If that is not an option, then at least make sure that the tracker is collecting the data needed for your indicators. If you have an existing HMS or reporting needs that's going to be served by this, by this tracker package. All right, and this is, this is also the next topic I want to cover. And this is another room of the house that Mike showed us earlier. Because in, in many use cases that the tracker will serve different purposes and one of the purposes is very often reporting and HMS. And the existing data in your HMS is quite different than the tracker data that you're collecting. And just highlighting some of these differences, the HMS would normally be time box reporting and, and your users would spend, spend some days entering the data and using the system and maybe much of the month is, is the users wouldn't enter data at all. The reporting happens in batches. Then the data use is more of something that's ongoing, but reporting happens in batches. And that's in most cases where the vast number of users actually log into your system and interact with them. This, this instance. The HMS aggregate that any errors you might go and fix at the central level and the, the central level might be the district or it might be higher up but you might fix, fix anything that is wrong with your data. Add a note to why you did it and pass on the report. Usually data is less sensitive. This is always, well, we think that this is, at least compared to tracker data, the aggregate data is less sensitive. It's not personally identifiable, which, which makes data leak less significant, for example. And if we compare this to individual record records or tracker. There's one point I want to bring up that that might always be so well known all tracker data can can be aggregated in your DHS instance directly. There is tools for aggregating the data. There's tools for making advanced calculations. There is tools for making indicators that directly count how many HIV patients you have on treatment, for example. And you can make dashboards from your tracker data. The reporting, however, is not, it's not time boxed in the same way reporting happens more in an organic way when, when real well events happen they would get entered into the system. And it would not be based on routines necessarily maybe there is a end of day routine to enter the today's patients. The, the reporting is more directly related on real world events. And if, if there is an error, you would actually have to go and, and fix that in the individual record. If your dashboard that aggregates all your data says that you have 100,000 on treatment and you know that the number should be 100,000 one. And there is an individual error in your data and you have to go and find that single patient missing and add the missing event to his profile to fix that fix that aggregation and fix that data. It's not just the matter of updating a report your aggregate would would would not change be changeable unless you actually fix the underlying data at the lowest lowest level. Also, if, if an event comes in late, if you find a piece of paper that should have been entered a long time ago, then your best course of action might be to enter that into the system, even though it's all even though it should be entered one month ago. It's better to enter it because it might be part of it might have significance in the clinical sense. It's, it might be an important part of the history of the patient. And it might also in the. It might also affect indicators that will be calculated later. So, usually fixing the data by updating the record is always the best course of action. And this is a bit breaking with the HMS. HMS world where you would. You would not go back and change and all and already submitted report in the same way. Another big difference between the tracker and aggregate and we get to this a little bit later it is that the data is very sensitive often and it might be. It might be a personal disaster at least. And it should be a national disaster if your HIV. HIV data gets leaked into the wrong hands, for example, exposing the names of infected persons. And the amount of people that should have access is different the actual access they should have to the data is might be very different. And the number of people that can see data from multiple clinic clinics is very low. So, there's other considerations when securing both the server and setting up the configuration for the aggregate part of your tracker. Yeah, sorry for your track individual record records based compared to your aggregates. Another point to bring up as a difference is that HMS aggregate reporting is is based on on reporting guidelines and there is there is a certain established way of using this data. But when you set up a tracker, there might be other useful indicators that is operationally useful, which can be very clinically relevant can be even spawning whole new ways of using your data. That is not using the HMS aggregate. It's not going into the your monthly report necessarily, but it might be data that is very useful in monitoring and and evaluation and it might be useful clinically. And it might even be a tool for for clinicians in other work tasks or planning. So this is something not to forget when we talk about the tracker data as opposed to the aggregate and feeding into the aggregate, there might be aggregations you want to do that does not fit into the aggregates. Yeah, the last point there is just that the your system might be critical and and access to the data might might be at the different well the criticality of having your server up at all times might be different in your tracker and aggregate instance. So, I'm pulling these two together, combining tracker and aggregate. What we often sees that one challenge is that if you have an HMS and your tracker data is going to feed into your HMS. There will be a period of time where some parts of the country would would not yet have the tracker and you would need you would need the individual data from from. You would need to have still aggregate reports for those parts of the country that does not have the tracker. You would still get the paper and enter the aggregate number somewhere and in some parts of the country this might be supported by a tracker and the data might even even be fed in directly from the tracker from the tracker system. When it is possible to create a report based on tracker data directly. But it might be sometimes a good idea to to combine the data from the tracker and from non tracker sites into an aggregate form and submit them together, instead of, instead of only relying on tracker. This, this would in many cases mean that you need to have some sort of mechanism to move data from your tracker to the aggregate. And we have a lot of scripts and we have support for this in in the API's. We have knowledge about this in our community and moving tracker data into aggregate is is definitely possible. And we're also working on the roadmap on a feature for making this easier and making this part of the user interface in DHS so that you wouldn't have to have any script skills or, or, or set up server jobs to move this data over. So that is upcoming we we haven't pinned that one to release yet but it's one of the one of the first releases. And we are seeing that this should be planted to. Another impact or thing to think about is the difference between how the data is entered the HMS reporting might not play so well with with the tracker indicators or the track the indicators based on individual records. The individual records with with potentially be entered outside the time box reporting that most HMS has. So you can very well add a new track into the instance or add a new individual record after the report has been submitted and then if if you do your calculation again on your day tracker data you will see that the number has changed. And then your report is already submitted but the number is now not the same on the tracker instance as on the aggregate number. And the last consideration is the access and security and we see that in many cases tracker and aggregate has different different requirements here and they'll get a little bit into that later. But all this adds up to that usually you would have a tracker instance and an aggregate instance running side by side and and data would be moved from the tracker to the aggregate with a script or by manual means. And the tracker data would would have a more fluent life also be secured much much tighter and might have different routines for for updating for example. The hosting and security is the last point topic I will touch on with you before we go into questions and we touched a little bit on it that tracker instance might be set up on the instance for the soul is on a different server for the soul instance that the security considerations is is much stricter for a tracker usually then it is for an aggregate you would be very much more careful with your user accesses you would be the disaster would be much bigger if you would lose this database or give too much access to someone that will steal and sell the data or or leak it. So, this is also an area where very very you it's more like going from zero to one than from one to two. We on the hosting side and the other part of this that I'm going to be touching mostly on now is the is the hosting and the performance and what you can expect because this is also is always a very central question. For for you who are thinking about starting or are setting up a tracker instance. You might be wondering what can this tracker instance actually do and we have some numbers from the field to help this decision a little bit. We have the Sri Lanka covid system where they have 16 million tea eyes which tea eyes mean means persons for those who don't know that which is tracking 16 million individual individuals and with covid surveillance. The person is mentioned under there for a reason soon apparent they are running 234.4 Bangladesh is running an older 233 and they have a very big instance of 38 million tea eye and around 200 million events. That's a modern child health tracker. Then I put Ghana here and I could also put the we're gonna fast talk to go. He spends best central Africa HIV server which is around the same size. Ghana is 2500 users and 225,000 tracked tracked people. They're running 234.2 and why did I put this up between between the others. These are all servers that's pushing the boundaries of the chess. The Ghana is no exception. They are also pushing the boundaries and I'll get a little bit back to that later. We are doing an initiative that is that is also pushing the boundaries for for tracker and DHS these days and that's the covid vaccine performance test initiative. As the covid vaccine is getting rolled out in the world one of the questions that everyone needs answer this can we actually support this with tracker. Can the tracker support the volumes I need so we have done we have started an initiative and run some some tests have some numbers that I will bring to you here.  It's database which would be the existing data in our test databases is three million people 21,000 users. We also did this with a much higher number of users and org units. We started with 400,000 org units and 21,000 users across these. Orginates and we found some bottlenecks that was unintended. So in our official tests and numbers we had to reduce it to 20,000 organets but at the same time we worked on the bottleneck and it should now be fixed. So when we do a second round of testing hopefully the the number can be 400,000 organets and 400,000 users on our next next iteration. It's worth mentioning for those in the room that in this case the database and application was running on the same server and in a tracker setup it's much more common to have a database server and an application server separate. But it's this is something to mention. In this setup we were running a sustained load of between 510,000 users delaying 30 to 50 seconds on the calls which which means that it was running 40 requests per second for six hours. Well how much is 40 requests per second for six hours well the number we can put beside this to illustrate it is that it was 520,000 tract entry instances that was imported with 1.2 million events over six hours. So this is like entering 500,000 new persons into the COVID vaccine program over the course of a new working day. We were able to run it a little bit faster but then it had some deterioration when requests clumped together. So running 60, 60 requests per second would mean around 750,000 new tract entry instances over six hours. But there was some we were starting to see some problems then or starting to see the limits of the server. We can also mention that we ran a smaller server with only eight CPUs significantly smaller than the first one there and it wasn't that much slower. It was around 30% slower. So this is of course one program there and the numbers we found from the COVID vaccine performance test and this is one way of looking at the performance and the capabilities. One of the main things we got out of this was that we will be actually were able to remove a lot of bottlenecks. I mentioned one bottleneck already with that we found when we used 400,000 or units and that one was was removed. We have found other bottlenecks and fixed a lot of bottlenecks in later versions of DHS2 and the main takeaway for us and the team has been that we have been able to really improve the bottlenecks that we have found. There is a but, however, and there is a big but we cannot deny the fact that the servers are very different and the use cases are very different. And this is what we see when comparing the numbers from different countries. Also, when we set up the COVID vaccine initiative in the last point, we see that these servers are extremely, extremely different. And the unknown bottlenecks might be coming from program design, for example, and just based on how your program is set up, you might uncover a new bottleneck. Looking at the numbers from the previous slide, you might say, okay, fine, it seems like we can definitely run our program. We will have less than 750,000 new persons into the system every day. But these measurements that we have presented here is just one finding that we have made. When you install or set up or make a different setup, if you take the COVID package and change it, you might introduce a bottleneck that we didn't find in our testing. Because with these packages, you can do whatever you want. So it's important to take these numbers for what they are. They are one observation. And if you change something, the observation might be different. We are, of course, working with these bottlenecks as we find them, but you might be the one finding them if you scale up and something is different in your settings. We also know some designs that are heavy, that should be avoided at all costs. Right now, the biggest one to mention right here is the, when you make these operational dashboards, then you might get very nice numbers for your clinic.  HIV might be easy to calculate on the clinic level. So it's very useful to see for your clinicians that we have 300 people on treatment now. But this same calculation might be impossible to do on the national level because the amount of data is too high. So there might be a difference in what dashboards you can provide to the clinic versus what you can provide on a national level. And this should be going into your design. It should be going into your planning. And you shouldn't sell very complex dashboards on national level to your managers or your stakeholders. Another thing we regularly see is that custom codes and apps isn't especially especially there's an especially high risk when when you have custom scripts, apps and middleware running against the same backend, the same DHS backend. And the reason is that these might be doing calls a little bit differently. Sometimes they are not so fine tuned as the apps that is delivered with the DHS backend. So that they might be doing calls that is heavy or that at least introduces unknown bottlenecks that we have to fix in the backend code. So this is one area of caution and we have seen in some instances in some countries that some unintentional calls are very heavy. The DHS backend is very open. You can do almost what you want and you can do very heavy operations and it might not even be intended. So this custom scripts and apps is something to pay extra attention to in your setup. Another thing that is very important and this might be the most important point on the slide right now. We have spent 236 development time on fixing bottlenecks and in some instances calls are orders of magnitude faster than before in the latest versions of 234, 235 and 236. So when planning a project now, if you are not already on one of these main versions, and you're introducing a big tracker, we would encourage you to look at the plan and see if there is room for upgrading to 234, 235 or 236, the latest patch version. So these are the fastest at the moment by a long shot. Some last considerations here is the team. If you're doing an infrastructure and running a server for a tracker, you should not be training your team on doing this and also expecting to succeed the first time. Running a mission critical and heavy loaded server like a tracker server usually is, is something that you should make sure you have at least one person in your team that has done before at large scale, maybe with different software but that has run such a big instance before. The requirements for your infrastructure is so much higher and you shouldn't train your team and set up a tracker at the same time. That red guy there should be working for you, he should not, you should not only start this if all of your team is the blue, blue persons are learning there. You have to set up monitoring and make sure to follow the server health. You should have mechanisms that warns you if there is trouble on the way. And these will help us if you run into one of the unknown bottlenecks. We have a team that is, that is often responding to servers that runs into these bottlenecks, known or unknown bottlenecks in the field. And if we're going to help what we need is server monitoring. And if you're going to find the problem before they, they, they affect your users and before they cause downtime and a part of your country is not able to do their job. You also need the monitoring. Another thing so that's easy to forget until until you needed it yesterday is disaster recovery plan. Do you have a plan for what happens. If the server goes down, do you have a plan for what happens. If you need to go back in and and load an old database. Did you test that the restore actually works when doing a tracker system. Then there is suddenly a need for disaster recovery plan which is a well known term and this should be well known to anyone who, if you have that red guy on your team, the teacher, he should be telling you this. But the red guy there should tell you is that you need to do a risk assessment for your, for your server and for your, for your application and and a risk assessment matrix might look like this. Where you would, you would go through every risk you can think of, you would scale them on this sort of on this sort of a chart. And he would work on the high risk high impact points first you would try to management them, then he would start working on the, the one studies, either medium in the yellow zone there, and then you will work on the green zone to where you have a low, low impact or low risk and you will try to manage your risks and try to make sure that you, you have managed them and thought about what to do. And then the disaster strikes this, this, this chart is what shows the world that you did your job. And yeah, and these are all topics that we'll get, get more into on the level two academy that's coming up. All right. That's everything I was going to go through we have some, we have some resources at the end there. I guess the slides will be shared with you all. And you would be able to, to click the links. And this is some of the resources we mentioned. There's also an registration for the academy that's coming up. I don't know if it's any spots left there, but Mike and Martin can probably tell us. I guess we're with that we're going to go into the, the last part of the session, which is the little bit of questions and answers. And maybe, yeah, maybe to jump in then so we, with, with such a large group of course it's it's a little difficult to do an active question answers there has been a lot going on in the community of practice so please take a look at the link that Alice has shared in the chat to see some of those. And we're still actively answering some of the more technical questions there, there are people on our side.  I thought maybe I would go through a couple of the responses that are there in the community of practice but also if you raise your your hand here or try to speak up we may be able to field a couple of live questions we have maybe 10 minutes or so to try to address some of these. But I'll start by by maybe going back to some of the more general questions that are in the community of practice. So one of them was about kind of the the structure of configuring your program so it was recognizing that tracker is well suited for for clinical services that have a schedule, such as immunization or anti natal care where you know kind of the purpose of each visit ahead of time you have a bit of a checklist for the clinical person to walk through. But the question was, is it is tracker also well suited for for health services where it is less of a checklist less of a schedule. I wanted to take the chance maybe to talk a little bit about how we see tracker fitting into kind of the concept of an EMR and how you would you would handle some of these more kind of ad hoc services or unknown services so what you can do with tracker is you can start with a very simple or lightweight program for tracker, you can be dedicated to a single service being provided at a health facility. You could set up for example, simply an ART register, but that same clinic is probably offering many other health services, maybe they are also seeing TV patients maybe they are also diagnosing malaria. One of the things that makes tracker really suitable for for this level of clinic is that you can add programs over time. And the programs themselves are linked because they share kind of the pool of clients or patients where if you, you know, you have in your catchment area, people that have come in for services. You can identify that individual, you can enroll them into a new program. And that this we've seen for many countries, this matches kind of their their the funding project cycle, where, you know, perhaps this year there's emphasis on building out the HIV program. Perhaps next year there's funding for TV, and you could simply add the TV program when you're ready, your users are already familiar with with the program because they've been using it for HIV the interface is the same but now they also have a TV program. So in in this way, many countries have kind of built out within tracker, a complete or comprehensive overview of all of the services that are being offered at at least one level of clinics. So they would think of it as kind of their entire primary health strategy. And this usually once you get to that point, you probably are adding in a non disease person specific program. So perhaps you have a lab program where all of the possible lab services are registered in that program. The contract entity can be the person can be coming from HIV or they can be coming from malaria, but you register the need for the lab services in this additional program and the information is related back, or you might have a generic kind of program for receiving clients or receiving patients, where you don't know why they've come yet, you don't know what service you're going to provide but you register them as an entity they now are kind of in the tracker system. And once you identify a diagnosis or you want to provide services, you can enroll them from there into a more dedicated program. So it's it's a very flexible kind of data model. We haven't designed tracker to try to be a full scale EMR for something like a hospital to use. It gets a lot more complex in terms of trying to link all of these different services together. But we also think that for many of the kind of levels of the country that we're trying to reach the EMR is probably not appropriate for the level that you're capturing some of the primary services data, or some of these kind of key health program data. There are many I think flexible ways of trying to configure tracker to allow you to cover a whole range of services that the person registered in tracker can be shared across programs. And we're working on extending our analytics model to allow for more sophisticated analysis across these programs, so that you can start to have better insights into, you know, TV patients that are also in different and then are diagnosed with malaria. I mean you can you can start to think of many different ways of approaching different subgroups and populations based on how they are enrolled across programs in your tracker system. So that was one that I wanted to share. There, there are a couple of more I think technical kind of questions specific to how to set up your system if you want a certain type of analytics, whether it's line listing or carrying a patient over from one program to the next. Again, we're answering those more technical questions in the community so please take a look. And that's my, my additional thing that I would say is that these are exactly the kinds of considerations that you want when designing your program. So it's it's very important to take a deep dive into what are your analytic outputs what do you need to be able to show in your outcomes, and then design the program that is related to that that allows you to do the kinds of analytics that you do. So if you're, if you're new to trying to think through what tracker configuration you would use. I think it'd be really valuable to take a look at some of the questions that are coming up here in the community of practice so that you can see kind of what it means to decide whether something is an attribute versus a data element or what it means for analytics if you do repeatable stages versus a single stage so these are these all have an impact on the types of analytics that you can do. And it's important to know ahead of time that you'll be able to get the data that you want based on the way that you've set up your program. Another question that was in here someone generally was about MDM mobile device management if you're using DHS to Android DHS to we don't have our own mobile device management solution. This is software that allows you to kind of have a comprehensive look and control over all of your Android devices. There are many existing MDM solutions that are out there. Some are paid some are free. We've written up guidance as far as we can recommend and the DHS to Android implementation guide. I think we generally do recommend that if you have a large number of Android users if you're providing the hardware that you really should have a mobile device management solution. This allows you to push updates to these devices or to identify where they are or who is the user that is logged in or when is the last time that they submitted data. So it gives you a lot more control over kind of your fleet of users and the devices that they have.  So generally we would say that it's important to consider the mobile device management. There are a number of platforms that are available to use. Many countries have done this already successfully using DHS to and the Android solution. So I've put a link in the community practice to some of the documentation that we provide there. Maybe I'm just glancing now to see if there are any hands raised anybody that wanted to ask a question. Maybe I'll give you a moment to think if you want to speak up and ask. I don't see any hands right now Mike, but if anyone wants to take on reactions, and then the race and icon, if you want to get the mic. I'm just also scrolling through the questions that have come in on the community of practice see if there are any others that we can spend some time on. So some of these are a bit specific so maybe the best thing to do then if I if we don't have any questions to ask right now. Maybe we'll close the session out will keep referring back to the questions that have been asked in the community of practice. Again, you can continue to post some questions there after the webinar as well. Feel free will monitor that thread in the community practice and we'll answer more. This webinar recording will also be made available so that you can share with colleagues and so that others can view it. We also will be sharing the slides that we've used. Also we'll be giving to you with all of the links that we mentioned. And maybe at this point I will now turn it over to Alice to close us out with a couple of thoughts about the annual conference that's upcoming and maybe the tracker implementation Academy. Thank you so much Mike. Yes, so I have posted a few information in the chat about the annual conference. So it's basically the largest events of the year, gathering. We've had more than 900 participants. So during five days from 21 June to 25 June this year, the entire community will be gathered to share experiences around the highest tool and learn more about the new features. I posted the link in the in the chat. So do not hesitate to click on the link to read more about the annual conference and why not register. Now, yes, we also have the tracker implementation Academy that will be hosted from 25 May to 4 June. Most of you actually have already registered to the Academy but for those who haven't, the link to the registration form is still live. So do not hesitate to go on the DHIS2 website, the Academy webpage and you will be able to find the tracker implementation Academy link for registration. That's it. Thank you so much. Great. Thank you Alice. Thank you everybody for participating and we look forward to seeing some of you in the Level 2 Academy.


The dynamical network marker, which predicts sudden changes in foreign exchange markets with high accuracy, can be extended to other scientific, technological, and societal contexts using recurrence plots and delay coordinates. This article was authored by Shansuke Weir, Kozuyuki Ihara, and Yoshito Hiota.