Welcome back everyone. So we are learning the exciting journey of the early church. We saw how amazing things were taking place in the church but at that point the apostles were caught and put on trial. So let's observe the various things that take place during this trial. We would need to read from verse 22 of Acts chapter 5 till verse 32. I request one of us to go ahead and read this verse. Acts chapter 5 verse 22. But when the officers came and did not find them in the prison they returned and reported saying, indeed we found the prison shut securely and the guards standing outside before the doors but when we opened them we found no one inside. Now when the high priest, the captain of the temple and the chief priest heard these things they wondered what the outcome would be. So one came and told them saying, look the men whom you put in the prison are standing in the temple and teaching the people. Then the captain went with the officers and brought them without violence for they feared the people lest they should be stoned and when they had brought them they said them before the council and the high priest asked them saying, did we not strictly command you not to teach in this name and look you have filled Jerusalem with your doctrine and intent to bring this man's blood on us. But Peter and the other apostles answered and said, we ought to obey God rather than men.  Him God has exalted to his right hand to be prince and savior to give repentance to Israel and forgiveness of sins. And we are his witnesses to these things and so also is the Holy Spirit whom God has given to those who obey him. So we have now seen that the apostles follow the instructions of the angel and they go to the temple and they start to share about God and preach about the matters of salvation.  So over here one more thing to notice that something unusual has taken place isn't it? Because we are being told that though these prison authorities went to look for the men they found the prison shut securely and the guards standing. So something very unusual, what did we see earlier? By night the angel of the Lord opened the prison doors and brought them out.  It's sounding like a miraculous occurrence has taken place and the guards didn't even know about it. Something supernatural which our minds cannot grasp. So they have come out and the guards are still standing outside before the doors. But when they opened they did not find anybody inside. I don't think this is something that happened only at that time and it doesn't take place anymore. I have heard one testimony of Sadhu Sukhdar Singh who was thrown into a well and the well was covered. The well had already some outcasts. Some people that the authorities had condemned to death. They were thrown into the well and they were already dead inside the well. So such a well they had thrown him and shut the well and locked it up. But there is a story somewhat similar like angelic intervention where there was a rope that came through and he had no idea. He was just happy that there is a rope and so he held the rope and lifted himself up through that rope. The rope was taken out of the well and he got free. When they found Sadhu Sukhdar Singh outside they were wondering about who set him free. But the story goes something like the well was still locked. And to everyone's surprise the key was the person in charge, the highest authority who was questioning. He was the one who had the key on his belt. So they were all amazed at how this man could come out of the prison when the key is in the most secure place with the highest authority. But God has his way of setting his people free. So something unusual has taken place to deliver the apostles and they have come out. But they followed the instruction of the angel and wherever they they went to the temple and there they were caught. But look at the confidence of these men. They are back at the temple and they are teaching the people the same thing that they were asked not to talk about. Remember the last time they were they were warned in Acts chapter 4 they were told don't preach in this name. Don't use the name of Jesus. But here they are only proclaiming the name of Jesus once again. Then what happens? So they get a hold of these people. They bring them back and they question them again and they ask them, didn't we strictly command you not to teach in this name? Why are they so worried about the name of Jesus? Because not very far before Jesus was under trial and Jesus the person the individual Jesus because of him the community or the society was at least the power that the authorities expected from the communities the support of the communities it was divided because somewhere in support of Jesus somewhere not in support of Jesus. Now again bringing the name of this man will affect their positions and their power. So that is their biggest issue. It's more of a political reason why they are after the apostles now we don't know but they were so worried about the miracles and the spiritual aspects of what was being taught. Maybe the religious leaders we read about the high priest who had a problem. So maybe they had an issue regarding the religious matters but it was more because of political reasons. So now they brought them back again and they are questioning them. They are telling them that you did not heed our warning. Why are you doing this again? You're teaching in the same name, the name of Jesus. So look at the boldness of Peter. Who is Peter? Fisherman. So the fisherman is answering all these members of the council. We have the high priest. We have the sector of the Sadducees. So they're all very well established educated people who are part of the council. And the fisherman is boldly responding to that. Even earlier we saw Peter and John. They were with Jesus which gave them that boldness and that courage. And even now though it seems like there is this, you know, like these levels where the people who are questioning are at a higher level of education and everything. That is not what mattered in that moment. We are not looking down upon education. You know, it's great education is great. We should all be educated. But in this situation, Peter is coming from another background. So he is, you know, an untrained person and an equipped person in terms of education. And even then he's so bold before the authorities and he responds to them. He says, we ought to obey God rather than men. So that is his response. I think Rosalind asked us a question a little bit before the previous session about the boss asking somebody to lie. So this is the response Rosalind. The way Peter responded, he said that he will only do the right thing, which is to continue to preach in the name of Jesus. So, yes, of course, we can do it in a nice way. But in this situation, there was some amount of direct response which was required. Please give me a moment. Please excuse me. Yes, thank you. So Peter boldly responds because there was a need for him to stand up for what is correct. And he says, we ought to obey God rather than men. So when the authorities are asking us to do something against God, that's the time we have to stand up. Isn't it? But we have other passages of scripture, you know, when Paul writes to the Romans, when Paul writes to Timothy. He says, you must pray for the authorities. You must uphold the authorities in prayer. Yes, all that is true. We must uphold them in prayer. We must, you know, uphold the law of the land. But at times when we are asked to go against God, that's the time when we have to stand up for God. And we find that Peter is doing that. Another parallel incident would be at the time when Daniel and his friends were asked to bow down to idols and they never did that. In fact, the friends of Daniel, they went into the fire for doing the right thing. They stood up for what was right. And God always rescues his people when they are facing moments of persecution for doing what is right. So Peter was standing up for the right thing. And then, you know, he is boldly speaking about Jesus.  Him, God has exalted to his right hand to be prince and savior, to give repentance to Israel and forgiveness of sins. And we are his witnesses to these things. And so also is the Holy Spirit whom God has given to those who obey Him. So it's very doctrinally rich. These few lines, we can even look at this as Peter is providing a defense for yourself. But he's utilizing the opportunity to proclaim the gospel. Why is he stating these things? Because he is establishing the Lord Jesus as the Messiah. When he says, Prince, savior, to give repentance to Israel. That's what they were waiting for. Somebody to come, the Messiah to come and deliver for give their sins. So he is proclaiming Jesus as the Messiah. This is another lesson for us. Every opportunity that we have in this situation, in the prison or in trial. Peter is making use of that opportunity as well to talk about Jesus and say that Jesus is the only way to repent. He has come for the forgiveness of your sins. And he says that we are witnesses to these things. And he also talks about the Holy Spirit whom God has given. And so he is presenting the faith that he himself is following. Now something like this reminds me of a man called Richard Wumbert. I don't know how many of you have read his life story. He was in a communist culture back in those days in Romania when times were very difficult. And those who believed in God were imprisoned. So he lived in such a time. And he proclaimed once he got born again, his life was transformed. Before he came to know the Lord, he lived a very morally corrupt sort of a life. And his lifestyle was not good. But once he was saved, his life changed. And that entire book in God's underground, it speaks about how he continued to proclaim the salvation. Salvation message wherever he went. So he was imprisoned. And for the most part, it talks about how he was in the prison. And even in the prison with the prisoners and the guards, he would share the gospel to the best that he could. Going through all the sufferings that he was going through physically, emotionally. He would still proclaim Jesus as the Savior. And one incident where he wanted to share the gospel with the person in the next prison. So the way he did it is thankfully he had memorized scriptures. And so he used Morse code, you know, by tapping to communicate the word of God to people in the other prisons. So amazing stories about people in their tough times. Still living for the gospel, for proclaiming the truth of Jesus bringing us salvation. To get this message out to everyone. And that's very similar what Peter is doing here. He's in a difficult time where his life is at stake. But what is important to him? The call of God on his life and the gospel. So to the authorities, he's talking about the Lord Jesus as the Messiah, the Prince and Savior. He's proclaiming Jesus to the authorities. It's amazing, amazing to see the boldness and the passion for the gospel. Now let's move on. So when all this is happening, there is a defense that Peter is providing. What would be the conclusion of this trial session? So from verse 33, let's go ahead. We will read till verse 42. Could somebody please help us? Gamelio's Advice. When they had this, they were furious and plotted to kill them. Then one in the council stood up, a Pharisee named Gamelio. Gamelini, something like that. And the teacher of the law held in respect by all the people and commanded them to put the apostles outside for a little while. He said to them, men of Israel, take heed to yourselves. What do you intend to do regarding these men? For some time ago, Theodias rose up claiming to be somebody. A number of men, about 400, joined him. He was slain and all who obeyed him were scattered and came to nothing. After this, Theodias of Galilee rose up in the days of the censuses and drew away many people after him. He also perished and all who obeyed him were dispatched. And now I say to you, keep away from those men, keep away from these men and let them alone. For if this plan, all this work is of men, it will come to nothing. But if it is of God, you cannot overthrow it least. You even be found to fight against God. And they all agreed with him and when they had called for the apostles and beaten them, they commanded them. They commanded that they should not speak in the name of Jesus and let them go. So they departed from the presence of the council, rejoicing that they were counted worthy.  And dated in the temple and in every house, they did not cease teaching and preaching Jesus as the Christ. Amen. Thank you, Lubega, for reading this section. From verse 33, we have heard what Peter had to say regarding Jesus and the response that he got from the council was anger. They were furious, it says, and to the point where they plotted to kill them. So now the situation has gone from bad to worse and the apostles are really in trouble. But in the midst of what's going on, God can still intervene. The prophet says the heart of the king is in the hands of the Lord and he can move it whichever way he wants.  Now Gamaliel had high esteem and good reputation among all the people. Because he had a title known as Rabban and Rabban is even above the title. We know Rabbi, we use that for a teacher. But Rabban is much higher than Rabbi and it means our teacher. Many wonderful things about Gamaliel. He was the grandson of an esteemed person known as Hillel who was the founder of Israel's strongest school of religion. So this man is not an ordinary man. Chappat is working through a very powerful personality at this point. And later on, we will understand from Apostle Paul that Apostle Paul was trained under Gamaliel. So which means that Apostle Paul had some of the best training in his past life as an unbeliever. So Gamaliel is introduced at this point a really influential Pharisee and Gamaliel has a suggestion. What is that? So Gamaliel, he commanded to put the Apostles outside so that the council can have a discussion. Now who is part of the council? Many different groups of people. You have the high priest, you have the Pharisees, teachers, you have the Sadducees. You have different people who all form the council. So now the council is having a discussion to come up with a solution for this matter. These people are speaking about Jesus and they don't seem to heed our warning. This might turn into a bad political situation for us. What do we do with these men? So that's the question that they want to address. So as they begin to address, Gamaliel takes them back to an example of a man called Theodos. So Theodos, apparently he was a man who had a following of people. A number of men, about 400, joined him and he was something like a rebel. And he misled people and after that there is another example that he brings up. Judas of Galilee who rose up in the days of census and drew away many people after him. He also perished. So basically he says both these examples of Theodos and Judas, they were men who tried to get the support of the people but both of them failed. Now they perished. It didn't work out. Their mutiny or their rebellion, it did not really work out. So the point that he is trying to make is that if these men are also going to cause a rebellion, it won't work out. Don't worry. But look at this. It's coming from Gamaliel's heart. Now it's amazing because a fisherman just spoke with minimal training. But Gamaliel, who is a highly educated man, he adds another point. He says if these men are trying to create a rebellion, it will cease like the others. That was 39. We don't know why he said what he said. So what is the statement that Gamaliel made? He said, but if it is of God, you cannot overthrow it. Lest you even be found to fight against God. Wow! What was Gamaliel thinking? Was he beginning to be convicted about the Lord Jesus? The Lord Jesus being the Messiah? We don't know what was going on in his heart. But an advice came through the most unlikely person at that time. The greatest man of influence. What is he saying? If it's a human effort or a human activity, it will cease. But if what is going on is of God, don't put yourself against God. You cannot overthrow it. Lest you even be found to fight against God. Wow! Amazing! So, we understand today that what started 2,000 years ago, so many people have tried to stop it. And we can read the incident after incident in history. We can read about the dark ages. We can read about the times of persecution. And in fact, even today in the world, there are pockets in the globe where so many things are going on against the faith, against Jesus. People are struggling, people are suffering, people are being persecuted. But the Gospel has not ceased to save people. The work of God has not stopped. It's been 2,000 years, roughly, since Gamaliel made the statement. He said, you can't stop God. He's right. Nobody could stop God. And nobody can stop God. So what Gamaliel said then is so applicable even today. If there is something that God is doing, no man can stop it. His work, Job says, who can do what his work? He's the God Almighty. And that was the advice of Gamaliel to the people. He said, hey, everyone, calm down. If what these people are doing is their own, it's going to cease. But if it is of God, then we cannot stop it. So don't go and put yourself against the mighty ocean wave and try to stop it. You just cannot hold out in the power of the wave. And so from verse 40, again, this is all supernatural, verse 40 says, and they agreed with him. How did they even agree to such a statement? But they agreed. The council agreed. And when they had called for the apostles and beaten them. See, they haven't changed their original attitude. They had warned them. They had threatened them. Now they're trying some physical harm and physical discipline and they beat them up. Okay, hoping if the warning doesn't, if the threatening doesn't get to them, the beating will. So let's try. So they beat them up and they commanded them again. Earlier, what did they say? Don't speak in this name. Again, the same point is being emphasized. You should not speak in the name of Jesus. Give them this instruction and let them go. Okay, let's see what happens, you know, after this. And so now the apostles, they departed from the presence of the council and noticed their attitude. Wow, excellent attitude. Verse 41. Second part says, Rejoicing that they were counted worthy to suffer shame for his name. So sometimes for the name of Jesus that we carry, we bear, we may need to suffer shame. It's happened throughout history. Men and women who stood up for God have faced opposition in various seasons of their lives. But what makes their testimony special? What does it say? Rejoicing that they were counted worthy to suffer shame for his name. So the love that these people carry for God, that even death or persecution or pain, think about this. This time it was not just warning. Basically they are hurting. They are in pain. They were beaten. And when we talk about the, you know, the beatings that are happening under, you know, the governors and the authorities, it would not have been just, you know, like an ordinary tap and then send them back. It wasn't, they were really beaten badly. So they are in pain. And they've gone through a lot because of the trial and being in the prison and so much has happened. But what is their response? They just thought, wow, we had an opportunity to suffer shame for the name of Jesus. We are so blessed, right? I don't know. I don't know how people can even think like this, but that's how these people thought when they suffered shame for the name of Jesus. So their lives had been impacted. Their lives had been so, you know, powerfully touched by the gospel that they were ready for anything, to face anything. And that's the kind of people that we see in the Book of Acts. So first of all, they are happy that they went through persecution and they're continuing their works. Nothing is hindering them, verse 42. And daily in the temple and in every house, they did not cease teaching and preaching. It's like, yeah, and you know, life goes on. It's as simple as that. But it is made to sound very simple, but we understand that it was not simple. But it only goes to show what had happened to their lives because of salvation that they received from Christ and the baptism in the Holy Spirit. So let me pause for a minute and if there's anything you want to talk about, let's talk about that and then we'll get into chapter 6. We're not saying that, you know, we must invite persecution so that we feel a sense of being loyal to Christ. That's not the point. When persecution happens, okay, maybe we've done it all right and then we find ourselves in a position of being persecuted. That is an honourable thing to go through and that's the attitude that we find among the apostles as well. So they did the right thing, but they got into trouble and at that point, they said, okay, we got this opportunity to suffer for the name of Jesus, we are rejoicing. So just wanted to add that point. So don't go looking for persecution. Any other comments before we proceed? There are many stories of the lives of people that we can actually read who lived in difficult times and who stood up for Christ who had that, it's like saying, I pledge allegiance to the Lamb, you all probably have heard that song. But they lived like that, that even if it means life or death, I'm living for Jesus, okay. So much for us to learn from the people who have gone ahead of us. I know that there are still places where people are facing such circumstances, but also we understand that things are so much more comfortable now. Yeah, technology, sometimes it's so easy to share the gospel, but then even those opportunities somewhere we hesitate to take it up. But to always remember that people have paid a great price for us to have what we have today. And because there are some people who stood up for the truth back in those years, we can, we now have, we have our Bibles, we have inspiring life stories because somebody has stood up. Okay, so let's not forget the way people were ready to suffer for the gospel. So shall we just move on to chapter 6 then? All right, great. So let's read about what is happening in the church. We talked about the rising persecution till now and now let's look a little bit inside the church. What's happening within the community. Even chapter 5 started with something that was happening within the community, right? The heightened presence of God, the mighty work of the spirit and sin showed up. And so there was judgment, right? And quicker judgment. So whenever we experience a revival, it's a given that, you know, sort of judgment will be seen and a quick judgment like the one that we saw in the case of Ananias and Sapphira. So in Acts chapter 6, there is a little bit of a crisis in the church which the apostles have to solve. So let's read from verse 1 all the way till verse 7. Acts 6 verses 1 to 7. Now in those days when the number of the disciples was multiplying, there arose a complaint against the Hebrews by the Hellenists because their widows were neglected in the daily distributions. Then there was someone, the multitude of the disciples in state. It is not desirable that we should leave the Word of God and serve the disciples. Therefore brethren, seek out from among you seven men of good reputation full of the Holy Spirit and wisdom whom we may appoint over this business. But we will give ourselves continually to prayer and to the ministry of the Word. And the saints pleased the whole multitude and they chose Stefan and men full of faith and the Holy Spirit and Pilate, Porosius, Niccardo, Timon, Paramanus and Nicholas Procelled it from Antioch whom they said before the apostles and when they had prayed they laid hands on them. Then the Word of God spread and the number of the disciples multiplied greatly in Jerusalem in a great many of the priests were obedient to the faith. Thank you Thank you Azali. Now within the church we earlier said it's a growing church. It's a thriving church spiritually in numbers and in high esteem among the people. So at this time when the number of disciples was multiplying the growth is continuing there arose a complaint. So as I said earlier it's not like this church we can have a very rosy picture of the early church but at the end of the day they were people and so there were issues in the church but the point is not so much that there are issues in the church but the focus should be on how to resolve those issues depending on the communities that we serve as pastors, as leaders problems will happen. So recognize the problem and find the solution. So this section is the beautiful way in which the apostles solve the problem. So Luke is stating the matter in the church which is simply regarding food distribution. So there was a complaint against the Hebrews. Who are the Hebrews? They are the Hebrews speaking Jews. By the Hellenists who are the Hellenists? They are the Greeks so among two communities there are two language speaking communities and they have an issue. So the Hellenists are saying that their widows were neglected in the daily distribution. Remember we saw earlier that those who had need there was a supply. So the admin office is working very well Church of Jerusalem administration is nice. They have their rosters, the food is being served, money is coming in, the food is being bought nicely given to the people. All that is going on beautifully but someone is noticing that the people who speak this language they are not giving the food to the people who speak the other language. So that's the last thing that should come to your mind. It is a real issue which needs to be addressed. Maybe it was not intentional maybe it just happened. We don't know why it really happened. Then the 12th someone in the multitude of the disciples. So what does it mean? It means that they are addressing the issue. So when there is an issue what is the task of the leadership? They are supposed to address the issue sufficiently by the wisdom of the Holy Spirit. So they are doing that and that is what we must learn from. So they are calling the disciples and in fighting the solution to the problem they are saying this problem has to be solved. However we cannot get into this role of serving food because we have other tasks which have been given to us by God and what is that task? They put it this way. The word of God. We cannot leave the word of God and serve tables. So now for people who are in leadership we ask the question oh that means that I must never serve tables. Serve tables is a way of describing the ministry of help where we could be handling the logistics the operational aspects of things that are going on. See Luke is not saying that operations are not so godly as compared to preaching the word of God. It is not like that. Every responsibility that God gives us is to worship and we honour it. But in the case of the Apostles one of the primary responsibility which they had was to uphold the word of God. So it is their primary role. Now if they give time to other activities they may not have sufficient time for the word of God where they need to really meditate and study and teach the people the word of God which was most important. So basically this is an issue of prioritization and they are saying we cannot give our time. We are not in a position where we can give our time which is meant for the word to other responsibilities. So other responsibilities need to be addressed. So how do we do this? We cannot give our time because we have to focus on the word so then they come up with a solution. Let's just find other people who can give their time in the operational aspects. So as those who are called in the positions of a pastor teacher this is also a reminder for us that when God calls us to prioritize the word of God teaching of the word of God we must not give it second place. I know it's really hard because there is so much work to do in the church we are running around, we have to sort out this, sort out that and study the word and teach the word but eventually a wise thing to do will be to delegate a wise thing to do will be to develop a structure where we can do everything. So slowly how about find good people of God according to the call of God on their lives and then start to hand off certain responsibility and guide them, instruct them how it needs to be done that way we can dedicate our time more into the doctrine and the word of God. So let's not get this understanding that serving tables is less than the word of God not at all in fact sometimes even if we are called to the ministry of the word we need to step into other tasks and take care of other tasks because if everything is worshiping God then let's move on. So now they have come to the conclusion that they must have this task delegated now who do they delegate the task to we are told that there were seven men they seek out from among them seven men with a criteria there is a list they must have good reputation they must be full of the Holy Spirit and wisdom whom they want to appoint over this business now wait a minute we want people to serve food why do you need people with good reputation why do you need them to be filled with the Holy Spirit to serve food you know this is how we look at it but when it comes to serving God even in the early church for a small role we were careful to pick people with a good testimony and testimony matters so that's why when we equip our young people we equip our volunteers we train them in all these things and we say look all these things matter in the early church the bosses looked for men with a good reputation full of the Holy Spirit and wisdom all this shows the kind of heart they carry for God and once we are clear that they are devoted to God and they are you know devoted to the church then you hand over any responsibility whether a small responsibility or a big responsibility so they are identifying devoted men that was the point devoted men, dedicated men, committed men men who are given to the purposes of God such men even to serve the food and they decided that they will give themselves continually to prayer and the ministry of the word so in a church setting as the church grows this is how it should be so the pastors the teachers they should be able to give more of their type for the word and prayer and delegate other responsibilities in a proper way in an excellent way to the right people so and what is one more thing beautiful thing verse 5 with us and the same pleased the whole multitude so it also shows godly leadership where they are not imposing themselves the apostles didn't go and say I will tell you what we have to do we will have these seven men they will serve the food and this is what will happen amen so that's not how they did it they shared this with the people and when the decision comes by the Holy Spirit notice how verse 5 says everyone is happy with it the others were pleased the same pleased the whole multitude so the others also were happy with it okay now please don't misunderstand this as when we make godly decisions every time there will be a positive response from the people no maybe sometimes we are making a godly decision but that is not being accepted by the people those are moments when we have to stand up for what god is saying we must really learn to communicate in a positive way so that what god is doing is acceptable by the people during that time however in general when we are sharing the wisdom of god with the people and it is at the right time the right decision is being made we notice that generally there is an acceptance for what is being done so the people were happy with this decision and then they go and pick up men who have this criteria fulfilled so one person's name is mentioned Stephen and a little more is spoken about Stephen and what a man is Stephen we will see in scripture that there is not so much spoken about him we will read about him in Acts chapter 6 we will read about him in Acts chapter 7 and that is about it but the devotion that he has for God it is life impacting and Luke is adding already we have seen the criteria for the people who need to be chosen but Luke is adding to the nature of Stephen and he says a man full of faith and the Holy Spirit so today if people were to describe us or if Luke had to make a mention of us while talking about the church what would he have said about us about Stephen he said a man full of faith and the Holy Spirit so these are all really good things to think about and there were people in the early church who were full of faith and the Holy Spirit and then there is a whole list of all the other names and it also says a proselyte from Antioch Nicholas a proselyte is a person who is from another religion who has come to believe in Jesus so we call them as proselytes so you might see this word here and there in the book of Acts so they picked all these people and they sent them before the apostles and when they had prayed they laid hands on them you have seen that earlier and laid hands on them means that they were seized or they were caught but in this situation they were commissioned by the apostles to go and do the food distribution now when the problem was solved in the church what does verse 7 say the word of God spread and the number of disciples multiplied greatly in Jerusalem and a great many of the priests were obedient to the faith so whenever we handle issues address issues in the right way within the church God's work can come to an end what if they had not solved this problem what if the apostles had said come to me with a question theological question please don't come to me with food problems but they did not treat a natural problem in a light way they solved it and praise God for verse 7 it says then the word of God spread and the number of disciples multiplied greatly in Jerusalem so and those of us who are leaders whatever capacity no issues a small issue in the house of God we need to deal with it in the right way and that will also help in extending the kingdom of God so we are just going to stop here and I will request Jafina is it possible to pray Jafina please pray we will wrap up for today Jai Havenday Father we come to you under the name of Jesus thank you for this day we thank you for the class that we have so we thank you for your Holy Spirit Jesus we thank you for the power of the Holy Spirit thank you for dwelling in us Lord the cross that you took for us you went up to heaven so that we can have this Holy Spirit so that we can walk in authority so that we can use your name and live mightily down here on this earth Jesus God I just pray for all my classmates and customers over here thank you for the teachings that we have learned God I pray that you help us to apply it in our life where we walk in the Holy Spirit and everything that we do Jesus where we solve the issues in the right way so that the word of God will keep growing among the places where we are serving for you Jesus we give you all the glory and honor builders of acupous Jesus as we are learning the subject to move under your acts to move under your authority your spirit and your guidance in Jesus name we pray thank you thank you everyone God bless you have a wonderful weekend ahead see you all in the next class bye for now


Welcome everybody. I see we have 32 participants in so far and two people sitting in the waiting room. Hopefully they'll be let in soon. The session today entitled scalability and performance tuning and we only have an hour so we're going to touch quite briefly on on very few issues I guess in what's really very broad topic. I'm fortunate today to be joined by Gintari and Daniel and Jason and Dan. I'll explain a little bit what they're going to present in the next slide. So I've just got a couple of introductory slides framing a few problems. Gintari is going to show us a little bit about some of the work that's been done back at base at University of Oslo around performance testing. This is work that's been inspired I guess by the huge challenge that was faced with moving into the COVID-19 response particularly vaccination campaigns where numbers got really really big. Lots of interesting developments there in terms of how we have started to be able to test more realistically more repeatably and the like. Daniel who's actually joined recently part-time assisting me with with sort of country service support. Daniel's working with a company called Solid Lines. He's got really great experience working with a very large databases with PSI and like. He's done in the last month or so a bit of a comparison around using Postgres 13 and Postgres 10 some of the results of which are kind of surprising but interesting I think for for people who care about their databases. Jason who I'm going to go back some years now. Jason Phillips from Hisp South Africa. Again long and battered experience working with Hisp South Africa managing quite a number of or quite large DHS2 instances. And then we're going to leave a bit of space I hope at the end towards open discussion. I know there's lots of lots of questions and interest around this topic and we're fortunate to be joined by Dan Kokos as well from BAO who will ask lots of difficult questions too as well. So I don't want to take up too much time. I want to leave time for everybody else but some of the challenges as I see it is I think probably one of the biggest challenges that the DHS2 is deployed in so many different ways right there are people running from in the basement as we say to in the cloud I mean there are some places where where there's a service sitting somewhere in a room in some ministry and the DHS2 is deployed on it and somehow or other they've got it onto the internet. And there are lots of particular challenges around managing a setup like that. Increasingly we we see more and more within countries that there's a government national data center which provides the enterprise service across ministries and DHS2 is deployed somewhere within that. Very common model is using cloud VPS systems. LinNode historically been very strong still very widely used. We've got good connections with LinNode. AWS used by great numbers of people. That should be Contabo not Conalbo. Contabo is a well recent player. It's proved to be very very attractive large because of cost. Very very good value cloud VPS solution. Deadly service has been used in a couple of countries as well. Through to the whole software as a service model which I think there likes a BAO, Blue Square, Hisp, South Africa I guess to a extent of offering software as a service type of offerings. And there's advantages and disadvantages and different kinds of challenges involved in all of that spectrum. The other challenge I guess we find is that there are very very different levels of skill and experience of the the the poor people who end up with the responsibility of having to host their DHS2 systems and often scale them into places where they haven't been before. Transitioning from managing quite small to large aggregate instances which has been the kind of skill set that people have built up over the last 10, 15 years or so to suddenly moving to managing very large tracker implementations. And where we've seen this really kind of bursting the through the ceilings is on the vaccination campaigns. Been looking at Nigeria for example the last few days. I mean then 250 million Nigerians and if you're going to vaccinate a reasonable proportion of that and track them all in tracker we start talking about very very large systems. DHS2 software as those of us who've been in the breach trying to manage it will realise is sometimes a very unruly donkey. It doesn't always do behave quite the way we expect it to do. The rapid increase in scale that we see often exposes what were I guess previously undiscovered performance issues. And so managing a system like HS2 is not just routine monitoring. There's a lot of troubleshooting involved as well. Particularly as we start stretching the boundaries of things. Sample of the kind of issues which is probably fairly random there are many others. Traditionally analytics generation time has been the big performance thing right. People managing big aggregate databases and they start the analytics job at midnight and hope that it'll be finished by five in the morning before the users start getting in. That's still a challenge in a number of places. There are still very large aggregate instances. For people who don't realise it skipping zeros in analytics tables has been a really major breakthrough in improving the time that takes to get through. The problem of zeros in databases I think could be the topic of a whole discussion on its own. We don't have time to get into it now. The other thing that we find and you find this because we're monitoring there's often you'll find that different API calls API calls cause different types of distress and the kind of distress that we saw a lot the last two three years is very very high Tomcat CPU usage associated with with particular API calls and one of the learnings that we found from that is the kind of general rule of thumb if you like when you see really really high CPU usage usually the solution to that isn't to add more CPU. The reason the CPU usage is high is because the system is struggling with memory and we've had a few quite inefficient API calls which use gigabytes of memory and I think we've done quite a lot over the last year in particular to hammer a lot of those on the head and again a lot of credit going to Gintari and her testing environment for identifying and dealing with some of those. The other thing we find expensive back-end database queries usually related to analytics some of these are really quite difficult to deal with and yeah lots of diagnostic skills and and hunches and things required recent problem that's been rearing its head for me at least certainly the last six months to a year has been this issue of database connections being made outside of the data of the connection pool in DHS too we have this this connection pool which is used for probably 90 percent of all the queries but there are a few places in the code where we make connections outside of that pool and we've seen cases where those connections start to get out of control and reach the reach the maximum connection count on the database and bring the system down. The kind of issues that sort of come up on a on a frequent basis I mean they're probably more but these are the ones that were in my head at the time of making this slide. As I said at the start I mean managing DHS too is a mixture really of of sort of routine monitoring and then frequently quite a lot detailed troubleshooting. The different tools people use for the real basic stuff things like moon in and monitor I always install moon in just by default not because it's beautiful it's actually quite ugly but with very little configuration I get most of the the useful information that's required on a routine basis. I think there's a bit of competition at the moment between the Prometheus Grafana combination and the Elk stack combination. I mean those are both two much more beautiful platforms in some ways much more flexible and I know they are both Prometheus and Elk uses in you know our little panel today so we might get a bit of discussion around that. Profiling particularly for troubleshooting is is really important. Your kit is something that's favored by developers really detailed stuff. I found Glowroot thanks to Dan for this Dan introduced me to Glowroot last year and I found it totally invaluable for fixing all kinds of issues over the past year. It's worth pointing out maybe that knowing what to look for is much more important than having pretty tools. I've seen lots of places where they have very beautiful tools but they have no idea what the data means. Besides monitoring, alerting is really critical and to be honest I think it's very often a weak spot in a lot of our implementations. I mean generally it's quite easy to set up things like email alerts but a lot of implementations find it quite tricky because of the DNS setup that's involved with having a reliable email alerting system in today's world. Yeah so we co-present us today as Gintari, Daniel and Jason. I'm going to give them spotter. We didn't discuss in advance the sequence to go in perhaps we'll just go in the sequence that we have here starting with Gintari who's going to tell us a bit about the performance testing that we're doing back in Oslo. Daniel as I say is going to give us just a very brief overview of some of the the analysis he's been doing recently on Postgres and Jason's going to tell us a little bit about what they've been doing down in Cape Town on the monitoring front. What I hope to go to after that is a bit of more open discussion and questions. I've got a couple of discussion questions here which we might get to. The big one for me is how do we balance that a very complex DevOps type environment with a kind of limited human resources. People who are just learning how to administer systems who are maybe taking on these challenges for the first time.  I will say that in almost every case I've seen people starting off trying to build configurations with with load sharing they almost all have failed and performed much worse than if they had done it more simply to start with. There's a lot of understanding that's required to to put more complex configurations into place. A lot of learning we still need to do around that about how to be able to package those things. I mean if we're really going to recommend more and more sophisticated deployments then how can we package them to make it easier for people to use. But there's just some sample questions from me. The discussion may well develop differently. But what I want to do now is to give over to Gintari and let's move forward. Because you guys confirm that you see my presentation. Gintari, we can see your slides. Perfect. I would like to stop sharing my video because I do want to show you guys performance. I know it's just me Gintari. I can't hear you. I can hear you Gintari. Does everyone else hear me? Now I got scared. Okay I assume you guys do. So yes I'm going to stop my video because I'm going to be showing you performance tests in action later. So we have been doing performance tests for around two years now and we are testing the releases. But this year has been special because of COVID pandemic of course. It actually made us to encourage us I would say to do some more extensive performance testing. We wanted to make sure that DHS2 can reliably support mass vaccination campaigns that the packages, metadata packages we deliver support large scale of data and we wanted to be proactive and find issues as soon as possible instead of you guys coming up to coming to us with issues. And yeah we were aiming to give you some recommendations for your implementations but later on that. So steps we took. First we started with database. We did not have any large scale databases that we could use for this initiative. So we had to basically start from scratch. We generated a lot of org units and users around 400,000 org units and users but gradually we actually scaled down on that. We imported four metadata packages. First of all of course it was COVID vaccination package and then we also used COVID case-based surveillance, COVID events and API packages just to be able to you know cover different scenarios. And then we generated data. So in the end we ended up with three million tracked entity instances, more than six million enrollments, eight million events and a lot of data values and tracked entity attribute values. Then we got ourselves a quite powerful server. It had 32 cores and 128 gigabytes of random access memory but we quickly saw that this wasn't showing us issues as fast as we wanted to. So we scaled on on that as well. We got another server or AWS instance that was that had only eight CPUs and 42 gigabytes of RAM. I'm sorry I'm hearing some background noise. Then we came up with the workflows that we thought it would be the most common in the vaccination campaigns. It included all areas of the HS2. So Android, analytics, tracker, aggregate, everything. And we implemented tests for that where it was missing. I'm going to show you the test board we're using. We're using a tool called Locust IO as a test load generation tool. It is an open source tool that is written in Python but we do use Java integration. So right now I'm running the tests and I have very low load just to be able to show you guys how it looks and what kind of information we're getting from it. So here, Martin did you raise your hand? Is it possible to zoom in a little bit on the figures? Thank you. Yes. So the figures, is that enough? Yes, thank you. Great. So this is how the tools dashboard looks. It gives us information about the endpoints we're testing, the response breakdown, which one of them are the slowest, which one of them are the fastest. It also gives us some nice charts that shows how much response times depend on user count and similar. But this information is usually not enough. It is enough when we want to compare the releases in terms of performance. But when we want to find issues, we actually have to monitor the test front better. So Bob suggested us to use this tool called Chloroots. Now I know that it wasn't his recommendation but we're letting the tool and it's very helpful. It allows us to find a lot of issues. I'm not going to talk about the tool itself a lot because I know that we will have a presenter speaking about monitoring. So the tests, I want to quickly go back to the tests and tell you guys that we write tests in the way that they can be runable against any implementation or we try to. So the test generates data and API calls based on the metadata you have in your database. So if you would be interested to collaborate with us on that, you can just drop me an email and we can make it work. We can benchmark your databases and instances. Okay, back to the presentation. So with this framework and performance tests, we were able to make a lot of impressive bug fixes and increased performance by a lot. So what we found was actually what Bob talked about earlier. It was slow database queries. There was high CPU load on the Tomcat and similar. Also we found some our own apps making very excessive calls like skipping paging or requesting all fields when they didn't need them. So this is the chart that you might have seen on Monday. Marcos actually presented that on what's new on the HS2 session and it just shows the performance improvements that we've made on Tracker. As you can see the improvements are very impressive and especially what's related to Android. We were able to speed up sync by a lot. There's also some findings that we have not fixed yet and that was because it needs quite a bigger effort I'd say. So that's related to general Tracker CPU usage and Tracker being very heavy on the server. We want to improve that. And also analytics. We saw that analytics is not very fast when it comes to large volumes of Tracker data. So we're going to be working on that soon. Hope so. Yes, and recommendations. So I already mentioned that monitoring is very important. You should definitely get that. And also one thing that I'd like to recommend is to not be scared of upgrading the solution to use out. You use that support the HS2. So Java, for example, we did find that Java 11 works best with the HS2. So this is our official recommendation. Yes. So that's all I wanted to share with you guys. In case you want to contribute to our performance test, please visit this repository on GitHub. And if you want to collaborate on performance testing your instance, just let me know. That's great, Kintari. I think very informative and interesting. And yeah, I hope people do take you up with your invitation there. That we could help by using our test environment. And country implementation can actually deploy it themselves and test against their own test databases or staging instances. Very good. Thanks very much. Thank you for inviting. Who do we have up next? Daniel. Now Daniel, I know you've got a lot of facts and figures there. So we have to squeeze you into 10 minutes. So good luck with that. I want to go quickly through the slides of the results. Okay, let me share my screen. Yeah. Can you see my screen? No. Let me... Yeah, okay. Okay. Hi, everyone. I'm going to talk about the performance between PostgreSQL 10 and PostgreSQL 13. And for this, we have run several tests for different servers with the DHS 233 6 and the others with 236. And each of the servers with different PostgreSQL version. Okay. Well, the tests were about the database sizing, some of them about the analytic tables update, some load testing about the track identity instances, about the upload and download the TI's, because this caused a high load in the system. And finally, well, some execution plans about the some sample queries and some conclusions about the test. Okay. Well, these are the specifications of the servers. We have used AWS instances with the ACPUs 64 GB of RAM. And all with the same operating system, the web server, Tomcat, and the different PostgreSQL versions. The database we have used was the same for all the servers. It was a production database, lurching off to draw conclusions. We have more than 65,000 data elements, a lot of programs or units, TI's, events, etc. Okay. The first test were about the db sizing. And as we can see, one of the main advantages about using PostgreSQL 13 is the size of the database. It's up to 42 percent smaller with full analytics. And the difference, as we can see, is higher if the years of the analytics are increased. The next test were about the analytic tables update generation. And well, the performance with PostgreSQL 13 is a bit lower, about 10 percent more or less. Well, these are the results of the test, about the results tables update. And about the analytics tables update, the timing is similar between 233 and 236, although it's better with 236. But it's better also with PostgreSQL 10, as I said, about the 10 percent lower. And if we go to the timing, we will see that the main bottleneck about the analytic tables update is when the analytic tables are being populated. As we can see, with this task, when the analytic tables are being populated, for example, for the events, it takes about one hour more. So it's not very good. The next test that we ran were about the load testing, about the upload and download the TI's with a different number of concurrent users to increase the load in the system. And then we'll get the average response times. And as we can see, the performance is much, much better with the latest release of VHS2. And of course, the improvement is about the 20 percent better with PostgreSQL 13. Okay. And more or less the same with the post test, about the target entity instances. It's much better with 236 and PostgreSQL 13. Okay. And finally, we have run some tests about the execution plans of some sample queries. Okay. The performance is better with PostgreSQL 13, but not with the analytic queries, as it was expected. And for the other kind of queries, the performance is better with PostgreSQL 13. And the gain with queries about the target entity instances, so for example, with the relationship item is much, much, much better with 236. And of course, with PostgreSQL 13. And finally, that's a conclusion. We can say that the upgrade to PostgreSQL 13 is recommended. And of course, the latest releases of the HHS2. Because as we can see, the database sizing is up to 42 smaller with full analytics on PostgreSQL 13. And, well, although the analytic tables update takes about the 10 percent more, more or less with PostgreSQL 13, the overall performance improvement outweighs the loss of that efficiency with the analytic generation. The global behavior is better with PostgreSQL 13. And of course, with 236. Thanks, Dan. I mean, there's a really interesting bit of analytical work that you've done there. I'm a little bit relieved because I've been recommending people to go to 13 for some time. I was worried that you were going to come up with the wrong conclusions. I'm kind of surprised that the analytics generation takes longer. But it is what it is. The disk space saving is hugely significant, I think for a lot of people with disks, actually quite expensive part of their plan often. But maybe we'll hear from Jason and maybe from Dan. I don't know where they are currently in their thinking on database versions. But let's move on to Jason's presentation and then hopefully we'll come back to this question. Thank you. Okay. Thanks. Interesting. You should say that, Bob. Funnily enough, Drive Space is one of the issues we don't really worry too much about in South Africa. It's one of the things we have an overabundance of. But I suspect that's specific to our least environment. All right. Good day everyone and welcome. Thanks for attending. And Bob, thanks for inviting me to talk to you today. Also, thanks to Renea Rousseau, an infrastructure colleague who helped put this short presentation together with me. And apologies again on behalf of Pochlaki Maloy, the current infrastructure manager for Hisp, South Africa, who was unable to attend today. That leaves you stuck with me. So let me introduce myself. I'm the ex-infrastructure manager for Hisp, South Africa. And I've been with Hisp since 2013, so nearly a decade now. And today I'm going to talk to you about the importance of performance monitoring in larger scale DHIS deployments. I say larger. We have a reasonably complex environment, I would guess, by comparison to the average NGO or NPO type environment, certainly. But I'm well aware that in a corporate scale, we're definitely small potatoes. But in our peak in 2018, we had around 400 servers. We're now at about 250 through some efficiencies and change in our design. And that's mixed, what we call the Bob model, which is a mix of bare metal and virtualization in our environment. At the time, we were hosting around 220 instances of DHIS2, hosted across 54 physical Intel Xeon E5s. And our total environment had around four terabytes of RAM. And amongst that lot, not only was DHIS2, but other applications as well. But we were hosting about 80 unique DHIS2 databases. And those were from multiple countries. And as I'm sure you surmised, they were mixed. Originally in the early years, certainly back 2013, 2014, 2015, most of our databases were routine or aggregate databases. But over the years, as Bob mentioned, the focus on tracker and event has increased. And so today, in fact, our tracker databases outnumber our aggregate routine systems. So the HISP South Africa infrastructure team has isolated between about two to seven people over the years. But we are supported by a large data science and data management team of over the years, varied between 12 and 25 odd individuals. So I'm sure you've all heard great stories throughout the conference of excellent work and success. Today, I'm going to tell you a slightly different story. It's one of trial, tribulation, tantrums and trepidation. And I share the story with you all in the hopes that you'll learn as we did from our eras and work towards ensuring they don't become mistakes. Any error is a good learning opportunity. And to paraphrase Robert Kiyosaki, the best way to predict the future is to study the past. So let me start by saying it's my experience that there is no normal. As Bob and the other speakers have mentioned today, there is a wildly different performance and behavior animal built into two different versions of DHIS, such as a routine aggregated national instance, and compared that to a, for example, a tracker-based instance that is, you know, receiving data from mobile devices in the field via the API. Likewise, over the years, depending on the version of DHIS that you've used or the specific configuration, the type of database that you use, the type of usage that it will experience, whether the traffic is more aggregated or whether it is highly transactional, you've got factors such as the resource allocation variation between the web server and the database server, which will have a massive difference on not only your system's performance, but the is dictated to some extent by the type of system that you have. And obviously there is a vast difference between an instance serving a population of 5 million versus a population of 55 million and over the years, the hardware has changed significantly as well. If you compare the CPU market of a 2002 era E5 Xeon with a modern day AMD EPYC 7700 series processor, they are just worlds apart. You can condense an entire data center of E5 Xeons into one or two EPYCs, 30 or 40 machines, and get comparable performance, albeit potentially with some troublesome bottlenecks here and there, but purely talking CPU power. And then of course, the other thing to consider is your connectivity. What is in place for your particular project and that will definitely affect your system performance, doesn't matter how fancy your server is or how slow the server is if the people connecting to it are doing so through the equivalent of a straw or using edge or some intermittent or poor connection type. So this absence of there being a normal, in my opinion, means that there is no shortage of ways to correctly monitor and track your system performance. There's definitely no shortage of wrong ways to do it either, but there are lots of ways to do it correct and they won't all be the same. And sometimes it will be inappropriate for certain circumstances, but absolutely appropriate for others. So my story of Woe begins a few years ago and it was a national mobile-based tracker capture program, sometimes five or six years ago. And it had an initial deployment with a couple of hundred devices to be scaled up to a national scenario. Each device would then be in the hands of a mobile healthcare worker capturing hundreds of records, which would then sync up daily. The program's timeline from concept to launch was extremely aggressive. So I meant to interrupt just a brief warning, you've got about two minutes. Sure. It was extremely aggressive, it was to last several months, but kicked off just a couple of weeks after it was conceived. So this rapid deployment meant that we didn't have a baseline, we didn't do enough testing. We couldn't use hibernate caching and database replication because of the time we didn't know how to. And so we were ill-prepared for what happened next. We had no performance monitoring capability at the time. We were using a company called monitor.us to do all our performance monitoring, and they had just a couple of months before decided to switch over to a paid model. And the cost implications of monitoring such a large infrastructure were just prohibitive, so we couldn't do that. The result, because Bob spoke earlier about the behavior of the API and the certain circumstances, back then it was worse, it's a lot better now. And because of the short timelines that the project was operating under, and because we had no way to monitor properly what was happening on our systems, we were caught unawares when we began to experience considerable performance issues, degradation in performance, and then eventually repeated tomcat death. So we had multiple systems load balanced to try and solve the problem through hardware at the issue, which is I think everyone will recall at the beginning of the session, Bob said that is not the answer. He's quite right. And we had to learn that the hardware, because we just did not have sufficient data to identify the problem and respond to it in the most appropriate manner. So we had a lot of people spending a lot of time propping up tomcat instances after they'd fallen over. And the result was a significant cost increase for us in terms of what that project cost us. It didn't need to, but it wound up costing us a lot more than it should. One minute. I won't be much longer. So that's where my message of understanding the problem is key. And some advice what we've learned from that experience. What is your normal? Achieve your baseline. Look at a combination of both long term or medium term and live telemetry. So we use Elk and net data, but there are lots of really good tools out there to do that. Use the pasta plan for the future and the standard systems spend the time required to do the testing in advance. And make sure that a good monitoring system is in place, which will decrease your response and investigation time significantly, which in turn reduces your overhead. Do stress testing, load testing. We're excited to speak more to Atari and experiment with the various configuration possibilities. And the six P's proper planning and preparation prevents poor performance. And we've since had some success in the ICSP program. There are speakers who have given presentation on this far better than I can. Thanks, Jason. I think we're going to have to leave you on a high note here of success. I'm glad we ended with the success slide. I know that the journey has been hard. We've only a few minutes left. I wonder, Dan, could we ask you maybe to respond to what you've seen or even just to add something from your own experience at BAO over the years, which might be useful to others to hear? Yeah, I guess that after seeing that presentation, I know I'm going to get a lot of requests to upgrade to PG-13. I think the problem with that is that sometimes it'll help you, sometimes it won't. The tracker improvements I think will be incredibly helpful. One of the things that we've been working on with monitoring is we're using a tool, of course, it's commercial, but it will work pretty well called Datadog. And what that allows us to do is it's built more for the model like Netflix where you don't have individual instances doing one thing, which is kind of the Glowroot model. What I can do is look at RAM uses across all 200 servers and see what's going on there. And so that's one of the tools that we've been kind of digging into to help find where the bottlenecks are, depending on versions is one of the things that we've kind of run into. Yeah, that's interesting. Datadog I know is very, very nice. I learn a lot from looking at the Datadog documentation pages. I've not had the pleasure of working with the tool. There's quite a bit going on in the chat. I wonder is there anybody want to ask any particular question to anyone who's presented? This is the point where people tend to go quiet. Otherwise, Dan, can I ask you something? Particularly around load balancing. I think you guys have got more experience with doing load balancing than anybody else. I've seen lots of examples, as I said at the start of people attempting to do it following the documentation, but not really understanding what they're doing. And as a consequence, making performance far worse than better. I don't know what your experience has been. I know a little bit. I'm particularly interested around replication of the database, for example, and difficulties with that, with analytics. Any points or advice?  The first project we did it on was with datum. So we had the resources and a pretty huge database to work with. And what we found is somewhere between once you hit six nodes, the overhead for everything else kind of makes it a moot point. And so what we found is three to six nodes seems to be where you want to do the horizontal scaling. The main bottleneck in that case is the database. And so you always want to have a replica that you can fail over. Database replication, as everyone knows, is incredibly difficult. And so for that, we actually leave it to Amazon and we use their RDS products because they probably have more engineers than we have staff just working on that particular problem. I will say the one thing, and we discussed it with Lars, is as it stands right now, there's not a good way to autoscale up and down because the DHIS2 comp has to know which instance talks to which instance. Is there any other questions about kind of the horizontal scaling? Oh, thanks, Dan. Yeah, I think that the Postgres replication is still really a vexing issue for many of us. And as I say, a lot of the cases where people have tried it, it hasn't worked out too well for them. It's really attractive, this idea of letting RDS take care of it. I'll give an example of, granted, this was quite a long time ago, but I worked at the travel company Orbits, and they had a team of probably 200 DBAs. And at one point, I asked about, do you do replication for databases to kind of take the load off? And he said it's just easier to have a larger database, a larger database server than it is to deal with replication. Space, and I think that's your experience as well, is it? You guys were looking at it, they were tapping with Glocky a number of times around replication. I don't know. Yeah, so we do use it, but I agree with Dan, our experience is probably by and large, and with the exception of very specific circumstances, it's more efficient to just use more resources, throw more resources at the problem rather than using replication. I was going to say the other example use case that we've seen is where you have the production database doing all the data collection, and then you use the replicated database for reporting. But the analytics process is pretty intense. And initially, we found that the replicated database had to actually be larger than the original database just to be able to keep up with all the analytics lines coming in. I think what we're all looking forward to seeing is taking that whole analytics generation task off the domain data collection database completely. And I know that there's some movement in that direction. I know Lars is actively looking at the problem as well. In Terry, in your performance tests, you haven't tested running against load balance Tom Katz or against replicated database at present, have you? No, not yet. But this is a plan for, of course, going forward. And we are definitely looking for any kind of recommendations that we can test. We are planning to test, of course, several setups and just see which one works best. So yeah, later. If I may have a question for Daniel. When you did your PG 13 versus 10 testing, how many years of analytics data were in your data set? We make tests for one year, two years, four years. And I don't know much year for the full analytics. I can't remember that. You're going to make him run a query live now. No, no. It was more, it was more along what I had mentioned in the chat is that we noticed, basically, you need to tie a CPU to the number of years. And we started getting to the point where you know, most systems are for eight CPUs. And so once somebody hit 10 years of data, we started running the bottlenecks and try to figure out what was going on. And we basically found that it was like, as analytics generates its tables, you need basically a CPU because it's each year basically gets a dedicated CPU for while it spins out the analytics. I saw that in the chat. It's actually quite an interesting little rule of thumb. We need many more of these rules of thumb, I think, to try to at least take an initial stab on what kind of resource allocation to throw at what kind of shape of database. I think we've reached the top of the hour. We're probably going to be kicked out soon. So if just in case we are, let me just thank everybody who's presented and thank Dan for making time to be available as well. It's been interesting for me.


Lipton Tea and Lipton Soup present Inner Sanctum Mysteries. Good evening, friends of the Inner Sanctum. This is your host to welcome you through the creaking door. We're going to begin tonight's festivities on a scientific note with something that will save you a good deal of money. Yes, it seems that quite a few people have taken to turning off their refrigerators during this program. Yes, they've discovered they can deep freeze all sorts of things by just listening to Inner Sanctum, letting their blood run cold. Well, Mr. Host, that's quite an idea. Have you any other scientific suggestions? Oh, certainly, Mary. Princess, here's a way to cut down on your laundry bills. Instead of having the laundry start your clothes, just put them on top of your radio Tuesday nights. Inner Sanctum will scare them stiff. Well, now it's my turn, Mr. Host, and I'd like to make a suggestion too. It's about an easy way to get more enjoyment out of your teapot. Here's the whole trick. Just use Lipton Tea. In every cup of piping hot Lipton's, there's extra tastiness waiting to delight you. And the reason is Lipton's grand brisk flavor. Brisk, you know, is the tea expert's own word for the tangy, full-bodied flavor of Lipton's. And what a pleasure that flavor is. No wonder more folks buy and enjoy Lipton's than any other brand of tea in the world. They've found, as you will, that Lipton's always has brisk flavor. It's never flat, always lively and spirited, with the hearty zest of tea at its best. So next time, serve Lipton's tea. Its brisk flavor gives you more contentment in every cup. Now, friends, plenty of your tonight's story. An original radio play by Robert Newman called Blood of Cain. Darring two of your radio favorites, Mercedes McCambridge and Carl Swenson. A tale of blood spilled in hatred and vengeance. A blood that carries a curse that is as old as man. You don't believe that's possible, eh?  A small square once fashionable, on the outskirts of New Orleans. The iron balconies where elegant ladies once sat are now rusty and sagging. The paint on the rambling houses is cracked and peeling, and grass grows between the cobblestones. The bios in the jungle have crept close to it. That night the cries of strange birds, the croaking of giant frogs can be heard. What was formerly a living portion of the old world has become a place of decay and death. More pigeons died this evening. I saw them towering up and up into the darkening sky, as if to escape the pain gnawing at their vitals, and then fall into the square and into the garden. I still didn't believe, couldn't let myself believe, and then I saw her standing in the shrubbery, the bag from which she had been feeding them still in her hand. I knew then that I could wait no longer that I had to find out. Louise. Eugene, the pigeons, there's something wrong with them. They're dying. Yes. How? And why? I'm no chemist, but I'd say it was poison. What? But who would do that? Louise, now listen to me, dear. Please listen. I love you. I've loved you since I first came down here. First met you. You know that. And you know that I'll understand. Now tell me, why have you been doing it? I... You're the only one who feeds them. The only one who could do it. Oh, no, I didn't. Louise, where are you? Grandfather! Grandfather! Oh, oh, my dear child, what is it? The pigeons. They're dying. It's the second day now. And Eugene said... He said what? Have you looked at any of them, Dr. Philippe? Examined them? Would you go into the house, my dear? I'll be along in just a minute. But Grandfather... Please, please, dear. He's Grandfather. Of course. I'm very sorry you did this, Owen. Mentioned it to her. Did you examine any of the pigeons? I did poison, probably from my laboratory. But then... Then you know... I know a great deal, Mr. Owen. I'm her grandfather. And I think it would be very wise if you kept away from Louise. I did not see her again. What? Well, that's ridiculous. I love her. I'm sure you do. But perhaps I did not make myself clear. If you continue to see her, it might prove dangerous for you. I didn't eat any supper that night. I went back to my room. I sat there in the dark, staring at the shuttered, brooding house. About eleven o'clock, the door of the doctor's house opened quietly. And Louise came out, without looking right or left, moving almost like a sleepwalker. She went up the street. I hesitated only a minute, and then I hurried down the stairs and after her. And just as I got downstairs, the door opened again and... Is that you, Owen? Yes, doctor. She just left the house. I was watching from the window. Yes, I know. She's done it several times. And this time I was determined to follow her and see where she goes. Look, she's hailing a cab. My car is right across the street. Quick! Louise went in there. What kind of a place is that? Yeah, that smell. That's the smell of death. From the noise, I would say it was a nepotron. A slaughterhouse? Good lord. But why? Why would she come down here in the dead of night?  I think you know that. And it was for your own sake that I warned you to keep away from her. There are things that you do not, well, that you cannot know about her. No? Well, we'll see. I'm going in and get her and find out from her. Just a moment. There she is, just inside the gate talking to the watchman. No sense arguing about it, lady. I just can't let you in. But you must. You've got to. You always did before. That's just it. Now, once was all right, even twice. But even if you want to know the truth, the men have been complaining. Nobody exactly likes killing steers, but they say that the way you stand there watching them, well, it makes them nervous. You've got to let me in. I'll make it worth your hard work. Watching the killing? I've just got orders. Oh, that's awful. It's horrible. I'm going in again. No, no, no. In the state she's in, well, having you come on with her suddenly would have a very bad effect. But you wait here. I'll go. No, no, no. Don't hit me. No. Louise! What happened to the watchman? There's stone here on the temple. Louise! Why did you do this? I don't know. I had to get in here. He wouldn't let me in here. Take her out to the car and come on. Let's go. Don't go now. Leave him lying there like that. He's hurt. He may be dying. No, no, it's not serious. He's just stunned. We can phone for an ambulance as soon as we get home. But we've got to get her away from here immediately. Sit down, Louise. Yes, grandfather. I'm sorry I ran away like that, Eugene. But I suddenly felt strange. It's all right, Louise. You've been feeling that way quite often lately, haven't you? Yes. Yes, I have. Just what makes this feeling so strange? Well, it's hard to describe. It's as if I weren't myself anymore. It's the opposite of sleepwalking. It's as if I were awake, but not really conscious. And I hear voices. Voices telling me to do things and whispering a name. A name that sounds like... you're not. You've been in my study reading my books. No, but I haven't, grandfather. You always forbid me to... And how did you know that name? Well, whose name is it? Well, what does it have to do with Louise? It's the name of a family which is almost extinct. And it is a name which means death. Well, what do you mean? I was always very interested in the you're not family for reasons of my own. And I've collected all the historical references to them that I could. These references start with the 13th and 14th century. But by the 15th century, they had become the traditional executioners of France. Executioners? And in those days, you know what that meant. It was a you're not who put the torch to you're not. You're not received a handsome bequest from Louis XI for services rendered. You're not operated the guillotine during the French Revolution. Perhaps that's why the family migrated here after the fall of the Republic. They came here? Yes. Twenty-odd years ago, there were a whole series of particularly atrocious murders here. The murderer was finally caught and executed. His name was... Max Genonne. Why do you tell me all this? And why should I constantly seem to hear that name? Well, even when you were little, my dear, you used to have those strange, fit spells when you would do unpleasant things. After which, you could never remember them. Now, do you remember poisoning the pigeons and going to the slaughterhouse? What? Oh, no, no. Well, I've never discussed the matter with you because I thought it might actually implant the idea in your mind. I'd hope that if you were left alone, you'd outgrow it, but... But what? You still haven't told me what all this about the Géonise has to do with me. It has a great deal to do with you, my dear. You see, Max Genonne was your father. Well, now there's a girl with a real tradition in her family, a murderer for her father and a long line of ancestors who were real killer-dillers. I was beginning to get a little worried about her, fooling around poisoning pigeons and things like that. But I can see now that there was method in her madness. After all, practice makes perfect. Good gracious, Mr. Host. Louise may turn out to be a very dangerous person. Yes, Mary, she's not the kind of a character you'd like to have ringing your doorbell. I should say not. Why, usually, as our Lipton listeners know, a doorbell is one of the nicest, most friendly sounds there is because so often it means friends are dropping in for a visit. Of course, your first thought is to make them welcome. So after you've taken their coats and made them comfortable, pour them a cup of flavorful, spirited Lipton tea. You know, it's wonderful how Lipton's adds squackle to the conversation, for there's extra enjoyment in its grand brisk flavor. Each sip is brimming with such lively, full-bodied tanginess. Yes, serve your friends Lipton's when they call, and when you say goodbye to them at the door, you can be sure they'll come back for another visit and another cup of Lipton tea. That's a friendly suggestion, all right, Mary? And now, are we ready to get down to business again? The kind of business that Louise's family has specialized in for a good many years. Murder. Well, I know that what she's been interested in so far is small fry, but I think from now on she'll really be cooking with gas. Just a moment later now. Sitting in Dr. Philip's study, Louise and Eugene Owen stare at the elderly gentleman with shock and horror in their eyes. You mean my father was a murderer? Your father and his father before him. Back as far as the family's history can be traced. Well, I don't believe it, and even if it is true, it's ridiculous to think this need to kill can be passed on from generation to generation. Of course, Owen, there is absolutely no scientific basis for it. Still, how else can you explain some of the things that Louise has been doing? Oh, it's true, it's true. These spells that come over me when I don't know what I'm doing, that name which I never heard consciously until... Oh, no, it can't be true, it can't... Oh, Louise... No, Eugene, don't. Don't come near me. Don't touch me. If there's even a chance that it's so... Well, I didn't want to tell you. I was never very close to your father because he was a rather strange son-in-law. But if you want to know more about him, there is someone you should talk to. Joel Ferguson, down on Gaylord Street. He was the very last person to see your father alive. Mr. Ferguson? Yes, who is it? My name is Louise Philippe. I'd like to talk to you for a few minutes. Just a second. Come in, in here. Thank you. What did you say your name was again? Louise Philippe. At least that's what I always believed it was until yesterday. And then I discovered that my name was Louise Jeunasse. Jeunasse? Not Max Jeunasse. Yes. He was my father. You could hear word of me throughout these years. It wasn't my fault. You know it wasn't my fault. I only did what I had to do. I don't know what you mean. I was told that you were the last person to see him alive and that perhaps you could tell me something about him. Yes. There are things that I can tell you. That he was evil with that he knew he was evil. There was a curse on him that made him do the things he did do. Voices that whispered in his ear that made him kill, made him kill. Voices. And in the end, at the last minute, he thanked me. He thanked you for what? For stopping him from doing any more killing in the only way he could be stopped. And when I put the rope around his neck. The rope? I was the state executioner. It was I who hanged him. You killed my father. They only carried out the sentence. It was passed on you. Why are you looking at me that way? I didn't know. I only came here to see you because... You're lying. You were his daughter. Then you were like him.  Kill you? That knife there in your back. Why didn't you even know it was there? You're lying. Keep away from me.  Stop staring at me like that. Put that knife away. Don't come near me. Good evening, Mr Owen. Where's Miss Louise Benson? I must see her right away. I'm sorry, sir, but she's not in. She went out about a half hour ago. Well, where did you go? I'm afraid I don't know, sir, she didn't say. What about Dr Philippe? He's not in either, sir. He left right after Miss Louise, as soon as he heard that she'd gone out. It seemed to me, sir, he looked rather worried. He said something about Mr Ferguson. Ferguson? Great Scott. She went down... Thank you. Who's that? I'm looking for Miss Louise Philippe. I was told that she... Is that you, Dr Philippe? Yes, Eugene. Is she here? Yes, she's here. But you're too late. Just as I was too late. What? What do you mean? Inside there. See for yourself. Oh, don't tell me that anything's happened. Good. Lord. Louise. She won't answer you. That's the way I found her when I got here.  Eugene? Louise. Oh, Louise, darling. Why did you do it? Yes, dear. Didn't you do it? I don't know. I didn't even know who he was when I came down here. And I found out that it was he who executed my father. Then I started to hear those voices. Voices telling me to kill, that I had to kill. Then there was a knife in my hand. Oh, I can't remember. Voices. That's what Max said at the trial. It was the only defense he offered, but that he heard voices telling him to kill. Didn't save him. But in your case, the woman... What do you say? There's no sense even trying to escape. That would only make things worse, you know that? Yes, Grandfather. Except that there is only one way that things could be really worse. Louise, don't say that. You must... Oh, it's true, Eugene. Centuries to be a journaum meant to bear the mark of Cain. Well, I'm the last of the journaums, and there must never be another. Well, I had not gone quite that far, my dear, but perhaps you're right. I am right. Goodbye, Grandfather. Eugene. Louise, come back. Wait. No, no, let her go, Eugene. She's my own flesh and blood, but I think that may be the best way, after all. Well, we'll see. Louise, wait, please. It's no, you're Eugene. It's all very clear. Clear as witches, bro. Yes, it is, yes, it is. Maybe it's because of our love that I understand. Maybe that was what was wrong with the journaums. They never knew love, only hatred. But there is a cure. It lies there, in the river. Louise. There's a curse on me, on all the journaums that ever were. There is a curse, but not the kind that you think, dear. Listen, I was at the library all afternoon reading, and I think I understand now, for the first time. You understand what? The nature of the curse, and how it can be ended. Because it can be ended, in only one way. Oh, Eugene? Is that you? Eugene? No, Grandfather. It's not you, Gene. Louise, well, I thought you... Why did you come back here to Ferguson's place? I don't know, Grandfather. The voices, the call in my blood, it's too strong. I tried, I wanted to end it finally, and completely, either in the river or with the police. But I couldn't. You mean you're not going to give yourself up? Grandfather, I'm not going to give myself up. But that's not all I mean. Louise, you still got that knife. Yes, Grandfather. Walking up the street, realizing that I was still holding it, that it was red with blood. I think it was then that I knew for the first time what it meant to be a journaum. Look, Louise, you're completely distraught. That's only natural. But now, if you put that knife down and let me take you home, I'll give you something that'll help you sleep. No, Grandfather, you won't give me anything ever again. Well, Louise, you're not going to kill me. Yes, Grandfather, that's just what I'm going to do. Oh, Louise? No, don't try to get away. I'm younger and quicker than you. Just one quick move. No, no, no, no, listen to me. Listen carefully.  There's nothing driving you to do it. You see, you're not a journaum. What are you saying? It's true. I am a journaum. Max, journaum was my son. But you're not evenly, distantly related to us. You're lying. You're lying. You're just saying that. No, no, no, I swear it's true. I changed my name when I studied medicine to avoid the stigma. But Max kept it. Your father was Louis-Martin, the judge. You're lying. How is that possible? Because I adopted you after your father's death for a reason. It was your father who condemned Max to death. He didn't know Max was my son, and I didn't tell him until later.  And when he was desperately ill, I offered to adopt you and take care of you. Then when the papers were signed, I told him who I was, what I was going to do, that I was going to destroy you to avenge my son. And that's what killed him. All those horrible things I've been doing. Poisoning the pigeons, killing birds. You haven't been doing them. It was I who did them. Using drugs and suggestion to make you believe that it was you, so that you would either destroy yourself or... Thank heaven, Eugene! Eugene, did you hear? Yes, dearest. I heard. And I told you, didn't I? Eugene, so it was a trick. A trick to trap me. Yes, doctor, it was. You were in the city hall all day looking at the records. Not that night. No, and this won't be as poetic a death as the one I'd planned for her, but... The lights! Louise, put the lights out! That won't help you, either of you. You won't get away. I'll find you in the light or in the darkness and... There you are. Die then. Now you'll die! Die! Die, Louise! It's not I! It's not you! What did... Then who did... Let go of me! Lights! Louise, quickly. Eugene, what happened? Who did it? Yes. He stumbled into him in the darkness, thought it was one of us. Then he must have tripped. A knife? He must have fallen on the knife. But somehow it... It seems only right that the last of the Junows who killed so many should be destroyed by the dead. Too bad, Doc. You were a character we certainly could use on this program. But now at least you've got the perfect answer for the rest of your family when you meet them in the... wherever it is that dead murderers congregate. When they ask you who you were with last night, you can always say, that was no lady, that was my knife. Well, I'm glad you could put a new point on that old joke, Mr. Host. Oh, I like to sharpen up an old saw now. Well, let me see what I can do along those lines. How about the best things in life or tea? I'm afraid you're enthusiastic for Lipton's. It's got the best of you. Well, maybe you're right, Mr. Host. But it's so easy to be enthusiastic about Lipton's. Once you taste that marvelous brisk flavor, you can't help being a real Lipton fan. And that lively, zestful taste is something you folks should start to enjoy right now. Next time you visit the grocers, get a package of Lipton tea. Try it. I know you'll enjoy it. May I add a word of advice, friends? If you should happen to bump into an elderly gentleman dressed all in black on a dark street some night, don't get into an argument with him, especially if he happens to be carrying a bloodstained knife.  Oh, by the way, this month's In a Sanctum Mystery novel is The Pavilion by Hilda Lawrence. And next week's In a Sanctum Story brought to you by the makers of Lipton tea and Lipton soup and directed by Hyman Brown. Next week's story is called Skeleton Bay. It's about a lady novelist who writes murder mysteries. Until she decides she'd rather be a character instead of an author. She chooses Skeleton Bay as a vacation spot, but it turns out to be her last resort. Now, join us next Tuesday on Skeleton Bay. Until then, friends, good night. Pleasant dreams? Ready in a jiffy and as tasty as can be. That's Lipton's noodle soup.  You'll love it's fresh cooked, homemade flavor. And Lipton's noodle soup is ready to serve in a few quick minutes. It's economical too. It costs less and makes lots more than canned soups. Ask your grocer for Lipton's noodle soup mix tomorrow. And don't forget to tune in next Tuesday night for another Inner Sanctum Mystery. All names and characters used on Inner Sanctum are fictitious. Any similarity to persons living or dead is purely coincidental. This is CBS, the Columbia Broadcasting System. All names and characters used on Inner Sanctum are fictitious.