The coverage of evaluation of nutritional status in seven municipalities in Brazil has been low since 2008, despite a slight increase in some municipalities. Factors such as population density, GDP per capita, and the number of community health agents have been found to be associated with this coverage. Understanding these factors could help inform strategies for expanding coverage of nutritional status assessments. This article was authored by Irene DA Silva Araujo Goncalves, Patricia Feliciano Pereira, Mariana Belcavalo Lino Silva, and others.


By about 1700, the English language differed only slightly from present-day English. The most important development was external. It concerned the position of the English language among the languages of the world. From a regional language, it developed to the most widely used language of our time. Hence, over and above the discussion of the linguistic changes that affected English after 1700, we will look at the spread of English and we will discuss the position of English as a global language in this e-lecture. So let us start with the linguistic changes that affected the English language after the early modern English period. The main linguistic properties of present-day English had been manifested by the end of the early modern English period. The changes that had begun during the middle English period were by and large completed. Second-person singular forms, along with the pronouns thou and thee, had disappeared from ordinary educated speech. And the auxiliary do, well, do had come to be used as we use it today. In the N-I-C-E context, do you remember? N negation, I don't go. I inversion, do I go? C coding, I go, don't I? And E emphasis, I do go. Nevertheless, a number of changes between early modern English and present-day English need further consideration. And these concern changes in the sound system and changes in the vocabulary. So let's briefly look at some sound changes that affected the English language after 1700. In general, the sound system of present-day English was established by the end of the early modern English period when the Great Baal shift was completed in most varieties of English. Most changes since early modern English have been alophonic rather than phonemic. And now we have to distinguish between the English spoken in the heartland England and the Englishes elsewhere. Let us look at two examples that affected the present-day English sound system. The most important, consonant aspect that affected the English language was the reduction of roticity in some varieties of English, especially in British English. You remember roticity? The use of the post-vocalic R in words such as farm, far, etc. Now, as you know, in the North American variety, roticity is now pretty high. In Irish English it is high, whereas in England, in RP, it is extremely low. Scottish has a medium degree of roticity. Well, let's move the label high over here. And the varieties on the Australian continent in Australia and New Zealand are pretty low. In the vocalic system, the central changes affected the diphthongs. They were either smoothed to monophthongs. For example, A became E in some varieties. Or new diphthongs entered the system, I, O, etc. in some varieties. Let's now continue with the vocabulary. During the Middle English and early modern English periods, great increases in the English lexicon could be observed. The vocabulary of present-day English has continued to expand at an incredible rate. The main process of vocabulary expansion was borrowing. However, while the bulk of long words in Middle English were borrowings from French that ranged across the entire spectrum of semantic areas in early modern English, borrowings were primarily from Latin and Greek, and they tended to be more learned words of a stylistically formal kind. However, while in previous periods, all areas of the vocabulary of English were affected, the bulk of recent borrowings belonged to specific areas that can only be understood by educated experts. So in present-day English, growth via long words continues and affects by and large scientific and technical terms. The words have often been created in present-day English out of previously borrowed elements. Thus, the classical vocabulary of present-day English is larger than the total known vocabularies of Latin and Greek. Since present-day English has composed so many new Greek and Latin words by means of affixation, for example, by prefixes such as auto, hypo, x, intra, and so on. Beyond these scientific borrowings, present-day English has borrowed freely from other languages. For example, from French words such as beige, blau's menu, or restaurant. From Afrikaans over here, Trek, Artwark, to name two examples, from languages such as Italian, Salami, Fiasco, Inferno. Here are some German words, Seminar, Gestalt, Pretzel, and last but not least, least words from Spanish, such as Bonanza, Gaucho, or Taco. Another method of expanding the vocabulary of present-day English is the creation of new words by means of word formation processes. And these processes have led to an enormous rate of vocabulary expansion by means of the standard processes such as derivation and compounding. Now, in derivation, several new affixes came in such as autosuggestion, ultrasonic, neodermic, and so on and so forth, and so the pool of derivatives could be enormously expanded. And in compounding, as you can see over here, the most common type by far is a combination of two existing nouns as in Asset Rock, Cable TV, and Ghetto Blaster, to name just three of them. Further word formation processes involve, among others, blends such as brunch, breakfast and lunch, words such as these over here, so-called acronyms, UNESCO and NATO are two examples. Re-duplications such as Yum-Yum or Ding-Dong, back formations such as editor that becomes edit, diplomatic becomes diplomat, and clippings where parts of words have clipped off, violin cello can be used as cello and chimpanzee can become chimp. Let us now turn our attention to the outer development of English. The first significant step in the progress of English towards its status as a global language did not take place before the end of the 16th century. By the end of the reign of Elizabeth I in 1603, literature had boomed through the works of Spencer, Marlowe and Shakespeare, and Francis Drake and Walter Rawley had laid the foundation for expanding the English influence in the New World. According to the Indian linguist Braj Kashru, the spread of English around the world can be defined in terms of three concentric circles. The inner circle refers to the traditional historical and sociolinguistic origins of English, where it is used as a first or native language, and that of course concerns Great Britain and Ireland, then in North America, Canada and the United States, and of course Australia and New Zealand, the inner circle. Now the inner circle constitutes about 400 million speakers, almost 70% of which are from the United States. The outer circle includes countries colonized by Britain and the United States, where English is spoken as a second language, and where it plays an important historical and governmental role in multilingual settings. Here are the main ones. In Asia we have countries such as India, Pakistan or Bangladesh, to name just three of them. In Africa we have South Africa, Kenya, Ghana, Nigeria and Tanzania. And then the Philippines over here, and let's put Jamaica over here to name just a few. There are other countries such as Malaysia, Sri Lanka, Singapore, Zambia. So a total of 400 million speakers belong to the outer circle, that is the circle that defines English as a second or official language. The countries in the expanding circle did not institutionalize English as an official language, but recognized the importance of English as a foreign language. These countries include Israel, Japan, China, Egypt, Korea, Nepal, Russia, Taiwan, Indonesia, Zimbabwe, Saudi Arabia and of course Western Europe and the Caribbean, and of course the South African subcontinent with more than one billion speakers. Let us take a brief look at the inner circle development and let's start with North America. The first permanent English settlement in North America began with the arrival of England's second exhibition in 1607. The first by Sir Walter Raleigh and his fellow explorers who arrived in 1584 had been forced to return to England as a consequence of conflicts with the native people. The new Southern colonists came mainly from England's West country. Their tidewater accents still exist in some isolated valleys. They are said to be the closest to the sound of Shakespeare's early modern English. And then in 1620 the first group of Puritans arrived on the Mayflower. These people did not want to return to England. What the pilgrim fathers had in common was their search for a land where they could found a new religious kingdom which should be free from persecution. 1681 brought new shiploads of immigrants, the so-called Quakers from the north of England and the North Midlands who settled in Pennsylvania. Later there was a vast wave of immigration from Northern Ireland and Scotland to this area. By the time independence was declared in 1776, one in seven of the colonial population was Scots Irish. Today the following major dialects can be defined in North America. On the one hand we have ethnic varieties such as African American Vernacular English and American Indian English. On the other hand we have the regional varieties such as North Eastern American English, Southern States American English, Canadian English to name just three of them. Let's take a brief look at Australia. In Australia Britain established its first penal colony at Sydney after James Cook's visit in 1770 in order to relieve the pressure on the overcrowded prisons in England. The about 130,000 prisoners were followed by a wave of settlers. By 1850 the population of Australia was about 400,000 and by 1900 nearly 4 million. Today almost 80 million people live on the Australian continent. The British Isles provided the main source of settlers and thus the main influence on the English language in Australia. And what about the future? Well in so far as there has ever been such a thing as a world language, English is one today. During less than 300 years English developed from a regional language to a global language. Today English constitutes the most widely used individual language with more than 300 million native speakers and more than 1.5 billion official users. In considering the future development of English the following main aspects have to be considered. First there is the role of the British Empire. Some sociolinguists expect a strong reaction against continuing to use the language of the former colonial power and in favour of promoting the indigenous languages spoken in particular regions. The role of the United States and the North American influence has to be considered. The United States has the largest percentage of native speakers of English and the country has been influential in many ways. For example it is more involved in modern developments than any other nation. It is a control of the digital revolution. Hence it dominance and the dominance of American English may even increase. Next we have to consider the rise of new Englishes. The spread of English around the world will have an important consequence. English will become more open to linguistic change with totally unpredictable results. The emergence of new varieties of English in territories where the language has taken root demonstrates this. These new Englishes are somewhat like the dialects we all recognise within our own country except that they are on an international scale applying to whole countries or regions. Well and last but not least the emergence of a world English. In the future with many national Englishes little would change. People would still have their national dialects for use within their own countries. However when the need came to communicate with people from other countries they would use a globally understandable form a type of world standard spoken English. This standard already exists in its infancy and it continues to develop especially through the use of the internet and modern mass media. Thereby international intelligibility is guaranteed while at the same time national identity is preserved. Well let's summarize. In this e-lecture we discuss the changes that affected the English language between 1700 and today and discuss the development of English from a global perspective. Today English constitutes the most widely used language of our time. From a regional language it has developed into the world's number one communication system if you wish into a global lingua franca. Thanks for your attention.


All right guys, we're going to make something super special today. I think that everyone's going to like it. It is called the conversation. What we have here is an academic journal. I'm afraid not very many people are going to enjoy it just like this though. What we're going to do is turn this maybe a little dry academic writing into something everyone will be able to enjoy. You definitely want to make sure to cut out the jargon. Nobody wants that in their dish. Just the parts that will make your dinner get smarter and make them want to share it with friends. The goal here is to keep just the good parts of this article. Perfect. You're doing great. Next we're going to go ahead and add a can of clarity. Yep, just pour it all in there. You can't really mess this up. But remember to really mix in the clarity with those ideas nicely. And while you're pouring, you're going to want to stir, stir, stir. You're really going to want to get rid of all those lumps of unsupported opinion and highlight the facts. Now we're going to add a pop of color to our dish with some photos. And if you're feeling extra spicy, go ahead and throw in a chart or two. People really eat that up. Okay, I think we're ready for the oven. Go ahead and pop your dish in there. Cooking times may vary. The breaking news recipe only takes a few hours, but sometimes chefs can take days to get the perfect dish. You just have to keep checking the oven. Almost ready. Okay, I think we're ready. I think it turned out great. If you've enjoyed this recipe for our fact-based journalism, subscribe to our channel and consider a gift to the conversation. It's food for that delivered to the public every day for free, and your support is a key part of this recipe.


The ongoing COVID-19 pandemic has killed at least 1.1 million people in the United States and over 6.7 million globally. Accurately estimating the age-specific infection fatality rate, IFR, of SARS-CoV-2 for different populations is crucial for assessing and understanding the impact of COVID-19 and for appropriately allocating vaccines and treatments to at-risk groups. We estimated age-specific IFRs of wild-type SARS-CoV-2 using published seroprevalence, case, and death data from New York City, NYC, from March to May 2020, using a Bayesian framework that accounted for delays between key epidemiological events. IFRs increase 3-4 fold with every 20 years of age, from 0.06% in individuals between 18 to 45 years old to 4.7% in individuals over 75. We then compared IFRs in NYC to several city and country-wide estimates including England, Switzerland, Geneva, Sweden, Stockholm, Belgium, Mexico. This article was authored by Chloe G. Rickards and Amarm Kilpatrick. We are article.tv. Links in the description below.


Welcome to Toffee TV, it is the Everton News Daily. Sean Dight has given his pre-match press conference ahead of Everton's trip to Brighton on Monday. O'r shiims of it mad to me when the ad-press conference is on a Thursday when we're not playing till Monday. This is Everton, a lot can happen in that time, but he has done it. The first thing he was asked about was Everton captain Seamus Coleman, the update on his injury, obviously Everton's skipper yesterday. He put a message out on social media saying there was no ACL damage, didn't give any, well said, I'll be back soon, didn't really give any update. He was asked today the extent of Coleman's injuries, what the Everton manager had to say. Yeah, I mean it's still going to keep him out, you know it's still a ligament injury that needs sorting out but it's not, obviously everyone feared it was ACL and it's not, so that's good news and good news for him. Do you think he'll be back this season? I doubt it, I don't think so, no. There you go, good news for Seamus that it wasn't an ACL, but the manager seemed to indicate that he won't be back this season if it's a medial damage or whatever, that'll be it for the season, but like I said, good. That it isn't an ACL. Dominic Calvert-Lewin has returned the last couple of games, scored his first goal since coming back the other night in the draw at Leicester. He gave a much improved performance again, he seemed to be getting stronger each game. He was asked, Sean Dice was asked about Dominic Calvert-Lewin and has he been impressed with the striker's return? Well you can't expect it, you never know, players come back from injuries in different ways, but the goal was to get him to come back at a fitness level that could handle the Premier League, which he is showing his doing. And I thought he put in a real good physical shape at some of his quality, you know I found a goal obviously from the penalty spot but his movement to get there, movement on another chance, I went away from him but I want him in there, not bothered if they miss, that's part of the game, but it's getting in there and he's, he looked freer, you know since I've been here that's the freest I've seen him, interesting in his physical presence and the way he was running and that's a good sign, but he's been trying very well as well and he's put, he's been very frustrating for him, very frustrating for us. He's done back and hopefully that's the first goal of some important goals that he needs to get between now and the end of the season. Everton did get a point, did play a lot better at Leicester City the other night, again I said the other day felt a bit like two point stop just because of the way we played. But the much improved performance led the DTN Leicester as John Dice has there been a mentality shift from obviously when he come on Everton of one once away all season but was asked whether the performance against Leicester was an indicator that Everton are improving their mentality on the road, this is what the manager has to say to that question. It still remains obviously just the one away win all season, does that result you got Leicester change the mentality and the approach? We showed that, Forrest it was a mistake cost us the win there, a good mentality to come back to Chelsea, a very good performance at Leicester, there are all signs that it's correct in itself but we've got to continue, we've got to go and we've got to make sure that we're taking every game with that kind of commitment to the cause. It's not just the physical commitment, the detail in the performance, some of the passing movement was good, some of the chances created was obviously where you've mentioned already the amount of chances but also the quality of chances and I think that shows a strong sign of the team getting back to where we want it to be after. There you go, let's hope Everton can improve that away mentality and pull off what would be a surprise when it would be very surprising if we won a Brighton on Monday but you never know, stranger things have happened and we do need a big result quickly. The final thing really that Sean Dyche was asked about was the form of Michael Keane obviously being subject through a lot of debate certainly on social media in the general media as well, Everton can see a lot of goals of Leic Keane, hasn't been in great form but the manager was asked about whether he still has faith in his sense of act and this is what he had to say. Up and downs of the day, ups and downs of a footballer, ups and downs of a team, ups and downs of careers, it all goes into the melting pot. These are things that players have to work through, have to handle, goods, pads and indifference. A lot of noise grew in the new cars ago and I thought he was probably our best player until the second goal so that's how it can change very quickly but he's a very good player amongst many good players here, sort in the best 11 at any given time, form related to go out and perform. There you go. Everton Defender Tiara Branthwaite is set to return to the club this summer when his loan deal expires and PSV has been in excellent form for them all season play 33 times in Holland, had a great season for them. A season for them, Ryfann Nistro, he's very impressed with them and wants to keep them but the Dutch press report and that he will be too expensive to make that deal permanent. There's no figure but we've seen figures of 15 million mentioned earlier in the season. Personally I hope that 20 year old comes back and gets in to show on Dijkers team or whoever. It's a team and stays at Everton for a good few years and becomes a main stay but it's full of potential still only 20 years of age but just right now it seems though whatever the ask and prices for Tiara Branthwaite it's a little bit more than PSV can afford. Finally Everton have been joined in the race to sign Coventry striker Victor Djurochers by Wolves apparently. That is what's being reported today. He's been linked with Everton quite a bit. This season he scored 21 goals for Coventry City and Wolves very interested in taking him to Mollingale apparently so there you go. We'll see what happens with that one. That is it for the news day. Do give the video a thumbs up. Subscribe to the channel if you haven't. Thanks for watching. See you later.


Hello everyone, thanks for joining. I am Prabhakar Paranival. I'm a consulting engineer in Oracle Cloud Infrastructure. So what is in it for you, right? Like what you have to spend next 30 minutes here. So I would touch upon few internal aspects of HITCity and we would share some of the learnings we had over the last 7-8 years operating tens of thousands of HITCity clusters. And as we evolved, we re-architected our HITCity architecture. So we will talk about how we went about re-architecting and also migrating the HITCity deployment from the legacy architecture to the new architecture. And also we will talk about some of the learnings when we did this migration from the legacy to new architecture. Just a very brief introduction. So like I said, I work with OCI and OCI has a global infrastructure across the globe with multiple regions. And in each of these regions, OKE, which is our managed equipment it is offering, is a day one service. And there are multiple internal as well as external applications are onboarded to OKE. And in fact, many of the OCI services run on top of OKE itself. So with growth in the adoption, we were looking at operational efficiencies. And one thing we identified was to improve the way the HITCity was deployed and operated. Like you know, each OKE cluster has a backend HITCity and hence we need to find a way to operate it better, to operate Kubernetes better. So we went through this journey of re-architecting and at this moment we completed the migration to the new architecture with zero customer reported issues. I think having a great engineering team really helped. But the equal credit goes to the HITCity SIG because of the high bar they maintain with respect to the patch releases and major releases they make. Because we had HITCity clusters running all the way from HITCity 3.3 to now 3.5.x. But we could do all those migration with zero customer escalation. So I just want to thank the HITCity SIG for their focus on the high bar they maintain. So to start with some brief HITCity internals, as you all know, HITCity is a distributed key value store. I do hear that some folks run as single member HITCity in production. I don't believe that, but I do hear it from some of my friends. But it doesn't make sense running a single member HITCity in production. Yeah, it has to be a distributed key value pair. And HITCity uses GRPC for both its client as well as peer-to-peer communication. And you would know it internally uses HTTP2, so in case you are interested. So RAFT is the heart and soul of HITCity. So many features you think is feature of HITCity is actually inherited from RAFT. So like I said, HITCity is a distributed key value store. So you need to have a consensus between the HITCity members. So the consensus is through RAFT. So RAFT defines, for instance, how the replication of updates happen. It defines that it has to be a leader-based replication basically. It defines how the leader election is done on such things. So how RAFT works is it treats the system that it manages as a replicated state machine. So it defines how the updates to that state machine is done. So internally it has a concept of replicated logs as well as data state basically. So it controls the update. First the update goes to the replicated logs and then that is what is called as committing the update. And then this update goes to the backend store which is called applying the update. So those are RAFT terms inherited into HITCity. So within HITCity we use B-Bolt for storing the state. So that's why you'll be surprised, you have, newbies might be surprised, that we have a key value store inside another key value store. It's because HITCity is like a distributed key value store. Under that each of the members have a standalone key value store. In our case it's B-Bolt. So now I would jump into how we did, I mean, I'll start with how my legacy architecture looked and then how our new architecture looked. And then in the subsequent slides I will dive deep into the migration process and our learnings and other things. So what you see in the screen right now is the legacy architecture. The vertical rectangular boxes you see in the left, they're all the control planes of our Kubernetes clusters. So each control plane has three compute instance. And in our legacy architecture we only run the Kubernetes containers there. And what you see in the right side is our internally managed Kubernetes cluster where we run HITCity as HITCity parts. You can think of it as a variant of a QBception model where we run HITCity parts on an internal cluster for the customers Kubernetes clusters. And we had an internal gateway which was Envoy-based. And since we use TLS everywhere for communication between the QBAP server and the HITCity, we use the SNI mechanism in, supported in Envoy to route the traffic to the right HITCity store. Like if one of these Kubernetes clusters want to talk to their corresponding HITCity data store, they would talk to the gateway and the gateway knows which HITCity to forward to using the SNI mechanism. And in our internal cluster we ran the CoreOS-based HITCity operator for maintaining the life cycle of the HITCity parts as well as for the backup and restore mechanism. So as part of the migration what we did was we decided to move the HITCity members into the control planes which constituted the, I mean the compute instances which constituted the control planes. So like I said we had three compute instances so we moved three copies of the HITCity into those compute instances. So each control plane talks to the corresponding local HITCity. In our case the QBAP server which is running in that compute instance only talks to the corresponding HITCity. So we do that for all the members and once all the members are migrated we dismantle the corresponding legacy architecture in the given region. So now I would talk about the improvements that we made. So we were operating this legacy architecture for seven, eight years and then we decided to migrate to this new architecture. So we obviously had some learnings along the way. I listed a few of those here for the space constraints so there are much more I could add. So the first and foremost we continuously patch our nodes for security updates and things like that. So we have to make that error less error-prone and simplified. So what we did was we assigned a permanent identity to the HITCity as well as a permanent storage. This is a fancy way of saying we created DNS entries and block volumes and assigned to these HITCity members. Every time we go and create a Kubernetes cluster we provision three DNS names as well as three block volumes and we bring up three compute instances, attach the block volume and assign the corresponding DNS entry. Next month when I am patching those control plane compute instance all that I need to do is to terminate this compute instance, bring up a new one and attach the block volume and DNS entry there. From the perspective of the other HITCity members it's just a blip. The member disappeared for a few seconds and it came back because I have nothing has changed because the identity and the persistent storage is exactly the same. So this significantly simplified our operational process. Another thing that we learned from the day one is one size fits all doesn't work because we have clusters with one worker nodes and some with thousands of worker nodes. So we cannot expect the operators to jump in and tune the compute resources or memory or IOPS for the block volume. So we built in the auto-tuning from the day one so we continuously monitor the cluster's characteristics and if we see that this cluster is much larger than I thought. So we would automatically scale up the compute instances, memory and as well as the IOPS assigned to the block volumes. So 2GB is the default quota set for HITCity. So I spoke about the bbolt. So the bbolt has internal backend storage. So this quota setting which is by default 2GB limits the size of that backend store. So initially in the legacy architecture 2GB was sufficient but as we evolved and grew we consistently hit that limit and the operators get alarmed, they jump in, they do a defrag and then possibly increase the quota. It was unnecessary. So what we thought from the day one we would set the quota as 8GB. In fact we are talking to the HITCity 6 team to even consider increasing it to 16GB. Maybe 8GB is too old for today with the size of the clusters. Maybe 16GB or 20GB makes sense. Hopefully the HITCity 6 team might prioritize that. Another important thing with operating HITCity is the IO latency with respect to disks. So now we have built in the monitoring of HITCity and we continuously monitor the backend latency. So we have alarms configured against the block volumes provision and based on the alarms the operators jump in and then scale up the IOPS assigned to the block volumes. And lastly defrag is one of the critical operations which as you know can momentarily pause its remember. So we have more intelligent ways of defragging now. In our legacy architecture what we used to do is we wait for the quota to be HIT and the alarm to be raised and then the operators would jump in and do a defrag. Now we incorporated the automatic defrag in our code and we do it in a much more intelligent manner because running this unplanned manner may bring down the HITCity cluster because it's a like they say it's a stop of the world operation. During the defrag time that member is completely unavailable. If you are doing a defrag for the leader then it is quite possible that the leader doesn't send the right hard beats to the other members and you end up losing the leader election I mean you triggering the leader election and all those things. So we have to tune the way we do the defrag so that we have incorporated and now based on our discussion with HITCity we are tuning it even further. So I'll take a couple of slides about our migration process. One thing is obviously it has to be a zero data loss migration. No questions about that. And it has to be a zero downtime migration. So the Kubernetes control plane has to be available throughout this migration. Our intent is the customer is not even aware we are doing the migration. The only thing that we do which the customer which is observable to the customer is the customer cannot delete the cluster during the migration. So we typically take about 10 minutes per cluster and of course we do migration in a concurrent manner. But during that migration you cannot delete the cluster but we thought that's not a major limitation it provides additional safety for us. So we prevented deletion from happening but otherwise the customer can do anything he wants and he wouldn't even be aware that migration is happening in the background. And like I said considering the scale of the deployment it has to be fully automated. Operators cannot jump in and do a cluster I mean specific operation. You see set up the migration in for environment and it takes care of everything. And a couple of important things. While doing the migration we had a choice of updating the HITCity to a latest version for instance to 3.5 because many of the new features would have helped us to do the migration in a much more simpler manner. For example there is a new concept called follower model where you can add a new member as a follower and then wait for it to catch up with the data and do a promotion. So that would have simplified our job but we wanted to stick to the best practices. For example for older versions like 1.15 or 1.14 of Kubernetes we have to use corresponding older HITCity versions. So we didn't want to deviate from those best practices. So it complicated our migration code but we wanted to still do that because we didn't want to have any surprises by deviating from the best practices. And lastly leader election is pretty bad particularly for software and particularly for HITCity. So we don't want to do frequent leader elections. Let's say you are migrating the leader again and again you would end up triggering 2-3 leader elections during the HITCity migration. So we consciously took steps to ensure that the HITCity leader election doesn't happen during the migration.  Yeah the orchestrator had inbuilt capabilities to do fine grained migration. So we had mechanisms to do migration in a given window and we had concurrency control like these many clusters are migrated at a given point in time. We usually start with one or two migrations and then we scale up all the way to 20. And most important thing is it automatically blocks the migration if there is any failure because we want the operator to jump in and mitigate the migration and understand whether it's a region wide issue or a specific cluster issue. So we automatically block the migration on first failure and another value I mean another important thing was to run a migration canary which is like a test application outside this migration window. There are so many things that happen outside the window like there are so many dependencies we depend on like block volume, DNS etc. Things can change overnight. So outside this migration window periodically run the canary and if something goes wrong we immediately block the migration and the SME has to jump in and see that okay the migration can be unblocked now. So that really helped us in many ways because there are many new changes that are rolled out which we haven't validated with if it breaks we don't want to do it with production cluster so we run migration in a given window and do the migration canary in the rest of the time. And for each stage of the migration that I would show in the next slide we had specific alarms and matrices so that way we get specific alarms and corresponding run books so that the folks can jump in the operators can jump in and know exactly what they need to do and potentially engage the right SME because the alarm and run books are very specific. So now I will take next few seconds to show how we did the migration at a very high level. So this is just one cluster like I have one control pane in the left with three compute instances running Kubernetes control plane and in the right I have one its city cluster. So first thing we do is we allocate the block volumes and DNS entries and assign to this control plane because before we moved to this new architecture these compute instances which hosted the control plane were pretty much stateless in the sense they didn't have anything associated with them like block volumes or the DNS names. So first before even we proceed with the migration we attach the block volumes and assign the DNS entries to those compute instances. Then we scale up the its city data store to five members. This is because by default we run with three but we want to be more resilient during migration so we scale it up to five members. And the black icon corresponds to the leader just to show that we won't touch the leader till the end. And then before we do any mutation to the system we take a snapshot into the object storage. We do have our core OS based its city operator taking the snapshot but we wanted to do it just in time before touching the members for migration. And then we go about moving one member at a time into the newer environment. So every time we do that we first ensure that all five members are healthy and then we go and move this first member and the same thing we do repeatedly for the subsequent members. Once all the members are moved and healthy we dismantle the objects for this cluster in the new environment. I mean in the legacy environment. And once all the clusters are migrated we dismantle the corresponding legacy cluster that we had which hosted the city parts. Now I would jump into some of the issues that we ran into. So I will start with the DNS solution issue. I mean I can talk about this for the next 15 minutes but I will try to simplify it as simple as possible. So I started off with the premise that we always ensure that all five members are healthy before we touch a member. So our assumption is we are going to manipulate one member. The rest four are healthy. So all our five members are healthy. I mean even if that member doesn't come up the member which is being migrated the rest four are healthy. So the control plane would not be impacted. So that was our premise. But in this case what happened was when we start the migration of the first member all five members started crashing. We don't do anything with those members but all of them are crashing with the error failed to update and member is unknown. So I need to set some context on this. Again like you would expect again this is narrow down to DNS but yeah blame it on DNS. But I would elaborate why it happened like this. So what we have things of okay let me go to the yeah this will give a context right. So let's say I'm moving one member at a time to the new environment. So where the member in the new environment talks to the legacy environment is through this gateway. And we ran code DNS in a wildcard mode where it looks for a suffix and if that suffix is there it will route the traffic to that gateway. The idea is in the legacy environment the part of the city parts don't have their identity. They are basically the pod names. They typically end with svc.cluster.local. So we had this wildcard code DNS supports this wildcard based DNS resolution. So we had configured saying that okay if it is star.spc.local route it through the city gateway. So a member in the new environment wants to talk to the peer in the new environment or the legacy environment. It queries the code DNS code DNS gives the IP address of the gateway and through that it talks to the city pod running in the legacy environment. But the member which is in the new environment when it wants to talk to other member in the new environment we have the DNS name assigned right. So based on that it communicates. So there is no code DNS involved there. Our BCN DNS gets the query and then it translates it and it communicates locally. But what happened in our case was let's say I am adding a new member and I am bringing up the new member. The member builds up a member table and it has a key value pair. So key corresponds to the DNS name of the peer in city member and the cluster member ID. Basically the member ID assigned to that member. So it tries to build that. First it builds it with default values which are basically Shah hash based. It creates a Shah and builds that member table and then it talks to the peers in the cluster and queries the table queries the cluster ID so that it could get the valid cluster ID. So and then tries to update this member table. So for that it uses a concept of URL comparison. The idea is you don't compare the DNS names directly you translate the DNS names and if the IP address you get is the same then these two URLs are same. So that is the idea of URL based comparison instead of string based comparison. It does this basically a DNS resolution and sees okay the two IPs are same so it's this is the right value I will update that corresponding entry. But in this case what happened was when a new member is coming and it's trying to translate it. When it's trying to translate one of its local members the query goes to the VCN DNS in some high load the DNS translation fails and so it falls back and it queries the core DNS and core DNS gives the gateway IP address and that ends up in messing up the table which is maintained by the ITCD member. So this again depends on the again this behavior is not well-defined in the library. So if you say how it G-Lipsy works versus how Muzzle works the behavior is different. In case of goal library it works more similar to G-Lipsy. So just to add more context in terms of translating right we have two things basically. One is end dots and the search domains. End dot says that the number of dots that should be present in the DNS names. If the DNS name doesn't have these many dots I will append the search domain and then do a resolution. Say for example you are trying to resolve Kubernetes.default and I said the end dots to be fine. Kubernetes.default has one dot so I immediately the library knows oh I have to query the search domain and append and do the resolution. That is the purpose of end dots and search domains. This is very common and this is the cause of many issues as well. So with the way the library is handled this is different from one library to the other. If say for example Buzzle library if the DNS name has enough dots already it wouldn't even fall back to append the search domain and do a resolution. But G-Lib C or in our case Go library what they do is first they try to do the resolution with the actual DNS name even if it has enough dots if the DNS resolution fails let's say the upstream DNS times out they will append the search domain and try to do the resolution. Because of the behavior we were exposed because we were trying to we have enough dots in the original DNS name so we are querying for some reason the upstream DNS failed and our library thought okay I will fall back and do append this SVC.cluster.local and our core DNS responded to this based on its wildcard based translation and gave the gateway IP address. So this totally messed up our table and all five members started crashing. Luckily for us we created this issue in pre-prod so we got away with it and then we adjusted our search domain so that we don't face this issue. We have created this issue against its city so that so basically how the other members perceive this is they are getting a connection from an unknown cluster member ID. So ideal expectation is when a member is connecting to you is an unknown thing you should just ignore the connection and get on with it. But here they are logging a message saying that it's an unknown member they are crashing. So we have raised a ticket against its city so that folks can look at it and we would also try to help them in this. I'll move on with other issues. So other issues are much more lighter so we can relax I guess. So we have PRTL has enabled everywhere like both for client communication as well as peer-to-peer communication. So every time a peer tries to communicate with another peer the receiver ensures that the certificate is valid. It does few things it tries to do a DNS resolution of all the entries in the subject alternate name. It also does a reverse look up of the source IP to ensure basically a DNS PTR request and ensure that it finds the entry in the subject alternate name. It works fine in a flat environment where all the peers are in the same subnet but in this case legacy and new architectures are communicating through a gateway. So the source IP is not going to match because the source IP is always going to be the gateway's IP when the legacy architecture is receiving the request. So the only way out for us was to disable this validation. Initially we had concerns maybe this is a security issue but fortunately we had our VCNs which write cyclist rules so that only these two VCNs can communicate so we are okay with opening this up and we disabled the sand validation during the migration. Another learning we had is it city aggressively does this DNS resolution so we did not remove this so our peer TLS had the entries for both legacy and new environment but once the migration is done the legacy environments DNS names are invalid but every time there is a connection from the peer it just tries to validate the DNS entry of the peer members that overloaded our VCN DNS team and they notice that there is a spike in the DNS queries coming from our tendency. So and then we fixed it by making appropriate changes. So the important takeaway is to keep the certificate free of craft and remove whichever entries are irrelevant to that environment. So I spoke about this follower model so where let's say when you add a new member what you typically do with the follower model is you add a new member and wait for it to catch up with the data and then promote it as a regular member instead of a follower. But since we are not using the follower model because we wanted to have the support of the devisions what we were relying on is the health checkup. But health check is not meant for this purpose because I have listed down the things that it does as part of the health checkup. First it ensures that there are no ITCD alarms like no space alarm or corrupt alarm and it ensures that the cluster has a leader and finally it does a quorum read. You might be aware there are two kinds of reads in ITCD. One is linearized read and other is serialized read. So linearized read is as good as the right. It tries to read from a majority, a quorum of members and then acknowledge and give back the read. So by default the health check does a quorum or a linearized read but you have an option to disable it and make it as a serialized read where it just reads from that member and respond back. So this is what it does. Ideally a quorum read should have been sufficient but we were not very convinced because it could be possible that there are some issues with the older versions of ITCD. So the way we went about this, I spoke about this raft index like commit index and applied index. So you can query those commit index and applied index from the ITCD like I can query a member and say what is your current commit index and applied index. So using that we ensure that okay a new member has actually caught up with the existing members by comparing this applied index and then we proceed with the next member. So we are pretty certain that the migration for that member is done. These are minor issues but I will still mention it here. So unlike other key-value stores or maybe other databases, ITCD expects that you have enumerated all the ITCD members when you are bringing up a new member. If there is any mismatch it will start at the very beginning. Ideally I would have expected I will talk to any one member and get the updated list. But if the list that is provided as a CLI, I mean as a command line parameter and bringing up ITCD, the bring up itself fails. So initially our orchestrator was populating this ITCD pod manifest. Basically we run the pods in a headless mode where in our control plane we push the ITCD pod manifest and Kubelet picks it up and runs the pods. So we were initially populating it with the members but let's say there are something happening in the legacy environment parts get recreated or things like that there is a mismatch and the pod doesn't come up. So we moved this enumeration as part of the start-up endpoint of the ITCD itself. Again, we didn't want to build a special ITCD image for migration because new images get rolled out and we don't want to get caught in the process. So what we did was we had init containers which come in first and push to scratch volume and then we made ITCD pick that init entry point. So that sort of thing. So we leveraged the existing ITCD image but we had additional init container which was pushing these additional configuration files for initializing the member. So one last issue I hope I will have time to finish this. Let's say when you are adding a new member so there are few things which are enforced. First is if you add a new member it wouldn't break the quorum. And another thing is all the existing members are healthy. So the second, the first issue which ensures that the quorum would not be lost makes sense because we don't want to break an existing cluster while adding a new member. But the second issue basically the restrictions on adding members kind of a bitters because let's say we are doing a migration in the new environment we are bringing up a new member but let's say something goes bad in the legacy environment and the operator jumps in and tries to add a member there. So the addition of the member fails because the HCD sees oh there are two member additions being tried. I am not going to support this member addition. So that bitters but there are ways to disable this but unfortunately the flag that was provided disables both these features like basically preventing the quorum loss as well as preventing multiple member additions. So we decided to live with it because we don't want to lose quorum if something goes wrong the operator has to jump in and manually remove one member to unblock the migration. Basically to call out that one good thing the HCD6 team does is it does a good job of back porting many of the features to the older patch releases all the way to 3.3 that really helped us because when we started out only thing we needed to ensure is for each of those minor releases we are in the right patch release so that really helped us with our process. This concludes my presentation. I have two minutes so if there are any queries about HCD or with the migration process I can try to answer. Thanks. Good talk. A couple things the whole DNS search domain fiasco I've seen this as well and I think over time we started just putting dots on the end of the host to avoid search domains in the first place from exploding the number of queries but my question is more about actually the persistent volumes and the choice to use persistent backing for HCD. So it sounds like when you are doing the migrations those persistent volumes aren't coming with you anyway. Is that correct? So in our legacy architecture we did not use block volumes because we are packing too many HCD parts in one member and we can't attach for example 40 block volumes to a compute instance. It depends again on the shape of the compute instance. So there we were relying on the scratch storage for the backend data but our backup operator was backing the data and we had an RPO of 15 minutes so it was taking the backup. So in legacy environment the data was in the scratch storage but in the new environment we didn't want to have that and the use of RPO is too much so we decided to have assigned block volume for each of the HCD members so that's why during the migration we had to create the block volumes. Yeah that's still on the lines of what I was wondering is the performance difference and the IO latency that you'd be able to get from using local MVME versus persisting everything to a disk I guess the trade-off of that versus if you lose the data you're just relying on HCDs to HA in the first place to recover that was a worthwhile trade-off. Right, so yeah like I said we do monitor the latency of the block volumes and we have the block volume provides all the IO throttling thing like if a control plane is being bombarded and there are some IO throttling we get alarmed and proactively assign additional IOps to the block volume that's how we handle this currently. Yeah I think it always feels like you can hit the same amount of IOps that I know are here attached but IO latency is there's still a pretty big gap there that's hard to overcome. I think I'm over so thanks for joining the talk and honoring me with your presence thanks a lot.