We'll start the session briefly. Thanks so much for showing up. I'm really happy to see a packed room But today we'll be talking about Something a bit specific and an eventful day that we had Kind of a year ago at CERN now So my name is Ricardo Yeah, we'll be talking about today. We will be deleted a significant amount of our production workloads and Everything that comes with it and how we we live through the day and hopefully this will be a bit entertaining as well so Welcome to our Session, I think this is a way a good way to describe it. This is our way to deal with it after a year as well So again, I'm a competing engineer at CERN I work a lot in Kubernetes containers a bit of machine learning as well. I'm also Have a couple of roles in the CNCF. I'm in the TOC and also I call it the CNCF research is a group And I work at the Cloud in Metsern Working a lot of with Kubernetes Open stack and networking So a few numbers about the infrastructure that we try to eliminate We have currently after the incident we'll describe like more than 400 clusters almost 3000 nodes and Quite a few cores and RAM as you can see like 15,000 36 terabytes of RAM In the first pie chart, you see that grouping of clusters like per group. You see that we have a few Let's say power users that have quite a few clusters and on the second pie chart You see the number of nodes that they have like you can see that the second user, which is the color is matching Has quite a few more nodes with less classes, so they have bigger clusters so about Production Kubernetes service we have at CERN. It's a central API service that users can provision Clusters, it's like one click provisioning for their clusters and then they can scale them up down at different kind of nodes And like the purpose of this service is that to allow users to create Clusters with different flavors different Kubernetes versions To select an HA control plane or not and they can select which kind which add-ons they want which CNI Which CSI drivers enabled monitoring if they want what kind of fingers controller and In our IT department and also but also other departments in the organization We have multiple teams that consume the service and they run multiple IT services each team and they have usually Some production clusters at least one QA and a few testing clusters And there are also like personal environments for users to experiment with new technologies and learn new things one of the reasons that Users or teams have multiple clusters is that they treat our cast clusters are as cattle They try to implement best best practices and not have like single points of failure Usually they create one cluster and then they are start adding more clusters and usually those clusters are behind a single or like multiple load balancers So that they can distribute the load They can do read the green deployments and they can roll out new features new clusters like new versions As you can see here for example, you can have like 121 122 they can add later 123 etc All right, so as per said we we have quite a few clusters We have this model where we offer clusters a service a bit like a public cloud provider would do we offer this internally I'll go quickly through some of the use cases we have for this The important thing is that what we are talking here is mostly about services There are some specialized deployment as well of Kubernetes that are single use case and can be kind of large scale as well I'll mention one as an example So the first example of something that runs in this service where we try to delete the full thing It's a it's Atlas is one of the big experiments at CERN for the large hydrogen collider. It's a big Physics Detector that is in the ground and they have pretty big numbers They generate a lot of data and just to have an idea of the scale of the thing We can see they actually move something like 50 petabytes of data a week They generate a lot But they also move it around between CERN and other centers and also within CERN in different places So the control plane for that and all the bookkeeping that needs to be done to do all of this is run And managed through the Kubernetes service another one is Physicists publish a lot of articles and they need a nice way to kind of find this data So this is also something we run in this service. There's a team that runs this inspire Inspire HEP and HEP data projects and this is a nice portal where they do some sort of also machine learning over the papers And then they do they allow people to discover nice contents to go through for their research topics and Then even the campus services So if any I've been I've seen a lot of people that are either still at CERN or were at CERN before during conference So if you see these logos, it probably doesn't mean much to you But if you've been at CERN before working all of this is very familiar These are all the internal services we use for all sorts of things They're kind of critical and we also run them in the service. You can see it's It's kind of even if it's just for the campus management because it's a large organization It can be quite big. So they have like a total of 400 nodes just for this in the different services All right, and then as I mentioned, this is kind of the managed Kubernetes that we offer on demand There's also use cases where they are much larger scale and more critical And things like running the the actual triggers or the event filters for the experiment I mentioned that less before you can see a picture of the of the detector there generates one petabyte a second needs to Reduce that to something like 10 gigabytes a second Traditionally, they've been doing this using a large CPU cluster They are evolving that to use GPUs and other resources but actually the big transition they will do in the Next couple of years is that we did an evaluation where we what they want to do is instead of having their traditional way of Managing applications there. They want to run a large Kubernetes cluster and the reasons for that is the flexibility and changing the workloads and Managing them. So just to have an idea of what they are trying to do They they run something like 30,000 applications at any moment when there's beam running when the beam stops They want to switch that to simulation workloads and they have to do this in like one minute or two And when being comes back up, they have to relaunch the 30,000 applications again within one minute So we we did a lot of work also with six scalability in Kubernetes to make this scale We had a test cluster that had two and a half thousand nodes and where we Cooked these numbers that you see here and we Verified that this can work. So the production cluster will actually be a single cluster with five thousand nodes That will be serving the critical part of the Atlas experiment computing So let's start going to the event that we will describe today This is like a spoiler alert in the end Things went fine, but even that they see it now. It still gets me a bit stressed But it kind of worked and those are colleagues were quite nice with us and they were very understanding so the incident was that by accident We had the maintenance tool that started to delete the full production That we had and thankfully it tried to do that one by one. So not everything in one go it actually deleted on almost 120 clusters and Thankfully this resulted Okay, initially we didn't know that it was only degradation and lost of capacity But there was no downtime for like Like no production services had downtime only if you like testing testing instances So the root cause is that this maintenance script that I described so we had implemented like a tool that we run regularly to clean up like orphan clusters and orphan clusters are Kubernetes clusters that have their VMs deleted and a lot balancers and other related computer storage resources But there are still entries in the database Of the service and this makes it a bit cumbersome for us to do operations Do monitoring and take matters of the service so regularly we just clean up similar to what the control loop would do But it's like an external tool The resolution to fix The incident was just to recreate Whatever that will have deleted, but this was quite challenging as we will see in the next slides Yeah, so I'll start with this so we'll go through the timeline of the day So the day started as most days start at CERN, which is over coffee and a lot of discussion This was early in the morning over coffee. We usually talk about dark energy dark matter and the expanding universe Actually, not really This particular day. We were actually discussing the change. We were about to apply Even if we didn't feel it was like a big change, we still review what we are about to do during the day So first we'll give a bit more details So there were discussions like, okay, everything is fine. Let's do it. It's a small change. It's nothing big we have tested this and All all the return codes and tests are correct We spent two three weeks validating this so everything will go fine. That was before lunch Next So at 2 30 we said, okay, we'll go for lunch. Everything is fine. And then we clicked it and we started and Well, we were waiting a bit for the tool to kick in and then we will start monitoring the logs and A few minutes after like one or two minutes after the start of the script Like one of my colleagues from the identity service came in and said, I think we're hacked Like all the identities of the clusters are getting deleted. I'm not worried. It's fine And we are just doing this to clean up the orphan cluster. So it's expected And then the other colleague that the registry The container registry runs in a Kubernetes cluster that we eat our own food He said I don't see the registry clusters. Something must be wrong. So I was a bit worried so I clicked the abort button to stop all of this and Then I tried to list the set of clatters because I thought okay, maybe my colleague was looking at the wrong project Or was doing something wrong in his environment But it turns out that it wasn't and the next minutes were felt like a bit like the next slide Like this one it lasted a few minutes in this mode and then I I passed by Ricardo's office and Yeah, so the next few minutes after this This is a very important point in any organization and CERN takes a lot of care about the safety of everyone So this is actually a screenshot from the learning catalog at CERN Anyone can apply for this training. It's first aid and life-saving. It says things like how to react in case of external hemorrhage Wants loss of consciousness, which is pretty much what was about to happen here So it's also very important that the target audience is all personnel working at CERN which covers also IT people So this is what what we had to do for for for the next couple of seconds at least This is an actual picture of the defibrillators and and fire extinguishers You can set can find at CERN so this lasted a couple minutes then we kind of calmed down and Started looking into the issue So around like what is important here is that we started this operation at like 235 239 we detected it We got the first report from a user about something is wrong like really wrong around 41 so we got this we were also getting alarms and things but those We kind of were ignoring trying to understand exactly what was happening because there was way too much coming So some user reported that the UI for the registries failing for me. This is okay. Something is big So we spent the next 45 minutes after this trying to do an assessment I understood okay something went really wrong How do we how do we deal with this and this was really important to kind of structure this So the first thing first thing to see is like what's still up and what's not so we immediately Identified that one of the services that was down and actually the only service that was down Was the registry and the reason for that is that? We actually got unlucky. Let's say like this and we deleted all the registry clusters behind it Which ended up bringing down the service now Normally and we've tested adding new clusters. This wouldn't be a big issue And we could get to the cluster back and and the service back in a couple of minutes In this case, it wasn't quite the case and we'll go into a bit more detail of why this was the case In in this event at the same time We were lucky because we run the registry clusters that it was the only service One thing that became very clear also after this experience is that the registry is really like completely critical It's important to launch new new applications Of course like for stuff that is running and that we didn't delete They kept running without a registry But if you delete the cluster of users and you ask them to recreate it's kind of critical to how the registry running So this was the first priority and what we did is we branched out the team So there were people focusing on okay Go get the registry back and do whatever is needed and the rest of us We started doing the right what was needed on the other side Which is basically this is way too big to to do the normal procedure So let's start contacting the different teams at CERN to fully understand. What's the impact in their own service? And this was really essential So the COVID era actually helped a lot because we have this direct communication even using zoom With many of the services and we basically established this communication. We had quick calls with them. What's up? What how can we help is it down and what we started realizing is that? No one was down I think there was one case of a service that was down but most service we're saying okay We are still up. We're degraded, but we're still up. So this kind of helped us kind of relax a bit and the reason that that These services were still up is again back to these clusters a schedule and the dissemination we've been doing internally For for best practices on gate ops and automation and I show here like the our best user It's a team that manages multiple services. You can see here Actually, I see the person there. So it's even better and this is a one one of the Like the most well-behaved user and you can see what the impact of this event was for his own service Or that the team services you have like for each of the services they're maintaining they have something like eight clusters in total we deleted six and Still the service was still running. So for the other ones in some cases none was impacted but in no case we actually had a loss of complete capacity and This is really a shout-out to everyone in the certain teams also for for doing the right thing This took quite a long time to contact everyone So really it's it was really important to branch out and do and split the tasks You can see on the right plot. You can see a similar view in number of nodes Of course the services were not behaving as expected But but it helped quite a bit So yeah, as Ricardo mentioned like the first part was to bring the registry up And and we didn't realize that we had created the circular dependency in the Kubernetes service by introducing at the Harbour registry before the Harbour registry were using the GitLab registry and Then we set up Harbour and we were advertising to users to consume it set up like rules for making immutable tags and Using a vulnerability scans etc And then we thought we should use the same best practices and we started using them But by doing this we created the circular dependency I mentioned so it was quite critical for Our colleagues for our team members that were at the task to to bring it up and they managed to do it quite fast we were also Kind of lucky because the load balancer was not was untouched So the DNS name certificates all of that would be easy To bring back so this lasted like 40 minutes. I think or something and then I Users would be able to start recreating their clusters and next slide Yeah, so usually this would take like maybe five to ten minutes to bring it back because of the circular dependencies and some Worker runs we had to apply to something like 45 minutes And so for the majority of services as soon as we notified users that okay Everything is fine now for the registry you can move on like adding capacity to your services back Most of them took them like 15 minutes maybe 30 some and then at the end of the long tail There were some cases that needed more more time because they had special firewall rules that are not managed centrally Like DNS name propagation that were kind of special and manual manual steps Which shows that everything that involved manual steps was a bit slower because the services were was built like that In a very long time so in the incident they had to recreate everything very fast and then we did a survey to understand what happened and how it was fixed and how everything came back and Really get ops it helped a lot here and as we can see with the survey even for a cluster provisioning There were quite some best practices used by Bar by our users. They were using Terraform Others wanted to use Terraform But because of the setup of their cluster and the lack of Features supporting the provider and they were not using it, but they wanted to Crossplane also is the ultimate goal, but it's not available yet for our environment Other users had very detailed documentation for their own team how to recreate everything or how everything is built So they managed using the existing documentation to recreate everything very fast even though they don't have automation but also for the application which was Quite a success about implementing best practices like almost everyone was doing github's With our goes your flux V1. This was one year ago. Now a lot of users are using also flux V2 and other users had helm charts, but everything was packaged and configured in the values, so it was very easy to reproduce the deployment and That's why they were able to bring up the capacity in such a short notice Yeah, so another thing to highlight here is that you see how diverse some of the tools we use are and this is because we have very diverse teams We have best practices and for like the most critical services. We we kind of follow the same Policy in terms of tools being used, but at the same time different people have different requirements And the teams have different types of knowledge, so we don't currently enforce the the kind of tool they should use So the the rest of the day, so now we are end of the afternoon We we spent kind of looking at the tail of the issues For most services people Realize this pretty quickly and they reacted quickly. We did realize that some things that were impacted Didn't get an immediate reply which probably means they are less critical But we wanted to make sure that this was followed as well. So this was kind of the tail of the work here so as a summary of all of this There are some highlights to do there are things that went quite well the main one is that we had no data loss This would be the main problem Why this happened we will talk later a bit But it's it's also like not we we don't have well-defined policies to to guarantee this unfortunately yet The deployments are really well managed. This is really something we invested quite a bit of time in dissemination of GitOps Processes not necessarily enforcing a tool but making sure that people are aware that this should be the case that they should be able to deploy Their applications very easily There's something that is really good In one way which is cluster creation has been optimized over the years and it's really fast So it takes only a couple of minutes to get a new cluster, which means even in an event like this Which is pretty big we can get the capacity back pretty quickly Multi-cluster and cluster as catalogs and workload splitting it does work like these are the cases where we kind of even if we go Really hard on it reducing the blast radius for the full service is really a big thing We ended up with degradation and almost no downtime Direct communication. It was also very important And on the what went wrong side The singular dependency was the biggest issue and that caused the most stress to us after the initial assessment But also what went wrong is that clustered deletion is also optimized and it's very fast and successful So I don't know if we should do something about it to make it less Less efficient and then other things that went wrong is what I mentioned that some some changes were manual like DNS updates firewall rules, etc And also what we are slightly lacking behind apart from the use cases with Terraform is cluster bootstrapping is a manual process So if we have something that Consolidates all the time Also that would have been addressed by itself like they should have been others by itself Yeah, so this is this is one of the main investments we are doing and we'll talk a bit more about in the demo Finally like the where we got lucky the final for the final part of this post mortem is We kind of got overconfident about what was clearly seen as a small change We knew that the impact could be big but it looked trivial enough to and well tested enough to To kind of let it go and this is something that is constantly in our minds in the in the last few months So we are a lot more careful and we define some criteria on how to follow these changes The other thing we get lucky is that we identified this circular dependency What happened is that we wanted to consolidate the service. We ended up running the registry in the service itself Just kind of grow grown by itself this idea without verifying necessarily So what we did after is really we did a Detailed analysis of where are the other circular dependencies that we missed over time So this is an effort that is quite important as well The final one is that we have no visibility on data persistency or backups This is nice in in Kubernetes You have this possibility to declare snapshots and even generate backups This is not something we have in place right now for some limitations of the integration We have with the storage systems and it's where we are also investing a lot because this will allow us to look at the clusters and See where the data is and what's backed up. What's not and kind of trace To prevent data loss in the future in this case. We did not have any and another part that we invested some time apart from making The cluster provisioning like more automated is to change a bit our policy and never like ever delete Anything on behalf of the user if they even if they request about it We should always delegate to them and tell them how to do it So we never like we block ourselves from even being able to do it Right so this this will be a very quick demo I must confess that initially the idea was to do a real live production deletion of real CERN service and discussing with colleagues involved and not involved in this incident They said if this is a therapy session, I don't think that's a good idea because it might just ruin it So we'll do the second best thing I'll try to demonstrate a bit what we Try to advertise as Automation of the cluster management This is really the part that we're missing all the services are well well done And we understand well how to to manage them the cluster part is still a challenge Ideally we'll get to integrating this with cross-plane as well very soon So just a very Simple description of how this works. So in this case, we are relying on Argo CD We have this bootstrap, which means we can pop up a cluster Deploy this in the cluster and we'll have an Argo CD deployment Which also has a secret management service internally and manages itself Argo CD So that's good We also deploy Argo workflows in this case and this is because there's a lot of manual tasks where You might want to go and write a CRD and a controller and do proper reconciliation, but that can be quite a lot of Work and not all users will will be willing to to develop and maintain those So the second best thing is just to offer them a way to integrate it with workflows Then we have the clusters description. So this is the dream It's to have one YAML file where all the clusters are defined or a couple of YAML files And you can see here like I have a couple of clusters And for each cluster we have the configuration which is basically the version that we want to run and then some flavors for the nodes and the masters as well as Node groups definitions a lot of these clusters are heterogeneous So we want to create like nodes in different availability zones or different flavors GPU CPUs things like this And then something that is really important is this part here You can define Argo CD labels, which are basically the type of workload that this cluster should be running And this will allow us to do a kind of a matchmaking With the applications without having to do a hard coding of the the mappings and you can see other clusters here if we look at the existing Argo configuration we can see the clusters here I have 124 125 and I have an extra one that has nothing deployed right now So and you can see here all the details about how they are deployed again We don't have crossplane integration yet. It will come soon. So the second best thing is to use a workflow for that And then we have the services. So if we go here quickly For a service, so this is a very Simple example of what the service could be like. This is a serving service for a machine learning application The the really nice thing here in addition to all the things I'm sure all of you that use the Argo know about Is that we are using this label selector so that we can match the workloads to the clusters that are saying please give me this type of workloads and this is really a separation of That is very useful in this case And I think that's it for the description. So what I will do here Again if I look at the applications and I select Yeah, and I select here quickly Specific project we see we have two instances of this ML service application. It's in two different clusters This is what we want to provide. So what we will do will do two things. The first one is we'll add The same label so this ML true, which means deploy ML applications in an additional cluster And at the same time, I will also deploy a new cluster. So let's call it 10 I'll just keep the same configuration to speed up things and I will say that I also need ML in this cluster So now I'll just quickly add this push it and just for all right there and if I go back to To my clusters I do a refresh quickly and you can see that it's starting to to to launch this new workflows And there's two one for the previous cluster because it needs to update the labels And there's a second one which is deploying the new cluster, which is this 010 cluster I just showed so in the new one we see the first pods here starting Fully quickly at the logs we can see that the cluster is already in progress So this takes a couple of minutes and because we want to leave time for questions What I will show you quickly is the changes that are being applied in the other cluster Where it's basically saying It's applying a bunch of resizes so I'll come come come back to it when it registers it as well So the way the workflow does is it tries to create the cluster if there's a cluster just ignores it And then it tries to go through all the node groups and this is what it's doing like for each node group So in this case this was node group Yeah, it's trying to resize the cluster and then applying all the changes again Ideally we should have this done with the proper Reconciliation so that is applied faster workflows are the second best thing and honestly There are a lot of processes internally where it's much easier to offer workflows and offer this flexibility to users While we wait, I'll just highlight from the Argo documentation this when you start doing application sets Which are the key here. There's this different generators One of them is this cluster generator and this is where the magic happens Which is you can define your application set and then have the labels saying where where the application should be mapped So let me see if this will be fast enough to To actually appear Yep, so this is the one It's about to change the labels. It did the update ml true. So if we go back to applications We'll come back I need to turn off the CERN. So sorry about that If we come back we see actually an additional ml application there If I do ml instead of 2 now we have the application running in a new in a new cluster Fully up and running and it's just a couple of minutes so this is really what we are trying to advertise to the users and Convince them that this is useful for their own sake as well So yeah, it worked in the end All right, so I think that's pretty much it what we have and we have a few minutes for questions So happy to to answer. Yeah. So just ending here with another quote from from our nice users that say in the end It was a chaos chaos monkey test and we kind of passed it. So it's really nice So yeah, any questions Yeah, I have a question regarding I imagine that you had persistent volumes in the clusters and I know that for example using sef you have a reference with the image id But if you lose the cluster then you how did you recover that image id to To recreate the the persistent volume Connected to the right sef image Right so for for for sef, there's two steps. One is the provisioning of the volumes. The second one is the Attaching of the volumes. So in most cases people use provisioning, but then they reuse the provision volumes meaning that you can actually Mount the application in multiple Clusters if you're doing rbd, this is much more complicated So that's why actually the usage of sef fs at CERN for Kubernetes is much larger than the usage of rbd itself Because you can have this multi-attach much easier Okay, so basically you didn't have These rbd images, right? There were cases, but you can also have multi-attach in rbd, but it's done differently It's like kind of a active passive mode instead of having multi-attach. Okay. Yeah. Thank you Quick question. How did you end up solving your circular dependency on harbour? So what what's the Solution that you chose? That's a very good question Should I answer? So Yeah, the solution it was It's like kind of obvious like you can have a second backup of the images outside in another registry So this is in the end the the solution Yeah, and the service the service itself allows us to specify the registry where the core images should come from So we can flip that when you launch a cluster you can choose which backend registers should be used for the core images So right now we have the core the main registry We have a replica on premises and we have a replica of the core images in a external registry Outside CERN as well for for a disaster recovery. Thanks makes sense. Yeah It's good question


Live, this is Think Tech, and we're doing Community Matters with Robert Sue, and Robert is with the United States Census 2020, doing his job. Welcome to the show, Robert. Aloha. Thank you so much for inviting me to the show. Thank you. Well, I mean, I think it's very important we talk to people like you. Lord knows you've been through some issues in the census. The administration has intimidated a lot of people, trying to scare them away from participating and answering questions candidly in the census. I'm reminded of the move on the Hispanics, the Latinos a year or two ago, which I'm sure it's scared a lot of people away, and they're still scared away, too. That hasn't changed. And then it went up to the Supreme Court a couple of times about certain questions, and I'm not sure that the administration has actually changed, even in accordance with the rulings of the Supreme Court. So we have a census that has lost public trust, lost confidence by the people. This is very important. We have a census that has the confidence of the people, because the census determines a lot of issues, including money and voting boundaries and so forth for the next 10 years. And actually, that's a long time. So it's very important that workers like you, A, get good information the best you can, and you can override all the intimidation, and B, that you build public trust in the census. So Robert, it's really important that we talk to you. Tell me what you're doing in the census in order to build trust in the census, in order to get the best information in the census. Sure, yeah, my role as a partnership specialist, basically, I was sent from L.A. Regional Office, tried to be a liaison, a bridge between the community here and the Census Bureau. And it's so important because, as you just mentioned earlier, without a public trust, people are scared. I try to convince people saying, based on the title 13 in the federal law, everything is confidential. And I'm here with everybody. I don't want to ruin my reputation by showing my sincere attitude, showing, don't worry about it. You have my promise, and I will try my best to protect everything. And I'm your bridge. So I'm not here alone. So your voice can go through me to be heard by the top authority. If you don't speak out, then nobody will hear you. So that is my main role as a liaison, the bridge, and to communicate with people, to bring back a public trust. I think that's always a trouble. Yeah, well, yeah, I mean, and we saw that happen with the Latinos, and they ran away because they didn't want to have ice knocking on their door in the middle of the night. In some kind of raid or roundup and ruin their lives. So they didn't want to answer questions. And then it was a question about citizenship, the same thing. And people really lost confidence there a couple of years ago. But what we didn't hear about is the Chinese thing, because the administration is taking all steps possible to criticize, to mislabel, to make it hard for the Chinese from China to operate in this country, and to be part of the census. So you have a special role that goes parallel to, but perhaps even beyond, what happened with the Hispanics. So what is the feeling in the Chinese community? Who are we talking about? Who are we worried about when we say we got to build trust? Well, again, we're talking about Chinese. They are Chinese that are new immigrants. They are Chinese locally here, maybe more over here, a couple hundred years ago. So in the new immigrant, they are Chinese from China, the Chinese from Taiwan or other region, Hong Kong, Vietnam, or whatever the overseas Chinese. So everybody has a different historical background, and they have different concerns. And I think early this year, actually since last year, you can see the very popular news about what happened in Hong Kong. I believe most of people in Honolulu here, if they're from Hong Kong, they will worry about their family back in Hong Kong, what's going to happen to them. So because of the agenda, a lot of those, especially new immigrants, they keep low. They try not to expose themselves. They keep silent. And that's the nature, the ethnic background. Because Chinese people, you did more self-conservative in the history for five thousand years. They don't care who is in China or the country, all they care is, as long as my family and myself is safe, we keep quiet. Interesting. What you're saying is that the pressure, the intimidation is actually from two points of view. One is they should say something that would offend the Chinese government in Beijing. And as the Lord knows, there's enough things you can say to offend the Chinese government in Beijing. And then the other side is they can say something to offend the Trump administration here, because he doesn't like Chinese. So you get it from both sides. You have a very difficult job, Robert. So we stuck in between. You're right. Take a sandwich, we stuck in between. So we had to be very careful to handle how we say we don't want to upset people. We don't want to intimate people and then more upset about it and walk away. So what do they say to you? Look, Robert, thank you for your advice and consultation. We really appreciate your efforts in this, but we want to lie low. And what do you say back to them? Well, actually, to be honest, in past year, my experience when I go to the community to talk to people, I would say more than 95% of people, first time heard about census. They don't know about it. They don't know about the consequence of the census to them. So awareness is a problem. It seems that there's a public education awareness issue. So the first couple of months is to build a trust. So they open their heart and they can talk to you. They are waiting to talk to you. So I think that's a big change. So now they understand. We're not doing this for the government or for government in USA or the government for China or China or whatever, but we're doing for ourselves, for our children, for our next generation. So now they understand. So I'm very happy that the outcome because after the first two, three months to build a trust, that is the most difficult part because they look at you. They're scared. No, no, we don't trust government. I don't want to talk to you. But now they voluntarily invite you to participate in the activity, especially during the COVID-19. You know, it's hard. They send me the Zoom or WebEx invitation. It is, I build a trust. I think that's the most successful one. They know I can trust rubber sooth. In the way they trust, you know, bureau sensors, which I feel part of it, you know. But what do you say to them when they say, look, this may or may not remain a product because the government gets all this information. The first, the United States government gets it. And they're supposed to use it in a confidentially. They're supposed to use it to develop, you know, to develop the distribution of funding and voting boundaries and so forth. But how do we know that they won't pass it off to some other agency? For example, Department of Homeland Security Immigration Service. How do we know that? What do you say if they raise that question? Well, I reassure them, this federal law, Title 13, you guarantee and assure nobody allowed to touch the data from Census Bureau that it doesn't matter you are FBI, CIA, Homeland Security, no any law enforcement, including even the social welfare, they cannot use your data to disqualify you because everything is 100% confidential. Even myself, if I say something, I could be sentenced up to five years in jail or $250,000 fine. It's very, very serious. I use working for the tax department, you know, before under Title 26. We're supposed to secure all the tax payers information, there's a lot of confidential information. But the tax department is not as rich as Census Bureau because if we get court subpoena, CIA, FBI, law enforcement subpoena, we can submit your data to them. But Census Bureau cannot, based on Title 13, everything is confidential. And on the Chinese side, on the Beijing side, it's very unlikely they're going to be able to the Chinese government unless they hack into the system. They're not going to be able to get the data about what people have answered on the census. Am I right? Yeah, that's right. Not get it from Census Bureau.  And that's a scary part. And when you hear people missing when they travel to China, well, because you're under the blacklist, so there's got to be something that's going on. I don't know, but I think that's what people worry. So I heard recently, I had some people up from China, the new immigrant, they say, you know, they got to watch out because it's still Chinese citizenship. They only have the green card here. So if they become blacklist, not only their family might suffer in China, also when they get back to China, they could be in trouble. So those are the people that are more scared, people from Hong Kong, people from China. So I don't see other regions have problems. Yeah, that is scary. So are you able to convince them it's okay? What kind of response do you get? And are there holdouts who, you know, are not assured by what you tell them? Sure. I basically explained to them. The 2020 census, the questionnaire itself, it doesn't involve anything sensitive. It's very basic. It doesn't even ask your social security number or your income, right?  And those children harm you at all because we don't touch something sensitive in the questionnaire. And that's up the record. No, that's no problem. So I just make sure nothing will be asked about your personal confidential information and nothing sensitive. You won't get trouble back to China. Say, I have a three, three, you know, family member in this household. No, those kind of questions is very simple. You know, so very straightforward. At least they understand I can explain to them, no, all this, you know, it's very, very safe for you. So how, you know, you're communicating with people. How are you doing it? I mean, we are, I am reminded. I reminded every moment that we are in COVID now. So how do you connect with them?  Do you talk on the telephone? Do you send mail? Do you have Zoom meetings? What do you do? Yeah, actually, to be honest, I have to mention, when I first started, that was before we shut down. And that part, I feel like the trust in the first two, three months, you know, and, you know, since then, one of the key things for them to build trust with them is, I speak the language they speak. Yeah, I don't come in here with official beige, you know, I'm from a federal government, I wanna talk to you. I come here to talk to you as a friend with your mother tongue language. So if I go to Mandarin speaking area, I speak to them in Mandarin. I go to Taiwanese speaking area, I talk to them in Taiwanese. So a lot of people told me. How about Cantonese, you speak Cantonese? Cantonese, I don't really speak that well. But, you know, interesting is, we have a same character. You know, the first emperor, one of the biggest contribution he has for the country, for the nation is he unified the language. It doesn't matter how, what kind of dialogue, we write it down, we're all the same. I can show them my writing, they can understand. Yeah, good, good, good. So that's a good part. We write it down, everything is the same. Just pronounced different. Yeah, anyway, okay, we have a slide show, Robert. Why don't we go through the slide show and you can tell us more about census 2020. Sure, yeah. So let's go next, yeah. I think the next screen, you know, that explains, we just talk about that. Why I put a slide there, two different big group of community. One is the Chinese Mandarin speaking group. The other one is Taiwanese speaking group. And you don't wanna miss anybody.  So that was the reason I put that slide. Go next, please. So the first one we know the biggest Chinese group is the local Chinese immigrant to Hanoi, I don't know how, move, you know, to Hanoi, Hawaii probably 200 years ago and they have a lot of people here. You know, they have many, many generations and Chinese Chamber of Commerce is a key. That is the first organization I got approached because under Chinese community, Chinese Chamber of Commerce, there are 100 organizations under them. So through them as my bridge, they can coordinate and communicate back to a lot, down to a lot of different organizations down there. The picture showing the other one is the former president Michelle, she is the president of Chinese Chamber of Commerce. When I first get on board and she's been very active, supported it. The lady next to her is the current president of the Chinese Chamber of Commerce, Elvira. She, I think she started from June as a new president of Chinese Chamber of Commerce. And they both are very supportive. They send out a newsletter weekly, even monthly for Lenten, the news article to remind everybody important and census and you got to do the census. So Chinese community is very, very supportive. Again, they are very excited too because just the first time ever have a census bureau of people come to talk to them. So they appreciate it. We recognize them as a one entity there. Yeah, that's a good thing. So how do you identify them? I mean, if you start out on a given Monday morning, how do you find and connect with the Chinese people or your group, so to speak, to talk to them about the census because they're not going to come to you necessarily. How do you get to them? Well, I actually, I did some research and come out. They actually, first of all, the well-known Chinese people here in town, like Chulang Kwok, she's very active in Chinatown area and is a very good old friend. I know her 30 years ago. So I have a meeting with her. She gave me a one book called Red Book and that has a whole list of Chinese organization in the books. They listen out and from there, I also opinion which organization have what kind of special unique features, temperatures. So I put a note in what organization, what I can talk, what I should not talk. So it'll be more sensitive. So from there, I get in touch with the key. Again, after that, Chinese Chamber of Commerce is the first one I contact to. And they are very friendly, very nicely because I'm not here to steal any confidential information as a spy. So I build a trust. Of course, it takes some time, but I say two months for me to build a trust and I'm very happy about it. I told my friends, you know what, after census finish, because I'm walking in China so much, I will be a mafia in Chinatown. You told everybody about Robert Su, they would know. Yeah, I know that's guy we're sure and tight walking around the town all the time. You're going to have a lot of friends over there. Well, a lot, yeah, exactly. So now they all know, oh, that's a census guy. Go ahead, Robert, go with the slides, more slides. So the next one is, when I talk to Chinese Chamber of Commerce, so their biggest activity in the year, of course, Chinese New Year, that's a very big one, but it's a very crowded, as I put it, something more unique and people may be more interested and not see the screen. And it was very interesting people in my life. This is the first time in my life participate the beauty pageant like this. So we were offered by the Chinese Chamber of Commerce to have free booths and we can make a presentation, talk to people over there. And the most beautiful part, we can take a picture with a beautiful lady just elected over there and they hold the census sign. I think that's very good, yeah. So it's very good publicity, you know. So that is one of the very good thing that happened. The other one is, we know, as you mentioned, nobody expected COVID-19 will happen suddenly like this. The good thing is, before that happened, we already encouraged, because of new technology, we should not just doing only the in-person, you know, promotions, something we can do, which use the internet, the web. So Chinese Chamber of Commerce put a front page as a link directly to 2020 census. So that allow everybody if they're interested in the above census or you want to respond to questionnaire, you could link it automatically. I think that's something very, very good by using technology. Yeah, yeah. So the census form, I guess it's online, but you can also write on it on paper. Do you ever deliver it to them? Do you ever help them fill it out? Or do you have other people who help you help them fill it out? Actually, we don't have the paper form. The paper form actually was sent out after Census Bureau didn't receive a response, because this is the first time ever that we take online response. And this is very unique, very nice that we apply the new technology. So up to today, we have 62.5% response in the state of Hawaii. Does that mean all the whole state, every group, or does it just mean that you're Chinese community? No, the whole state, every group, total our response rate, the sales, we call it the sales response rate, is 62.5%. And if you don't respond after a certain timeline, Census Bureau will send out, first of all, they send out invitation later. That invitation later has a Census ID. You can use ID to lock in, to website, to file online.  So we don't get a paper form. It's the household member, if you don't respond Census online or by phone, the other one you can use telephone to response the census. So, and then if you don't do either online or by phone, then you will receive the paper form. So I offer to those community partners, through the newsletters or email, put my content information, if any time you have any questions, you are free to give me a call. So that's how I can support them after we shut down mid-March, because of COVID-19, we actually can call over the phone. I can help those. And the organization, they will call meeting by using Zoom or WebEx for video conferencing. So I can present and show them what to do too, yes. Until the end of July, we actually recruit a lot of census takers. They are the one actually knocking the door door door-to-door to help. So my role, I don't go door-to-door, I go straight to the organization part and to provide education, awareness, communication, build a partnership, and then through their help, because you have thousands and thousands of Chinese or Taiwanese or Chinese here, you need to use hierarchical organization partner to help you to spread out the word. And again, they are the trust voice, because they go to the temple, they go to church, they trust the pastor, they trust the priest. So that's how we hook up with community leaders, with their trust voice to bring back more people to participate in. Do you have any idea what your percentage of success is within your community? You mentioned that 62% of the population has responded, of all the households has responded. But what about the Chinese community? Can you say whether it's 62% or more or less? Well, at this moment, we cannot see the data yet. The current online data, we can see it. We can see by the census track online, we can see what census track has how many people, population, and what is a sales response rate. And that's how I get involved with Senator Moriwaki, she's on district 12. I use the program I designed to show her, Waikiki is your district, but it has very, very low response rate. Can you help? And start from there, she's very proactive, she's tried to help, and very nice of her. Terrific. Well, what about the other countries? What about Korea? What about, I don't know, Let's see. Japanese. Japanese, right. What about Southeast Asia? There's a whole bunch of countries. Do they have representatives like you for those other countries as well? Yeah, we do have a partnership specialist here who speaks Japanese, and also Filipino language. And we have Hawaiians and Samoan Pacific Islanders. So we total have about seven partnerships special this year in the whole state of Hawaii, and a couple of them in neighbor island to help neighbor island. But if there's a need, we fly over to neighbor island to help. So you know, the census was supposed to last, you have to correct me on this, but it was supposed to last to near the end of the year, I want to say December, but it got cut by a month anyway by the administration. And so the window is going to be closing here pretty soon. What is that? At the end of October, is it? It's going to be cut soon. So what month is it? Can you tell me?  Oh, okay. And you know, we are not allowed to comment. All we do is we try to encourage everybody, try to respond now, you haven't responded yet. That is a key. Don't wait. Oh, there's another month or another week. If you do it now, it doesn't matter. And that's a key point. So we wait for, see what is the final just, you know, judgment from a court and decide. But at this moment, we try to keep the word out. You know, it's not too late. I don't need a response now. You don't have to worry about when with a sense of because, yeah, so. Sure. But you know, it just strikes me and you can, you know, you can agree or disagree that if you cut it off early, you're going to have a, you're going to reach fewer people. This is not as much time available for you to do your work for people to respond. So by definition, you're going to cut off a few people at the end. Am I right? Well, theoretically that is true, but amazing the team did a great job. We actually, we're talking about enumeration. There are 62.5% that did a self-response either online by the phone or through the mail. The rest of them are rely on the censor takers that go door to door, right? Even though there's some sort of challenging, you know, issue happening because COVID-19 people refuse open the door. So, but so far after this morning's records, total we have 99.4% of household has been enumerated. But I want to say enumerated is at least somebody already to communicate with them whether they respond or not. I don't know. Yeah. So, but we, What you can do is ask, yeah. Yeah, exactly. At least we've been a message. We remind them, encourage them, but we cannot force them. This is a free country, you know? So, we respect their choice, but amazingly is just from July 30th, we start on this, you know, non-response, you know, census taking, you know. Actually bring out our total enumeration. I don't say response rate, enumeration rate up to 99.4%. That's really terrific. Now, are you working in more than one district at the same time? You're working here obviously, but are you also working in California and in Los Angeles doing the same thing? Is that included in your area of operation? No, actually I'm focused on Hawaii. We do have a weekly or monthly meeting to exchange knowledge experience in other region. And originally we gather together in headquarters.  There are a big Chinese group in Los Angeles, right? Even other state. So we cover seven state in the Western regions for the LA regional office. Actually, LA regional office is the biggest one in the nation. That's what I heard, because the population, we are the biggest size. California, a lot of people there, so. It's a major thing. It's constitutional, it's important. I'm so glad you're doing this. Query, though, what will you do when it's over? I mean, I don't know when it actually would be over for you.  Where do you go, you stay with the census all through the 10 year period or you go do something else? Well, yesterday I obtained the statewide, the biggest, you know, census CCC, we call that the Complete Count Committee. That was shared by the Highway Committee Foundation. And I have mentioned to everybody during the last conference that we have, the monthly conference, I share with everybody, say, you have all my content information. I even give everybody my personal email, my personal cell phone numbers. I'm still around. And even until 2030, any question I try my best, it is for our community. When I engage myself into this, I'm engaged myself to the committee service.  There's one lady, 94 years old, and she came out to talk to me after my presentation. You know, she came with her, give her a hug, she said, Mr. Su, this is the first time in my life I never heard about census. Thank you so much. At least I can do it now. So I say, so great to hear that. So actually I told myself, census finished, but myself, my spirit, you know, my heart is still with everybody in community because I build a trust.  We are friends forever. And that's all, you know, I'm trying to share. So I believe it kind of takes a while to finish the whole census. Even though whatever judgment by the court, I'm not working in the Census Bureau, but I'm still, I actually send a note to my boss in LA regional office, say, if Census Bureau doesn't have a budget, I volunteer to be a point of contact here. I want to be a bridge to build the people in Hawaii to communicate census, you know, bureau headquarters. And don't worry about budget or pay. I volunteer until one day you say, oh, Robert, you know, we have budget. You did a great job, you tell me. But that's my mentality. I want to be sincere to show my care and love to our community, to the people here. So yes, I will be a forever ambassador for Census Bureau here in Hawaii, especially the Chinese community.  You know, my IT background, I talk to a lot of big IT company that even neighbor over there, Oceanic, they are very supportive. PR fortress, the biggest, you know, the networking company, e-world, the application company, you know, data house, all those company, they all know me. I was CIO for tech department. And you know, as my personal philosophy, when you deal with when you build a trust sincere, not just the business, it's a forever friend. So, and that build asset for me. You know, I learned from that, that build asset for me. So this census, I build a lot of assets and I don't want to let go like that. I want to continue to serve our community as long as possible. That's great, Robert. I am touched by your sense of community and your willingness to volunteer. I am truly touched. Robert's showing me the census 2020, an extraordinary man in an extraordinary time at doing extraordinary things with this country. Thank you so much, Robert. Yeah, thank you very much. Thank you. Hello. Hello. Hello.


In this video, we're going to present the solution to question 11 from the practice midterm exam number one for math 2270 and so we were given a transformation which goes from z7 to To z7 3 and it's given by the following rule T of x1 x2 is given as x1 plus 4x2 2x1 plus 5x2 and 3x1 plus 6x2 Now our function is something that's going to be working mod 7 here So we go from mod 7 with two entries to mod 7 with three entries But the fact that we're working mod 7 really isn't going to have much bearing on this calculation whatsoever So for this consideration, let say x be the vector x1 x2 and let y be the vector Y1 y2 and so these are vectors that belong to z7 squared And so what we're going to do is we're going to take a look at t of x plus y to show to prove that this is a linear transformation We have to show that the transformation preserves vector addition, which is what we're going to be doing right now So notice that t tx by give above is just x1 x2 and y is just y1 y2 and So adding these together we end up with the vector x1 plus y1 and then x2 plus y2 and So now we're going to apply the definition of the Function itself. So this function will create a vector in z7 3 so there's three components where the first component you're going to take the first entry plus four times the second entry So we're going to get x1 plus y1 plus four times x2 plus y2 So for the second entry, we're going to take two times the first input plus five times the second input So we get two times x1 plus y1 plus five times x2 plus y2 and Then for the last one we're going to take three times the first input plus six times the second input So we get three times x1 plus y1 plus six times x2 plus y2 so that's what it means to evaluate the Transformation at the input vector x1 plus y1 and x2 plus y2 So what I'm going to next do is distribute these coefficients. So we're going to end up with an x1 plus y1 plus 4x2 plus 4y2 Next we're going to get 2x1 plus 2y1 we're going to get 5x2 plus 5y2 and then lastly we're going to get 3x1 plus 3y1 plus 6x2 Plus 6y2 like so. So we just distribute these coefficients. I'm then going to add together The common x's and y's right so notice we can break this this thing up in the following way We're going to have an x1 plus 4x2 We're going to have a 2x1 Plus 5x2 and we're going to have a 3x1 Plus 6x2 so I just wrote all of the x's in a single vector And then I'm going to do the same thing for y you have the y1 plus 4y2 We have a 2y1 Plus 5y2 and then lastly we have a 3y1 Plus a 6y2 Which the first vector listed that's just t of x x1 x2 and then the second one is just t of y1 y2 which We know whoops, which we know that's just going to be t of x Plus t of y so we've now demonstrated that this transformation preserves vector addition We wanted the same thing for scalar multiplication, so we're going to take t of c times x well x remember is Just the vector x1 x2, so if you times that by a scalar you end up with cx1 and Cx2 and So let's apply the definition of the function here So our function is going to take the first entry Cx1 and add to it four times the second input Cx2 so for the for the second output we're going to take two times Cx1 plus five C of x2 and then for the third output we get three times Cx1 plus six times Cx2 Now notice that everything in entry one of this vector is divisible C, so let's factor that out So you get C times x1 plus 4x2 Everything in the second entry is divisible by C, so we can factor that out C times 2x1 plus 5x2 and then everything in the third entry is also divisible by C, so we factor that out leaving 3x1 plus 6 6x2 now since everything in all three entries is divisible by C. We can take it out So that's just scalar multiplication right there. So you get x1 plus 4x2 2x1 plus 5x2 And then 3x1 plus 6x2 and you can see that this vector right here is just T of x1 x2 and what do you know that is just C of T of x and so then what we have now done is that this shows That T is a linear transformation. So therefore T is a linear transformation. We just have to show that it preserves addition and scalar multiplication and that's what we've now done transformation