What's up guys, welcome to SPM Buzz. My good name is Ms. Kithingi. We are so excited to have you back here and for always watching our content, for always loving and subscribing and leaving a comment, we definitely do not take you for granted. So asante sana. And in case you haven't joined the family come on, make sure you subscribe. Alright guys, now today I am hanging out with a star.  Thank you for having me. I am very happy to have you here. I am very happy to have you here. I am very happy to have you here. I am very happy to have you here. Guys, in case you have not seen Don before, how come you have not seen him? Please tell us what you do because you do quite a number of things. Once I started working as a technician, I was a donor. I graduated from the English literature school. I was a master's degree in film and theatre. I love to have a great career. I am a character. I am a good actor. I am a good actor and I am very happy to have you here. I am a big fan of Real Housewives of Kang. I have a YouTube channel called Don the Black. on the black. On the black, I'm a hype man, super emcee. He hosts events. By the inspiration here, what you come across is a guy named Mitch Eguang. He's a funny guy. He's an emcee. He's a co-operator. He's a team building. He's a team building.  So any time, any day, any crowd anywhere. Trust Don to be the jack of all trades. He definitely delivers. So try him. Make sure you follow his social media platforms. Don, you have an amazing event and a great event that is coming up. Dance with Don. Dance with Don. Dance with Don. Dance with Don. Dance with Don. He's a big guy. He's a big guy. But at least I'm selling a product. I'm selling a product. So guys, I have a dance event in Itwa Dance with Don. It's an event for physical and mental wellness. In Itwa first of all, May is Mental Health Awareness Month. The theme for this year is anxiety. I'm a creative. To extend your branch to everyone. We all face anxiety at one point in time. Maybe when you're about to go, come away and I'll show you before the interview. For sure. Anxiety. Anxiety, depression, ADHD, OCD and all that. So I'm here to bring to make people start talking about these things. I'm depressed because I'm using it. I'm depressed because there's someone that is actually going through that. First of all, why dance? Dance is number 1. Why dance? Dance is a stress relief. It creates community. It creates end-of-fins. If you come to dance, you will feel good. If you come to dance, you will feel good. It is the Viking House 3rd Floor, 27th at 4 pm. 27th. So, what is the difference between dancing and dancing? Dance is the difference between dancing and dancing.  If you come to dance, you will feel good. I have met psychologists, they talk to us about these things. Because if you have mental illness and you don't know that you have it. I mean, that's a really great initiative because we have seen mental health is quite serious, especially in this country. And you know especially for men, because... Of course, if you have a problem, you have to do something. People say that men should be men and they should not talk about the struggles they are going through. Do you actually think that men are at a point they can speak about their struggles right now? Me, I believe if you can talk about your successes and talk about the good things that go on about your life. Men who are young, you have to buy as soon as it happens.  Talking about this, it solves half the issue.  I am thinking about this, why am I thinking about this? So I encourage talking about it, full time. You see there are also men, when you speak about it, one hand they say, be a man. Because you have seen men who have gone through gender based violence. One hand they say, be a man. What do you do like to tell men who actually discourage others from speaking? I think they do not belong in the 21st century. This is my perception of society, it is a different time. It is a machine as it is. The kind of problems that people are facing nowadays, it is tough out to see what is going on, what is going on, what is going on. That is why it was so hard back then to talk about mental health. Because you are always part of a society. Even when I was depressed, my mother used to say, In the process of intervening, there is a problem that cannot be solved. So this discussion was there in the 21st century. It was a very difficult time. It was a very difficult time, it was very difficult. It was very difficult to see. She said it was difficult, she said it was very difficult. But for the sake of mental health, just talk to someone. It was very difficult. I am interested to know why Dance with Dawn and of course with an agenda of mental health, is it from a personal experience? Is it friends? What is it about? I derive this from the environment. I am the one who is including me. I am the first of all anxiety. Before I was born, I used to go to school. I used to go to school. I tried to reach out to school. Personally, my anxiety has always been a part of me. Before I was born, I used to go to school. I used to say, in case I was born, in case I was born, in case I was born. Now, when I break with the crowd, I am the one who is getting the anxiety. There is a crowd and there is a charge. But I am the one who is getting the anxiety, several, several, several. But now what happens to someone probably who is not a performing artist, you know?  How do you deal with this anxiety? Because it is probably someone who is watching and the anxiety is on another level. Mystery is cut out first of all with a band. There is no miracle that will come from heaven. It will end and it will block out the stage. That is what is cut out. So, for someone who is not a performer, even if I am a psychologist, I am not an aggressor. This advice is what you can do in case you discover that you have anxiety and you don't know what to do about it. There are exercises where you can breathe in, breathe out. There are plenty of exercises you can do to calm down, to understand the story. So, that is what specialists pay on board. They will come and tell us. Some people say that they are self-diagnosed but they don't know what to do. Some people say that they are self-diagnosed but they don't know what to do. Some people say that they are on a platform and they don't know what to do. We have seen someone like Mustafa, the legendary artist. He said, you know, actually, when I went to the Olympics, I saw him taking pictures. And you know, people were like, Mustafa Misha is here. He didn't invest his money. He said there was something else going on. We actually have to tell people who jump into conclusions to judge, just judge a situation. Number one, they are not doing anything better for mental health. I am inspired by Mustafa Misha. Because he just said, I am the one who invests the money. He said, I am the one who invests the money. He is a businessman. He doesn't know what to do. He doesn't mean that the artist invests the money. He doesn't know what to do. Number two, you have no high moral authority. Okay, Number three, you have no high moral authority.  Number three, you have no high moral authority. Number three, you have no high moral authority. Number three, when the artist uses money, it is not different. Number three, when your life is alive, then it can work. Remember your right tracks are all linked up to the first cellphone. But if people work more hard to save money than the artist does. That's why it is a different concept and it will turn out to be a blessing. Mustafa is getting shows. He's back in the limelight. He's back like he's never left. Big up Mustafa. Hey, man. Hey. You do what you have to do. If it does not make money, it does not make sense. So make it make sense. So also now, we sometimes, just the other week ago, you know, like several weeks ago, we saw the Shakahola conversation here.  Religion, number one, for religion, you have to agree. Because no matter your opinion towards religion, either come on the extreme, normal, either on the extreme or on the normal, once you say it's religion, it's logic. It's one plus one is equal to two. It's purely belief. It's not just me. It's not just me. It's not just me. When I say it's me, it's not just me. It's not just me. It's not just me. Either you say it's me or you just say it's me. It's not just me. If you say it's you, it's may be it's me. It's not just me. It's not just me. It's a bit more, a bit more. It's a bit more. It has to be you. It's got a bit of a bit of a mental health issue. I am even more selfish if I say it's not just me. No, it's not just me. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that. You cannot do that.


We're going to get started in a couple of minutes. Thank you for coming. Okay, great. Let's get started. Hello, and welcome to my talk on multi-tenancy for Argo workflows and Argo CD at Adobe. I'm Srinivas Maladi. I'm a software engineer at Adobe who works on the internal developer platform at Adobe. So before I get started, I do want to give credit by emphasizing that the work I'll be preparing today is not just a product of my own efforts but a combined product of efforts of many of my colleagues at Adobe. So with that, let's get started. So this is our agenda for today. We will be starting with some background information about what my team does, some terminology referenced in this presentation, and an architecture overview before we dive into how we address multi-tenancy for Argo workflows and Argo CD. Let's get started with the internal developer platform. So Adobe has three main categories of product offerings. There's document cloud, creative cloud, experience cloud, which are themselves comprised of different services and products that are developed by various internal teams at Adobe. These products and services may be deployed and run using different internal platforms that serve their specific needs. And in order to build and deploy and run successfully, these different platforms leverage certain core mechanisms, infrastructure tooling, and services that are provided by the internal developer platform. As part of its offering, IDP, internal developer platform, uses and provides access to resources on various cloud providers like AWS, Azure, and Adobe data centers. The internal developer platform standardizes best practices and consolidates engineering efforts across the various internal developer teams at Adobe while providing a CI CD experience that remains flexible for the different use cases you see on the screen here. Next, let's go over some terminology that we will be referencing for the rest of the presentation. GitOps. So GitOps is an architectural paradigm where desired system state is first defined and tracked and get somewhere by some tooling. That tooling then deploys that defined state to a live state on a running system. The GitOps tooling regularly synchronizes the two states so that the live state can get automatically pulled in with any updates, and it pulls in any changes made inside Git. Argo CD is an example of one such GitOps tooling. It supports tracking of Kubernetes, rockets, manifests in Git, and supports their deployment and synchronization to, for example, a namespace on a cluster. Argo CD is implemented as a Kubernetes controller. It uses various CRDs. One of them is called an Argo CD application. And that stores information about what defined state to track and where to track it, as well as where to deploy it. So here's an example of what that looks like, the YAML manifest for an Argo CD application. As you can see, there are sections in the manifest defining the source and intended deployment destination of the defined state. And Argo CD itself supports automated deployment, self-healing, complex rollout strategies, monitoring, and much more. Through the Argo CD UI, which we will discuss more later, the Argo CD also visualizes the health and deployment status of the developer's deployed resources wherever it's deployed on the cluster. Argo workflows. Argo workflows is a workflow engine that can run CI CD pipelines, among other things, on a case cluster. There are many different kinds of workflows, but generally a workflow can be thought of as a set of tasks. And each task runs in its own pod and can also be modeled as a dependency on other tasks. The resulting workflow can actually be templated and defined using YAML files like this on the screen. And this demonstrates a directed acyclic graph type template. It's an example I pulled online. The use of templating opens the door to sharing a template to be used as a sub template in other workflows. Since a workflow is implemented in a Kubernetes CRD and each task runs in a pod, for example, this means that workflows natively integrate with existing Kubernetes objects, such as volumes, config maps, secrets, and much more. Now let's get a better understanding of the CI CD offering that IDP at Adobe provides and what it looks like with Argo workflows and Argo CD working together. So we start with the hub cluster, where we have Argo workflows and Argo CD installed. We also have client namespaces where client workflows are run. Additionally, there are remote clusters with remote client namespaces that are running client applications alongside any case resources that they need to support it. These remote clusters are usually pulling an image from some registry to run as part of the deployment. Now the defined state for all of this is actually stored in Git and using GitOps, this includes, of course, client repositories, which includes the application code as well as a deploy repository. The deploy repository contains CI CD resources as well as resources associated with any of the case resources deployed on the remote namespace. So you have your service case manifests and, of course, you have your workflow manifests, which are the CI CD resources themselves. So these repositories may also refer to other IDP-owned resources that they can refer to and we do this through Helm and we'll talk more about that later as well. So how do these pieces connect? So to start with, Argo CD is always tracking the client deploy repository manifests and it's doing this because it's constantly deploying them to the workflows on the hub cluster as well as the actual deployed resources on the remote cluster itself. Following this setup, any change made to the application code will trigger a workflow inside the client hub namespace and specifically what actually happens is this requests a workflow from the Argo workflows controller that's running and the Argo workflows controller will execute a workflow in the namespace that requested it. So one of the steps in the workflow is a build step and that actually builds and uploads an image to the image registry that we mentioned earlier. The other step after that goes ahead and takes that tag or whatever information about that build and enters it into the service case manifest so that it's recorded somewhere in Git. Now as mentioned earlier, that state is constantly being tracked by Argo CD and so that information is then deployed to the actual running state on the remote cluster. So now the running state knows where to pull it's the new image tag and it pulls it from the image registry that it was pushed to in the first place by the workflow. So in this way, Argo workflows and Argo CD are used to allow our clients to run CIC pipelines on the cluster, with a focus on GitOps. So now that we've gone over the general walkthrough, the general architecture, let's take a look at how we make this multi-tenant. So multi-tenancy, it's an architecture where multiple clients are sharing a single resource. In this case, it's a running instance of Argo workflows and Argo CD. It can reduce waste by reducing idle time as well as a number of running instances depending on how you manage load and how scaling is handled. It can save developer time, mainly because it allows them to spin up faster. They don't need to sort of spin up their own instance because they're using something that exists already. It provides an opportunity to consolidate best practices, fixes, maintenance and other updates and that reduces the need for domain expertise from developers or IDP clients. It's efficient because every improvement or feature that's made to the shared resource is a force multiplier that can be used across every single team that's leveraging it and because of all of this, it scales really well because by reusing previous work relating to setup and maintenance, we're reducing the amount of friction it takes for a client to get up and get started. Some of the questions we had to answer when addressing multi-tenancy with Argo workflows and Argo CD were how do we make sure that clients can only access what's theirs? How do we isolate client workflows and processes and deployments? And what's the best way to get updates and fixes out to clients for resources that we don't control? And of course, how do we design for reliability as load increases? So let's start with access for Argo workflows. This page shows the Argo workflows UI. It's exposed by the Argo server that's part of the workflows installation on the hub cluster and it visualizes client workflows that run inside client namespaces based on our architecture. We can see here that workflows are being returned for a specific namespace that we entered and also that a workflow is currently in progress. After clicking into a workflow, a client can view workflow tasks and logs as well as execute actions like suspending, retrying or even deleting workflows. And while this is great functionality to have, how do we make sure that clients are only doing this for workflows that belong in their own namespace on the cluster? So when a client wants to access a workflow, resource to the UI, whether or not they can access that resource depends on two things. The client's group membership as well as the presence of a service account in the namespace of the requested resource. So to achieve this, we use single sign-on and the workflows namespace RBAC feature. Let's go through this step-by-step. So after receiving a client's request to access the workflow resource, the Argo server will first get the group member information for that client through the configured identity provider. In our case, that's Azure Active Directory. The Argo server then goes ahead and checks whether a service account is associated with any of those groups inside the namespace of the requested resource. If it does, and the service account has necessary permissions to access that resource, then the access is granted. And if it does not, the access is blocked. So this access flow allows us to scope client access down to workflow resources at the namespace level. And it's using service accounts, so it's highly scalable since it leverages case-related RBAC through service accounts. This allows the many-to-many relationship between our clients and namespaces on the hub cluster. The namespace RBAC feature that comes with Argo workflows requires that the service account is annotated with the authorized groups that are allowed to leverage it for access to the resources in the hub cluster. You can see this in the example manifest here, which specifies that only clients in group A can use a service account for access. It's in a list by default, so you can add to those groups. There's also a precedence field, and that's mainly used to break ties for service accounts that might match, multiple service accounts that might match to a client. The Argo workflows command line interface, or CLI, is also exposed by the Argo server, but the mechanism for scoping access here is slightly different. This time, instead of an identity provider, the Argo server actually delegates authentication in our architecture and in the general Argo workflows architecture for SSO to the, sorry, for the CLI to the Kube API server. The client includes a bearer token in their request, which may be automatically included as part of an exported Kube config file locally, or as part of a token in the actual CLI request that's going out. The token is validated by the Kates API server, and if the token is valid and contains necessary permissions for the requested resource, then the request is granted. So here's an example of what a successful CLI looks like when I'm querying for my own workflows from the CLI. As you can see, this is my own namespace, and I'm getting back my workflows. And here's an example of what happens when I try to query for workflows in the CLI that I do not have access to using that flow. Now that we've covered how access is scoped or how we scope access in the workflows UI, let's take a look at the Argo CD UI and how it does it. The Argo CD UI visualizes applications that track and deploy state defined in Git. The goal here is that clients should only be able to view their own Argo CD applications through the UI. This is because access to Argo CD applications also provides access to the visualizations of the associated deployed resources, and it may also include access to syncing that resource or even deleting the underlying deployed state. So in the Argo CD, the UI and CLI use the same mechanism this time to scope client access for resources like Argo CD applications. Similar to workflows, these are exposed by the Argo CD server in this case. Argo CD also has other resources in addition to Argo CD applications, such as logs, connected repositories, connected clusters, certificates, as well as app projects. The app project resource is important, as we'll see later, because it can be used to scope or group Argo CD resources, which can then be used to scope access. Some, but not all Argo CD resources can be grouped together into an app project. Access to those resources can then be scoped to clients who have access to the parent app project itself. For resources that cannot be scoped to app projects, access is usually defined using a global config map. We generally don't use global RBAC in our architecture for client access because it can act as a single point of failure. It's prone to typos. On the other hand, the app projects are really great because they cannot by design be used to provide access for resources that don't fall under those app projects, making it ideal from an access scoping perspective. After setting up project scoping, a client can request access to a resource, and the Argo CD server gets the client's group information from, again, the configured identity provider, which again here for Argo CD is Azure Active Directory. The Argo CD server then checks if any of the client's groups are authorized by the parent project of the requested resource. If so, then the request is granted. If not, the request is blocked. This is an example of what the Argo CD app project that we use looks like. You can see that the project definition includes specified, authorized groups, and their associated role permissions over the resources in this project. So when setting up scoped client app projects, we usually set up three roles by default out of the box, an admin role, a read-only role, and an automation role for external Argo CD application access, for example, from the Argo workflows, right? So now that we've finished covering how access is configured for Argo workflows and Argo CD, let's go over how isolation is achieved for the respective tools, beginning with Argo workflows. So as mentioned, each Argo workflow runs in a pod, and they run in client hub namespaces. So far, we've ensured that clients can only access their own resources, but how do we make sure that they can't access other resources through their workflows, and how do we protect their workflows from other clients? First, we require that all clients bring their own secrets to the namespace. So this helps with workflow access, automatically scoping it through automatically scoped Git tokens, artifact credentials, Argo CD automation tokens, and any other secrets that we pull from Vault. Next, we leverage the lack of shared secrets to configure individual client artifact repositories, and this adds another layer of isolation for the artifacts that are passed between steps in a workflow. With regards to the executor that's actually running the containers inside the workflow pods, we allow the recommended emissary executor by default, as well as the Kate's API executor. The latter requires some network policies, but we lock that down by using default network policies to control for ingress and egress by default. When triggering workflows, we specify a service account that is not the default service account to run under. This defines the permissions the entire workflow runs under, or rather each task, and also helps us be intentional about what permissions we're giving to the workflow itself. We also have cluster level pod security policies that limit pod capabilities, and force default second profiles, and prevent privileged pods from even being spun up on the cluster itself in client namespaces. To accommodate this, we use Canico to build, and that's because Canico does not require any special privileges in order to build our client images, and we also use multi-stage bills in addition to that. So for clients that do require privileges, for example, Docker and Docker, we offer off cluster builds through code build, and this again helps to give them these privileges while maintaining the kind of isolation we want in our workflows. And this is also directly integrated with the Argo workflow step that we use. So the result of these layers is that client workflows and the underlying pods are isolated to the client hub namespace in which they're running, and this prevents their resource from being accessed by other clients, and also prevents them from accessing other clients' resources. With Argo CD isolation has to happen at the Argo CD application level first. So as mentioned previously, an Argo CD application is a CID that can be deployed to specific, that can be used to deploy to specific namespaces on specific remote clusters. The Argo CD application is also configured to track CATE's manifest from a Git repository, in this case, the Deploy repository that's owned by the client. In this situation, the Deploy repository might be referring to other repositories, for example, in Helm charts or Helm dependencies in Artifactory. So how do we restrict what client through their Argo CD application can deploy to and track? The answer again here is actually Argo CD app projects. In fact, the Argo CD app project has to specifically allow tracking of Git locations of the child Argo CD applications in order for those Argo CD applications to be allowed to track those. Similarly, it has to specifically allow deployment to namespaces on clusters in order for the child Argo CD applications to be able to deploy to them. If the Argo CD application does not follow these rules, then an error will actually be thrown when attempting to create the application or put it inside the project.  Finally, we also restrict Argo CD applications from creating any cluster level resources like creating or deleting namespaces because that's not good when you're sharing a cluster with other clients. The other Argo CD component that we look at with regards to isolation is the Argo CD operator itself that's running on the hub cluster. So the operators connect to multiple remote clusters that it deploys through a hub and spoke model. And this means that the operator running on the hub cluster is connected and deploys to all of these possible remote namespaces on different remote clusters. So how do we ensure that we're appropriately limiting the permissions that the Argo CD operator is running with? Well, we need to secure it so that in case of a vulnerability, and in general, we limit the extent to which a client or a malicious actor can actually take advantage of Argo CD to interfere with other client resources deployed on the remote cluster itself. So Argo CD when registering a cluster actually first needs to have a service account that's on the cluster. And by default, this service account is created in the registration process in the cube system namespace with admin cluster level permissions. Argo CD is then given the token associated with that service account and then uses that token to manage and deploy to that cluster. Now there are two things that could be wrong with this picture. The first is that by default, the registration process out of the box puts the service account in the cube system namespace. Now as we mentioned earlier, the cube system namespace, or rather as we mentioned earlier, you may be using pod security policies to be preventing privileged pods from being spun up on your cluster. A lot of the time, best practice is actually to allow this service accounts inside the cube system namespace so that they actually aren't restricted by those pod security policies. If that's true for you, then this default setup will actually allow the Argo CD service account to also be allowed to spin up privileged pods on your cluster, which means that clients through Argo CD might actually be able to spin up privileged pods through the default setup. The second thing that could be wrong with this picture is that Argo CD by default has admin cluster level access, which isn't necessarily good because there may be namespaces on the cluster that have nothing to do with Argo CD deployments, and so Argo CD shouldn't necessarily be able to deploy to, edit, or even view those namespaces and the resources inside them. So in order to begin fixing these issues, let's start with the cube system namespace issue. So the cube system namespace issue, if you just change the namespace, then if you have pod security policies that restrict privileged pods being spun up on your cluster, then it will work as expected, and we've restricted Argo CD's ability to deploy privileged resources and by extension clients that use Argo CD. Now let's go ahead and change that admin cluster rule to a read-only cluster rule. So cluster-wide read access is still required by Argo CD, and this is because Argo CD does a bulk read of the API whenever deploying to or syncing to a remote cluster itself. It's, this check will actually error out if it does not have a cluster-wide read access, and now Argo CD will generally be fine with cluster read access, but it still needs admin access to the individual namespaces when deploying resources to. And we achieve this through individual rule bindings in each of the namespaces, give the service account right access to the namespaces themselves, and this limits the blast radius to just the namespaces that are supposed to be managed by Argo CD themselves. So we can do better actually. We can remove secrets, especially if you use some sort of vault integration to provision Kate's secrets separate from Argo CD on namespaces, so Argo CD does not need to read secrets at a cluster level. We can also remove secrets from the right access that it needs for namespaces itself, because again, Argo CD is not deploying secrets. As mentioned earlier though, how do we prevent the cluster bulk read from erroring out when it's trying to sync? And we do that through exclusions. We tell Argo CD, please ignore secrets on this cluster or on all remote clusters, and it will do that, and so you won't have any issues when trying to sync or any errors when trying to deploy to a remote cluster. So the result of this isolation of the Argo CD operator is that we end up with a more secure hub and spoke model, and by extension, clients, they can't do as much on the connected remote clusters that are shared between them. So we've explored access and isolation for Argo workflows and Argo CD, and I've seen that clients have various repositories that are involved, in line with the GitOps model that we follow. Clients have an application repository that's running application code. They also have a deploy repository that contains service manifests for the actual running service and case resources on the remote cluster, as well as workflow manifest for the CI CD pipelines that they're running on the hub cluster. So unlike the shared instances of Argo workflows and Argo CD, these resources are actually client-owned. So how does IDP help distribute fixes, updates, maintenance to these client-owned repositories that can drift over time? Well, the answer here is that we try to centralize what we can. And so for the application code, for example, we provide generation templates that can be used to help clients spin up more quickly in terms of their application code. For the CI CD resources and the service case manifests, we use our own repositories and we connect them because everything's a Helm chart. So we can use Helm dependencies to automatically pull in any changes that we deploy to our centrally managed repositories themselves. And we actually use semantic versioning in order to achieve this in a more seamless way. So Helm supports semantic versioning, which is a system where version numbers actually convey meaning about things like backward compatibility and breaking changes. Through semantic versioning, client-helmed charts actually configure to automatically pull in minor or patch releases made on the IDP-owned repositories. Of course, minor or patch releases, they have to be backward compatible, which is what makes them minor or patch releases. And here are some examples of what such a release would look like. The first is a patch that moves from 1.2.3 to 1.2.4. And the second is one that moves from 1.2.3 to 1.3.0. Both such releases made for IDP, Helm charts on the right, would automatically be pulled into the Helm charts on the left due to semantic version integration. This is powerful because it allows us to get critical security updates, fixes, best practices out to clients immediately, provided that these are all backward compatible. Now, what do we do if a change that we would like to push out to clients is not backward compatible and contains breaking changes? Well, semantic versioning allows us to do this through the major version releases. This requires clients to manually bump the version of the IDP Helm dependency that they're using inside their own repositories. We usually try to include release notes on the breaking change, and once clients make the necessary adjustments, if any, to accommodate these changes, they can immediately pull in all the work and fixes that we've made inside our dependency that we control. Here's an example of a major release change and what it looks like, moving from 1.1.0 to 2.0.0. And here's an example of what a client's chart.yaml looks like that shows how we use Helm dependencies with semantic versioning. We can see the name, version, and repository that the IDP-owned Helm dependency here corresponds to. We can also see a carrot to the left of the version number. In semantic versioning, this means please keep me updated to the latest compatible version, which roughly translates to what we're trying to do is please pull in all minor or patch versions, but don't automatically pull in major versions. Let me do that. This is what it looks like in a client's repository. So as we build out multi-train architecture, how do we prepare for heavy load that may appear as we scale up? Well, for high availability, the Argo workflow documentation is actually quite great, and it details improvements such as pod disruption budgets and minimum replica counts for the Argo server, and this helps generally improve the components inside the Argo workflows installation. As we onboarded more clients running workflows in different namespaces on the hub cluster, we noticed that it helped to mirror common images that were being used across the workflows in the hub cluster, and this was because if all the workflows are pulling in images at the same time, we quickly hit Docker hub or rate limits in general. So mirroring them to our own internal repositories was quite useful for us. We also increased the workflow executor's default resource allocations for CPU and memory since this was limiting the size of the artifacts that clients were passing through or between workflow tasks themselves, and we also increased the default main container resource allocation to accommodate more resource-intensive workflows. The main container is the containers that's actually running the task. On the other hand, since we've removed or increased these limits, we also needed to be able to limit what a client workflow itself could consume on the cluster, and we did this through namespace level quotas that limit the impact a single client can have on the resources being shared by all the clients on the hub cluster running workflows. Argo workflows provides Prometheus metrics relating to workflow controller state and also provides support for custom metrics at the workflow or template level. Documentation on the default metrics expose and how to create custom ones, as well as example, Grafana dashboards are available online as well, and they're great. We encountered similar challenges when scaling Argo CD, but for different reasons. The Argo CD documentation has well documented, high availability configurations settings for each of the Argo CD components, and also conveniently packaged HA release itself that we actually use. As we onboarded more clients, we switched to using GitHub apps to register repositories. This was both due to the fact that GitHub apps have higher rate limits than generic users or personal access tokens that you may be trying to clone from, for example, when Argo CD is tracking state and Git, and also because it actually reduces client friction. Instead of having to generate a deploy repo, clients don't need to do anything except install the GitHub app on their repository, and we already have the GitHub app credentials, so we can go ahead and use that to clone their repository. So that's one less step for clients as well, as we scale up. We also looked into issues like limiting the ability for an individual client to be able to overwhelm Argo CD resources, for example, with a very large repository for the Argo CD repo server. And we also looked into sharding Argo CD operators across different hub clusters in order to help distribute the load as we start scaling with more clients. Argo CD as well exposes various Prometheus metrics that are well documented, as well as for the different components, such as the API server, as well as the application controller operations. And these are great for monitoring and they also include examples of Grafana dashboards online. So here you can see our own Grafana dashboards that draw on the metrics available for Argo workflows in Argo CD, and these really do help as you scale up because you can catch problems early as you debug them. So to recap, multi-tenancy in Argo workflows and Argo CD have similar challenges, but are addressed differently because they're different tools with different features. We saw how these existing features are leveraged today at Adobe with GitOps for a multi-tenant architecture that scales well with more clients, and keeps clients isolated from each other, provides them with scoped access to their own resources and helps us maintain a distributed support framework that helps clients continue to accelerate. And this generally supports the ability of the internal developer platform at Adobe to create a CI CD experience that remains flexible for different use cases and represents a force multiplier for all of our developer teams. Thank you for your time. Please provide feedback, thank you. Any questions? Feel free, yeah. Hi, thanks for the great talk. Just a quick question about the authentication and Azure AD. Yep. I think in your diagram, I caught that you just showed the user presenting to workflows a bearer token when they log in. I was curious, how does that process, the authentication process work? Where did they get that token? How is the single sign undone? Yeah, let me just actually go ahead and just find that real quick. Yeah, so the token itself, so that's for the CLI acts. Oh, the question was, where do users get the token in order to access the Argo workflows CLI? Because we showed the, this is, yeah, because this is the UI, but we showed the perspective from a client's perspective. So where did they get that token? So two ways. They get it from their Kube config, which the internal developer platform provides, and that's one way. So they have this Kube config that's exported locally and that's present. So the client is even aware of this. They just go ahead and run Argo commands. It automatically uses the exported Kube config and it just works for them. That's one. The other way is if they actually have any bearer token, if they have a service account token, if they have any Kate's token, they can actually include it in the CLI request as an additional flag, right? But usually the first use case is the more common one because most of our clients have Kube config, YAML files available that are exported locally that allow them to access the cluster itself. They're a scope to their namespaces. Yeah, great question. Thank you. You mentioned that Argo CD doesn't create the namespaces. What creates the namespaces and how do you manage those? That is a great question. It does not. So we have a tool as well as another team itself that sort of helps us standardize namespace creations. It's a provisioner tool and that sort of integrates with Argo CD as well and the GitHub GitOps model itself to create and manage those namespaces so that they're isolated from clients. That's a great question, thank you. So I've seen people use the Argo CD operator as well as like a management cluster pattern where you're using one Argo CD instance to deploy to multiple clusters. I'm kind of wondering what your options analysis was and maybe why you switched or why you use the Argo CD operator because I think it's a little bit less common. And if it's a performance like related issue if you can maybe say roughly like where you hit that performance sort of bottleneck and made the switch. Yeah, so just to clarify, you're referring to having an Argo CD operator in the hub cluster in its own namespace versus a separate one for each of the clients, right? That's what you're talking about. Yeah, exactly like a single Argo CD instance doing the deployments versus multiple. Yeah, so I guess the thinking here was we would evaluate that as we scale and see if it's necessary. So first we wanted to start out with seeing if we could shard across multiple hub clusters which is slightly one step lower than what you were mentioning and see if that could actually scale well. But I think probably you were also bringing up because of the security concerns that they have on the security blog about RBAC not being a silver bullet and stuff like that. But generally I think when we were building this out the main consideration we have was how do we help consolidate maintenance and best practices and simplify things for clients as while providing a secure experience. So because we had the RBAC and because it provides secure experience and it simplified things. So that's a great question. If I can extend just a little bit. I think one thing I've seen is people looking at trying to get a single pane of glass for sort of deployments across like an SDLC. If your setup's not like that, I've seen setups where you have a dev cluster and a product cluster and the same versions of applications deployed both. So if you have one instance, then you have that but I'm wondering are you tracking applications through a set of clusters or are they kind of orthogonal? Sorry, what was that last part or are they kind of what? Orthogonal, completely separate. Yeah, so I'm not sure if I caught the entirety of the first part, sorry about that but all the ArgoCD applications are technically living inside the installation names based on the hub cluster. So it's being tracked by the ArgoCD operator on the hub cluster but those applications of course are deploying out to remote clusters themselves. We can also talk offline because I feel like I might have missed something important in the first like sentence or so and I'm sorry about that, yeah. Yeah. A great presentation, Srinivas. So I have a question like you create the namespace by different tools. I think you can use a crossplane but right now we use this terraform and we have a resource quotas on the top of namespace but when you deploy using the Argo workflows or the task basically, I mean have you ever seen like any problems you hit the quotas and all? Yeah, that's true, we have. So that was part of why we had to modify the resource allocations out of the box for the workflow executor. So both for the main containers as well as the executor itself. When you modify the resource allocation for the executor that automatically trickles down to the init container as well as the wait containers that are sometimes affecting whether or not the workflow task will run properly. And then the main container itself is just where the task runs. But that's part of why we did that. The namespace level quotas that we enforce are configurable so teams if they think that they're running very heavy workloads they can actually come to us through a process, a process in place for them to increase that. But we have seen issues and that's why with the workflow level resources themselves and that's why we had to change those limits to just accommodate. Okay, I'm yet not an Argo customer but are there any specific reason that you are installing not utilizing the SES platform for Argo? Sorry, again the last three words I think I just.  Yep. I mean, are there any limitations in the SES environment for the Argo which? Well, I think I touched on the provision earlier. We just, it just gives us more granular control over how we manage certain parts of our architecture I think. Again, as we scale up I think we're still looking to grow. We have a good multi-terrain architecture right now we're satisfied with but depending on the needs of our clients and as you can see we have a lot of different use cases we might very well change based on where that direction goes. But for now, I think really we wanted flexibility. That was the main point, yeah. Okay, yeah, okay. I'm getting the, feel free to come up. I'll be in the hallway. It's lunchtime. Thanks for attending.


The women's team back in action tonight taking on Union in the first Emily Berglin with a great opportunity. She goes off the post and we head to the second period scoreless. Jacqueline Casa on the odd man rush. She fires at herself and beats the goalie for the one to nothing lead. And that was all that BSU would need because Lauren Bench was phenomenal all game 22 saves in the shutout. Berglin will get a short hand and goal Kiki Radke with an empty netter as the Beavers win three to nothing. If you enjoyed this segment of Lakeland news, please consider making a tax deductible contribution to Lakeland PBS.


Welcome to Sheboygan County Government, working for you. My name is Adam Payne. I'm the Sheboygan County Administrative Coordinator, and today we're gonna be focusing on the county operations, finance, finance department, and budget process. My co-host and fearless leader, county board chairman, Dan Lemming, you cannot be with us today, but I do have two very important guests. The first being County Board Supervisor County Zigglebauer. Connie is the finance chair, finance committee chair, as well as Tim Finch, the finance director. Today, we're gonna, as I said, be talking a little bit about the budget process, the overall county operations, expenditures, revenue. And to begin, if Connie, you could please tell us a little bit about yourself and your background and role on the county finance committee. Okay. My name is Connie Zigglebauer. As you said, I've been a resident of Sheboygan all my life, quite a few years. I've been active in the community. I'm a chamber of commerce, various other organizations. My church. I have four children, two great grandchildren, not great grandchildren, two wonderful grandchildren. About 10 years ago, well, prior to 10 years ago, we had a very wonderful lady in my district, Charlotte Williams. She was a county board supervisor for 10 years, and she decided not to run again. And she encouraged me to run. And as most people in the county, I had complained about government, so finally decided that it was time to do something about it. And so I ran for county board. I've been on the personnel committee, resources, property, and for the last five years, I've been on the finance committee and now finance committee chair. The role of chair really is just organizational. I only have one vote just like everybody else, but just kind of getting things organized for the finance committee meetings, deciding the agendas, and things like that. Now one of the things we're going to be discussing today is the budget development. Why is it that the annual budget is probably one of the most important public policy documents developed? Well, the budget controls everything. If there's no financing, nothing happens. All the departments or all the citizens in our county need services and ask for services and expect services and deserve those services. But unless those services are funded, there's nothing that can be provided. So the budget document determines what will be funded, what services the county will provide for its citizens. So what do you see? Some of the major initiatives are challenges as we go forward with the development of the 2001 budget this year. I believe that we're able to maintain and control our costs at the administrative area. Salaries are a problem because we're bound by union contracts for certain benefits and salary increases, so that's not always real controllable. But the big things that we have going for us now are our institutions, the three health care centers, and health and human services where we're providing for the developmentally disabled, the people less fortunate than us that need special services. Those two areas account for over 50% of our budget. With the consolidation of the health care centers right now, it's very difficult for them to keep a handle on their costs. And until that consolidation is completed, I don't think we have a big hope of being able to at least hope to run in the black for a few years. What happened last year with the budget for health care centers? You were mentioning the hole there. Last year, I don't know how deep to get into this, if our health care centers had been run as a for-profit company, they would have lost four and a half million dollars. However, due to the fact that we receive funding from the federal and state government, well, it's actually from the federal through the state, we did get, I believe, Tim, 2.9 million dollars in assistance to take care of the underfunding that we had. So our health care centers lost 1.8 million dollars last year. A few years ago, we were able to maintain the costs a little bit better. But what we're seeing right now is the county is left, and I shouldn't say is left with, it's the county's obligation to provide for those people who cannot provide for themselves. We get the patients in the worst situations requiring most care, and Medicaid only pays a certain amount of money. So it's very difficult for our health care centers to operate in the black. And we're going to talk about that a little bit more often, not only the challenges with the health care centers, health and human services, some of our largest departments. My next guest, as I mentioned, is Tim Finch, our new finance director. Tim started in May, and it's certainly a pleasure to have Tim with us today. He is the second department head that I've had an opportunity to be involved with the hire process, and he hit the ground running. He's doing some real very valuable things for us.  Sure, great. I appreciate it. Thank you. My background is, I'm not a Sheboygan, I'm a native Sheboygan person. I guess it's Sheboyganite. I am from Kenosha. I've been here for about a year. As I've said, since May 1st, I've been the finance director, and it's a pleasure to serve in that position. Prior to that, I was the accounting manager of the health care centers. So I have a little bit of background with Sheboygan County. Before that, going back, I had 13 years as chief financial officer for a group of related health care non-profit corporations in Kenosha. So I have a good background in finance. It goes back quite a ways, and I do enjoy doing it. I live here in Sheboygan. It's a great community. I love it here. I've only been here a year, but it feels like home already. My wife, Linda, and my daughter, Ashley, love it here and have adapted much better than I ever would have hoped for. So I'm very happy to be here. You just mentioned you've been here a year, yet you started as the finance director in May. What were you doing prior to that? I was the accounting manager for the health care centers. I actually was working out of Sunny Ridge. And that was an interesting position. When you work right in one of the facilities, you see the tremendous opportunity and also the responsibility that we have to take care of the residents of our county. So I got a real close look at what we do for our residents, and it's amazing what we do for them. Connie and I obviously know why you're here, but why don't you tell our viewers why you wanted to be the director of the finance department? Well, actually, that's an interesting story. When I was interviewed for the accounting manager position at the health care centers, I was interviewed by Linda Martin, who was the past director. During the process, she asked me a lot of questions, and one she asked me was, do you see yourself at some point taking my job, or are you interested in my position? And I think what she was driving at was, do you have ambition? Do you have goals? Do you see yourself in the position I'm in? And I took an instant liking to Linda, so I felt very comfortable and saying, no, I really don't want your job. I want the finance director's job. And she informed me he had not been here that long and that probably I would have to wait quite a while. So I said that's fine, and I still was interested in Sheboygan County and took the position anyway. But that truly is, I guess, what I love to do. I love to be in a position where I can be working with many concepts, many ideas. I love learning. It's something I don't think I'll ever want to stop doing. And in a position as finance director, you learn something every single day. We have some real challenges this year with the budget process and balancing the demands for services with the finance department. What do you see as some of your primary goals or responsibilities for the year ahead? Sure. Right now, we're in the midst of our budget process. So that's one of our number one goals now is to keep the budget process going smoothly, make sure all the steps are followed, all the committees are informed of what's going on, that the administrative coordinator is aware of what's going on, keep that process going. That's a major effort really from June through November. The other big ticket item on our plate is our new computer system, the JD-Edward system, which we implemented December 1st of last year. It has been, is a tremendously powerful system with potential for providing information that we have never had before. But along with that comes some frustrations in getting it set up the way we want it and working with issues. It is a fairly new program. So there's some things that are being adjusted in the programming. So that's not where we want it yet. So that's a challenge for us to get that where it is so it can provide us the information on a very timely basis and pay off for the effort and the money that we've put into it. Well, let's start to get into the meat of our discussion today. Tim, why don't you start by touching on some of the county expenditures and revenues so our viewers begin to get a feel for the operations here and the amount of tax levy as well as our overall dollar budget in terms of running our programs. Sure. Thanks for the graphs. I'll go to those. Don't lose your mic on the way. Thanks. I think I've got it. Okay, for expenditures for Sheboygan County, this is from the 2000 budget. We've got a pie chart that shows how we break up our expenditures. We had a total for 2000 of about $122 million. In the pie chart down below here, the largest category is the human service department. That takes about 27% of our expenditures. Another large one would be the healthcare centers. That's the one in pink right here at almost 26% of our expenditures.  They split fairly evenly between general administrative costs in blue here. Justice and law enforcement in pink also over here about 12%. And then you have smaller capital projects. These would be some of our bigger ticket items. That's the white pie, part of the pie there. Bigger items, healthcare center consolidation, things like that. And public works, the highway department, et cetera, would be about 9%. So it gives you kind of an idea of that. 121 million, how it's split up. And the two biggest departments as far as expenditures would be is the health services and the healthcare centers. Then for revenues, same thing for 2,000 budgeted revenues. Again, we have a budget of about $122 million. The breakout here is by type of revenue. Where the money is actually coming from. Property taxes is a very large portion of it at 26%. It's the blue portion. Service charges in red is actually just a little bit more at 27%. This would represent charges from the healthcare centers for the three facilities that we have, comprehensive Rockingville and Sunny Ridge. Also health and human services charge for their programs. And as well, we have charges, some charges from law enforcement and register of deeds, some miscellaneous areas that we do charge for. So it's split up into quite a few areas.  And use a small amount of retained earnings, which would be prior profits in departments that we've held over to try to cover operations in a future year. So that's the breakout on our revenue. So $120 million budget. We have 23 departments. You've touched on some of the key areas and lumping together some of those departments. Expenditures, revenue. When you have a $120 million budget, what does that mean in terms of our tax rate and how has it changed over the years? Sure. Our tax rate over the last 10 years is this is a bar graph that shows our tax rate over the last 10 years. And this is showing from 1990 through the year 2000. And it's the rate per thousand of equalized value on a home. And we have some highs and lows here. But for instance, if we go back to 1991-1992, we were really at a high point there of $6.87 per thousand. And what that would represent would be for a homeowner with a $100,000 home with that valuation be looking at $687 for their annual tax bill for the county. Now you can see it does kind of drop off. A high point here in 1991-1992. Now it starts to drop down during the period of 94, 95, 96, 97 was a low point at $5.37 per thousand. Now it's started to come back up. And one interesting thing to note here, this is not by accident. As it comes back up in 1999-2000, you can see that it's leveled off at $6.30 per thousand. For instance, this is a conscious effort on part of the administration and the county board to prevent these peaks and valleys and the pendulum effect of where it goes up and then it comes down. And then maybe expenditures aren't used or projects aren't done, they're kind of held off on. And you get back where you have to put money back into them. To try to level things out is much better scenario. And here you can see in these last two years, 1999-2000, you have a much better handle in this situation, much smoother curve. This is better for the taxpayer and it's better for the county too to manage the resources. Thank you, Tim. It's also a credit to the county board that in the last 10 years our tax rate today is less than it was 10 years ago. Not only do we have a great deal of operational expenses when you have 23 different departments, 1350 staff where the second largest employer in the county we also have a fair amount of capital projects or infrastructure improvements. Connie, could you touch on a little bit the five-year capital plan and what you see as some of the major projects that are coming up? Well, our five-year capital plan is attempting to be a systematic way that we do capital projects for the county. Trying to keep a level amount of spending so that we don't affect the tax rate as Tim just expected. It just explained by having a very high tax rate one year, a low tax rate the next year. In addition, when we look at the five-year capital plan, when we, generally speaking, we borrow for these capital projects and it's advantageous to the county to borrow more than ten million dollars. So we try to hold our number below ten million each year. Some of the projects that we're faced with now, and I'd like to go back a little bit, the years when we held our rate I believe our lowest rate was $5.37 in 1996 or something like that. We were postponing projects that probably should have been done at that time. And now we're kind of playing catch-up on some of those projects. There's so many things that are hitting us at one time. For instance, we knew for years that we needed a fifth courtroom. That project was postponed and postponed and postponed until finally the Chief Justice threatened to take that courtroom away from us. Well, we finally did something and we put Judge Bulgert in area for him. That resulted in having to do something for the clerk of courts, and we had to build an administration building and we had to do a lot of things in the courthouse. Had those projects been done sooner, we could have maintained the tax rate in a more level way. Now we're forced with a situation of catch-up. And of course now, due to public demand, we are restructuring our health care centers. We will be closing comprehensive and having two health care centers at Rocky Noel and Sunny Ridge. This is extremely expensive. The cost is going to be, I believe, 9.5 million, I believe. That will be funding in the next two years. We will be doing that. We're just finishing up in 2001. We'll have the last leg of the 800 megahertz system. Our radio system, throughout the county. We were operating with machinery that we could no longer buy parts for. We need to rent some tower space. We need to coordinate all the first responders in the fire departments and all those emergency personnel. That project was, I believe, 7.5 million dollars. So we're just coming in the last leg of that. Over the last, for the next two years, we're putting 13 million dollars into our airport. Our airport really provides a lot of economic opportunity for the county because people are coming in using our wonderful golf courses, eating at our restaurants and doing things like that. Much of the airport money is reimbursed to us by the federal government, but we have to expend the money up front. Also, we did the detention center. We did two. Our detention center is going to be three connected buildings, three circles. Two of them are done. We will be doing another one sometime in the future. It's in the schedule to begin next year, but it looks like due to some decrease in our need for jails that we may be able to postpone that. So there's a lot of things on the drawing board that need to be done. So that's essentially our capital project. Maybe you'd like to add to that, Tim. No, you hit the big points. There are a lot of things coming up that Shboygan County wants to do. This is an aggressive county when it comes to doing a couple of things, taking care of its people and providing a good quality of life here. It is expensive, as you pointed out, to do that. I think the steps that have been taken so far in the budget processes are very encouraging. We have a lot of things on the board we want to do, but we're able to do that within the target we want to set for our borrowing saw. I'm very happy the way things have gone so far. Very good. Now that we've given a little background on our overall operations and expense, I think we should talk a little bit about the 2001 budget process. Our viewers may be surprised to know, but we really start this process six, seven months before year end. And similar to last year, as Tim touched on, we established a goal. We established targets. And there's a real unified goal amongst the county board supervisors that we want to maintain our tax levy rate to the best of our ability. We want to eliminate the high swings and make sure we maintain it at worst, have modest increases. And that's what we're seeking to do again this year. The goal, again, is to maintain it approximately at 6.30. We're looking to level that out. The targets, all departments were required to evaluate their operations capital outlay and seek a 3% savings across the board. And thus far we've met with about 10 of the 12 departments and I'm pleased to share, and I know Tim is going to touch on that as well, that thus far it's going very, very well. Why did the county board develop this goal and targets? Well, again, clearly we want to make sure that taxpayers appreciate and recognize that we are trying to maintain that levy rate. We don't want to see the kind of increases that have occurred in the past as well as we're concerned about equalized value. That's a very important component and preliminary information is that the equalized value may not be as high as it was in past years. So we're taking a very conservative approach to prepare ourselves for that. And finally, and I want to be very clear on this point, a government in general does a very good job justifying new programs. But where we need to improve is evaluating and making sure we're accountable for programs that are in place. And my job as an administrator, Tim's, and all our staff professionals is to annually evaluate these programs, establish priorities, make recommendations, and then let Connie and the other county board supervisors determine what the priorities are going to be as they are the final decision makers. So with that, in terms of the budget process, why don't you give our viewers a little bit of an overview on how that works? Sure, I'd love to. I think it's a very extensive process and we've kind of discussed a little bit, you know, looking at some of the things that we have as far as information and prior year budgets begins in June. And actually, I think it probably begins before that. I think we start thinking about the budget for the coming year well before June, probably really starting in March and April, start to think about the kind of things we're going to have to do the following year. But it begins in earnest in June. And in June, the administrative coordinator developed the targets that they want to see achieve for that coming budget year. That alone is kind of a daunting process to try to make sure that we do the best we can for the citizens and keep the taxes at the lowest rate possible too. So in June, we have, after our targets are established by the finance committee and the administrative coordinator, we go to our budget kickoff meeting which is a meeting with all department heads. Everybody gets in one room and the finance committee this year, it was a joint effort with the administrative coordinator, myself and the finance committee to share the targets with the department heads and let them know what we were establishing as our goal for the 2001 budget year. After the budget kickoff, then each department head is given the charge to go back and develop his or her own budget. With their staff, when they do that, then they go back and they go to their review meeting with the administrative coordinator and myself to make sure that they've done everything they can to meet the targets and to achieve the goals that have been set by the finance committee. Then after the base budget meeting they go to their own liaison committee with their budget that's been reviewed by the administrative coordinator and in a sense then they get the blessing of their committee for what they're proposing for their budget. From their liaison committee approval then it goes to a joint meeting between finance and the individual liaison committees and they throw the budget back on the table again, discuss them, make modifications or amendments as they feel like they need to do. Then from that point it goes to a published notice of what the proposed budget is. From there it goes to public hearings and finally the county board would adopt the final budget in November so it's a long process. I mentioned earlier 10 of 12 of the departments thus far have met the targets and that's encouraging although we haven't come up with the largest departments yet they haven't met with Tim and myself and we're really quality control. Again it's our job to make sure they meet the targets are going through the process correctly and then the madam chair of the finance committee the committee as a whole and the county board really establishes what the priorities are going to be and make the final decision. Connie in your opinion are some of the more difficult departments in terms of meeting the targets this year? Well it's interesting that you say that the departments that have come in so far have come in and have the 3% cut because the department and that's wonderful, I mean I give all these department managers a lot of credit but the departments we've met with so far or you've met with so far have the smaller budgets and are able to perhaps they don't provide services to the elderly or to the disabled people so it's a little easier perhaps for them to do that. When we begin to look at the health care centers we're going to have a problem because of the inefficiencies because of the way we're operating now once we have the two facilities I think we'll be able to handle that the other difficult very difficult areas could be health and human services how can they cut 3% of their budget when 80 to 90% of their budget is matched funds so it's going to be extremely difficult for them and people those are some people who really need the county's help so we're going to be very careful in that area to take a look at that. Recently we had just a tremendous turnout at our health and human services public hearings and we're really looking for more public input and one of the misnomers where the cuts have already been made and that frankly is not the case as certainly you both know but I want to make sure our viewers appreciate that no cuts have been made at this time every department needs to look at their department budget see where they can prioritize see where they can garner some savings and then the county board supervisors will establish the priorities as if the majority of the departments are able to achieve some savings the supervisors may in turn apply that savings to a department such as health and human services I don't know what the end result will be but we're in the midst of the process and no cuts at this time have been made County how will people get more involved with the public with the input and process of developing the budget I know you want more input from the public how do they get involved from the public first of all when people come to me and ask a question or have a complaint or a concern the first thing I'll say to them is who is your county board supervisor and many of them don't know and I'll say where do you vote they don't know that either tell them if they don't vote so I think the first thing to do is find out who your supervisor is express your wishes the other thing you can do is and you can make your feelings known to the county board floor the last thing you probably can do is run for supervisor and then really make your feelings known well very good good advice I know that there's a sincere interest not only from staff but most importantly from county board supervisors to facilitate more public input and we need to hear from people out there if we don't hear from you we don't know what level of service you're looking for well with that I'd like to thank the board supervisor Connie Zigglebauer and Tim Finch for joining me today next month you can count on my co-host and again the county board chairman Dan Lemming you returning and we're going to focus on the sheriff's department there have been a great deal of infrastructure improvements made there in the detention center the 800 megahertz project right now that is being worked on and obviously is very important for public safety and we look forward to seeing you next month so thank you for joining us


Schmidt says hey there is it possible to delete a whole conversation in a channel and how do I leave a team? Boy, someone just wants to drop the microphone and get out somebody said something appropriate They want to clean up their mess and then they want to leave the team. That's what's going on Schmidt. Yep. We see it No, well anyone can leave a team number one It's it's just there you click on the team and lipses button and just go leave the team But when you leave the team you can still be at mentioned so that you can you can kind of be drawn back in being physically removed from the team means you need to actually go and ask them to to kind of Sort of delete you a little bit more. But yeah, there's that but then deleting the whole conversation But it's not all your work. You can only delete your own component. You can't delete others Right Sorry, yeah, because I think of that what's something that works different because a chat the chat capability versus a conversation in a channel Worked differently in a chat I mean, it's between you and the other members of the chat if it's a one-on-one with Kirsty and I if I delete Kirsty still has the full conversation because she shared in that discussion.  No, and if you have a discovery on or or anything like that I mean that's being captured. So if you did say something Schmidt chances are it's it could have been already captured That that could be a possibility, but there's no going back. I mean you delete it it's Something that you can't delete the other person's contacts just like Christian said it's it's still only gonna delete your portion and that You know, if you're trying to completely get yourself invisible from this space I believe the team would have to be deleted Yeah, for the channel at least the channel would have to be yeah, yeah I don't know. It's not like Facebook. Yeah, but so again, but if you have discovered if you have that That set, I mean even if you delete the channel, there's going to be a record of that somewhere. So I mean delete delete and leaving in a chat is coming like the that's that's in You know progress coming out very soon and this is chat not posts Having conversations in a channel but deleting in the chat and and being able to you know leave That's a that's one that's coming down the line very soon. It's being announced there I think that the the thing is a lot of people forget to realize as you know stated quite Michelle This is a business tool. So anything that you say is ultimately going to be kept. So I always say think Before you put something out there because when you if you think about it's like um Say things that aren't going to be you know hurtful and and uh, you know, just stop Just stop and think before you put something out It's a rule of thumb like if i'm going to have a private conversation because you're exactly right I'm very much the philosophy of Like as I believe like all meetings that have to happen that happen on my company's tenant Should by default be recorded and transcripted and made searchable Uh and so and a lot of people don't like that But if I'm going to have a private conversation about something non-work related I'm not going to have it in teams I'm going to go like where where do private intimate discussions happen Facebook messenger, of course Yeah, I would say if you really if you really want to get into it pick up the phone or go sit and have a coffee You know, don't be sitting there and typing it into into chat. It will be discovered. I have being part of many scenarios of Of exactly this where we've seen groups of people Layed off from a role multiple because they were engaging in bullying and they were colluding In the bullying. So, you know, we've seen it. It's like you might want to delete yours But no, this is a business tool. It always has been it'll always be kept So why do you why do you wish to delete and as you said if it's something that you don't need anymore archive it Or take down the whole, you know, take down the whole channel But he's still going to be discovered if it needs to be even if you delete it Yeah, there's also communication compliance. So if you did say a key word and it was being monitored That might have already been scraped and captured. So Yeah If you're upset that your company is monitoring and doing all this Like look, there are liability issues. There's there's so many other reasons that the company is responsible for Again, it's their platform. It's their tenant. It's their intellectual property. You're there because of them They do have a right to that now. There's still You know sensitivity rules There's you know visibility into and doesn't add like we answered a question in a previous recording about Admin access to secure areas and all those things are valid things But at the end of the day if the company gets sued and they do a discovery like It's wide open to them, right? They will get access because it's their intellectual property Yeah, that's right Okay, wait the whip on On I think so But but curse that you know what you said about it's coming on the roadmap I was reading about that that you will be able to delete your conversation But other people will still be able to see it. So it's kind of like more of a remove your own noise But it's still available for everybody else Yeah, and I still got it there even if you've gone


Very exciting today to have Daniel and Jason join us in terms of multi-quip being a successful Western Sydney company, formed based, succeeding over the decades here in Western Sydney, very solid pillars, very solid foundations and joining the academy. You see what we've done in the work and the space and developing the academy here, the players that have come through the first team, very excited, multi-quip here with Wanderers, very excited. Seminal moment solidifies the program, it's been in place now for several years, the players are coming through and multi-quip solidifies the belief in the academy program. One of the foundation pillars of our club, very important structural piece of our future of the organisation, multi-quip with the Western Sydney Wanderers, very excited. It's great, you know, we're really proud to be part of the academy that develops the youth. You know, we think it's a great initiative to bring the youth through and they got a path all the way to the A-League. Western Sydney is where we started, so it's great to be in there with the youth of Western Sydney, bringing them right through, so we're super excited about it.


Now the Federal Executive Council has approved the implementation of the Employee Compensation Act 2010. Minister of Labour Chris Ngige spoke to state house correspondence after an emergency feck meeting. The Employee Compensation Act 2010 makes comprehensive provisions for the payment of compensation to employees who suffer from occupational diseases or sustain injuries arising from accidents at the workplace or in the cause of employment. Apart from the private sector, that is already implementing the public sector which is government, federal government, state government, local governments have now to adopt this for the protection of their workers. The act provides that the worker who is injured or had an accident or contacts a disease or disabled or deaths even in the course of work.


I quite often will have Apache installed on most of my computers if not to serve up websites but to serve up files in many cases. In main case I have my Pogo plug in the other room which is a small cheap little Linux file server. I removed the original firmware and have Debian installed on there and most time I use it for SSH-ing into and I have screen running most time have applications running but I also have it as a place to store files that I can easily access with my phone and tablet and by default when you install Apache using you know pseudo apt-get or aptitude install Apache or if you install through another package manager Debian system I'm pretty sure by default the index is enabled if not there's configuration files you can Google that real quick to enable index like this which is great you know it shows you the names in the last modified size in description and you can organize stuff you know click in on this or click it by name or by size however you want to organize it and you go up directories it's not the prettiest though and also on a tablet you got to make you got to constantly be zooming in every time you move to a new folder zoom in to be able to click on one of these small links today I'm going to show you an option to make this a little bit prettier and a little bit nicer and give you some more functionality out of this and it's really really simple I found this project online H5AI, HTML5 Apache index although it works for other web browsers as well and if you go to the link right here which is L-A-R-S-J-U-N-G dot D-E forward slash well if you just go there you click on the link for H5AI but I'll try to remember put a link in the description that this guy has some other projects here some which are incorporated into this project and it's basically just makes this a whole lot nicer and it's very very simple it has instructions right here on the page but I'm going to go over that right now so first thing you do is download it so what I'm going to do is I'm going to right click this and say copy link as currently looks like he's at version 0.24.0 and once I have the URL I can go into my terminal here on Debian systems once you have Apache installed the default folder for your web browser is VAR www and we will download the files that folder you'll need permission to do that so we'll use sudo and we'll say sudo wget I'll paste in the URL for that zip file and shouldn't take very long to download it's about a half a megabyte and there it is listed you can actually see it now if I refresh this page it's right there so it's in this directory and I am again going to use sudo unzip and I'm going to unzip that file which creates a new folder right here and also includes a license file and a readme file which if we scroll down here you can see that it's under MIT license so is open source and it is a GPL compatible also as I mentioned earlier this works with Apache but it also works with other browsers as well such as Cherokee and light HTPD so if you're using one of these you're good to go as well and they give separate directions on how to get set up but once you unzip it there's only one more well two more steps to do and since we're using Apache we're going to modify you can either modify the HT access of a particular directory if you just want this to affect certain directories or what I'm going to do is I'm going to edit and what he says here is the HTPD dot config file but by default on Linux you're probably going to want to put this in the Apache config file so if I sudo and I'm going to use Vim as my text setter use whatever text setter you prefer etc Apache to Apache to dot config and I'm just going to scroll all the way to the bottom and I'm going to take what he has right here directory index index dot html index dot php and then going to basically point it to where that file is so what this is saying here is for the index of all folders basically when the browser is looking for an index dot html or an index dot php which are the defaults it looks for it's going to basically forward it to your root directory but it's the root directory of the web server not of your computer and all the way down this is basically what we just unzip so you paste that one line in there save that file and then sudo etc initd apache and I know there's something else you can type to do this this is just how I've always done it restart give it a few seconds it will restart and now let's go back here so this is what it looked like before if I refresh now this is what it looks like now we have our files nice it gives everything little icons here we control the size of our icons we control whether we want a detailed look a grid look and I just an icon look looks I haven't even tried this a filter so I guess I can just start typing oh yep and it will filter out the files that's nice I didn't even notice that till just now I've only been playing with this for a few minutes before I decided to do a tutorial on it because I thought it was great you also notice when I'm hovering over files here you'll see a little qr barcode down at the bottom that's part of this guy has another project that generates qr codes and he's incorporated into this that I can hover over these files and take a picture of that qr code and open up my phone now right now I'm going to local host so it wouldn't work but if I typed in a domain name or the IP address to this computer which we'll do here in a moment I would be able to take a picture that with some with a qr code scanner on my phone and open up these files on there so we have this nicer look on the side here we have you know a quick search a basically a tree view is what I'm trying to say if it's a text file and you click on it it brings it up in a nice little viewer like this I can skip to the next file and these are the files that came with it again different icons for different types of files png files jpeg files so I mean it's it's beautiful a lot more a lot nicer than it was before and it works great with mobile devices when I brought it up on my phone big you know icons to click on easily scrollable it it fits to the screen properly and as I say I installed this on my pogo plug running Debian just before I started doing this tutorial and if I go there here I am I can see these files and again I got the qr code now that I actually typed in the IP address of the server rather than local host I could take a picture of this with something that's a some sort of qr code scanner on my phone with my phones uh... camera and it would open up the link that file as you can see as I scroll over the different files it automatically generates qr code for it I think this is just a great project I said that again the guy has other projects one of which being is the qr code here um... so check out his site check out this uh... I recommend it this is great you know uh... nothing built in for uploading files uh... to it uh... but I don't really like that in the web interface I like most part when I'm doing stuff on my own personal server like this I'm copying files over uh... using ssh uh... although you could incorporate some sort of html file upload uh... again if you put in html file into uh... here anywhere it'll work just like an html file uh... like if I go into here and I click on here and click on this and it opens it up this is from another tutorial I did so I mean your web page is still gonna same it's just affecting um... your index your folder views and lot nicer than the default and again good for mobile devices tablets phones and makes your own little cloud experience a little bit nicer rather than giving your files over to a server somewhere else that you're not in control of install patchy install this and you have a great little little interface hope you enjoy this again uh... this project is called h five a i and uh... the guy's site is l a r s g u n g dot d e uh... i actually found this through github i believe so it is on github i'm pretty sure that's where i originally found this just looking for something like this is the first one that came across that really really nice and um... as always i hope that you enjoyed this little tutorial i hope you are going check out this guy's site maybe uh... you know let me know how you enjoy it maybe uh... you know donate something to him for his project you want to donate can also donate to me on my website that's always nice if you like this video go ahead get this video thumbs up if you don't want to miss a my videos be sure to subscribe and i hope you visit my site films by chris dot com that's chris of the k there should be a link in the description i hope that you have agree