Hello, my name is Ankur. I lead the machine learning as a service, MLAS for short, team at Capital One. My colleagues, David Harrington, Patrick Hannis, Trevor Halleck, Cruz Hall, and Christian Langholm, and I are excited to tell you more about our platform and share some of the operational challenges and lessons learned in running the platform on Kubernetes. I also want to give a shout out to Suman Gadapalli, our architect, and Jason Stryker, Allen Mann, and Michael Andrews from our Kubernetes SRE team, who have been incredible partners on our Kubernetes journey. Before proceeding further, let me take a moment to mention Capital One's commitment to the open source community. Capital One made an open source first declaration in 2014, and that's when we made our first contributions to the open source community. We sponsored PINOS, Python, continuous delivery, and the cloud native computing foundations to help keep open source sustainable. Capital One's contributions to the open source community have been significant, and we have released more than 40 of our own software projects. We have invested for years to build the culture and governance required to be open source first in a highly regulated industry. I will now hand it over to my colleague, David Harrington, who will walk you through a high level architecture of our platform. David, over to you. Thanks, Ankur. Hi, so for today's agenda, we're gonna walk through an example architecture of building a simple software as a service platform. We're gonna talk through how we use Kubernetes and how we organize our teams and software in order to accomplish that. Many of the concepts like we aren't gonna be new for most folks who have basic understandings of the primitive Kubernetes objects, but rather our intent with this presentation is for those to walk away with a better appreciation for how we combine the basic concepts of Kubernetes into a higher order system. So diving in at the heart of it, our platform runs ML and data processing pipelines at scale on Kubernetes. We seek to make it easy to run out-of-the-box analytics on the desired datasets in a standardized, secure manner. In this talk, we're gonna cover our journey in using Kubernetes, cover some of the foundational processes, design considerations, and pepper in some examples of incidents to elucidate why getting these patterns right is crucial. And before we dive deep on any one area on the agenda, it's important we level set on what our requirements and high level architecture look like. As for requirements, we need to one, be able to run batch jobs on demand that connects to end users' data stores. Two, we need to be able to enable non-technical users to configure and launch these jobs via a UI. And three, we need to enable least-privileged flexible data access. Now, as a large organization, it's important that we adopt a multi-tenant architecture in order to help ensure least-privileged data access. We wouldn't want people to be able to access data that they shouldn't be able to. And now, one of those important requirements for this platform is that not everyone is gonna be an engineer with direct system access to our cluster. Rather, we need to be able to serve users via UI in addition to API services. Accomplishing the above, provisioning of a job is not as simple as having someone run a coop cuddle apply. Regardless of where the user journey starts, we must ensure the same properties of compute, network, and data isolation. Now, in this diagram, we have somewhat of a stripped down basic version of what our system does. As a given to this presentation, we're running on a Kubernetes cluster, more on that later. And like any platform, we have APIs, UIs, databases, and most importantly, our platform does something of hopefully value to the end user, in this case, running some standardized analytics jobs. Now, what the jobs do for this presentation doesn't matter all that much, other than the fact that they might require customized networking, but these jobs need to be able to connect to the end user's desired datasets as we see here on the right. So, a huge part of this platform is running a reliable Kubernetes cluster. How do you upgrade a cluster of production? What production configuration meets the requirements for your enterprise? What add-ons are necessary for the operation of your platform? These aren't easy questions. We have a central SRE team whose responsibility is to provide platform teams like ourselves the automation tools to provision and manage a production grade cluster. This is the bedrock on which our platform is built upon. Without it, we cannot securely or reliably do any of the fun things like running thousands of pipelines. It's important to mention that a central SRE team does not mean we run all Kubernetes workloads at Capital One of one big old cluster. Rather, for large organizations where you may have many complex platforms, it is not advisable to share clusters across platforms. Needs may differ much lower in the stack making coordination of releases thorny to say the least. This hub and spoke model provides the best of both worlds of a central team of experts and the right size scope limited clusters with more predictable behavior. Now onto the next layer in our architecture. This is a big one, our platform APIs. It is conceivably the entry point for all user interactions on the system. An entire talk can be dedicated just to how to build multi-tenants offers a service applications but that's not the purview of this talk. Rather, what's important about our platform APIs is how they interact with our cluster and the layers below it. When building a multi-tenant system like this, you're always faced with the question of whether you wanna provision a copy of your stack for tenant or utilize a shared service model. What does that mean in terms of Kubernetes? If you have a platform API that needs to create jobs on behalf of users, is it better to have a single service that has permissions to deploy jobs for all users, perhaps across many namespaces? Or is it preferable to have a deployment of your service for each tenant where the permissions of each API is specific to each tenant? The blast radius may differ depending on your approach. And so that's a judgment call that cannot be made universally for all platforms as both options have their own sets of frozen cons. Regardless of whether you have more of a single tenant model for your services or a shared multi-tenant model, we still need to manage resources on behalf of our users, which leads us to the next part of our architecture. Next slide please. The primary function of this layer is to maintain and manage the specifications of resources for your users. The API is not much different in responsibility from your SRE job function, maintaining say, home charts for deployment. The engineer must maintain and upgrade the deployment when appropriate, defrecate it when it's time, and have a disaster plan in place for when the service or cluster goes down. Only in this instance, instead of an engineer committing code to version control and kicking off builds, all of this management has to be codified and automated as to be repeatable arbitrarily many times, which leads us to this layer, the tenant sandbox. For our example platform, we're running jobs for our users. We wanna limit what these jobs are capable of doing as we laid out in our initial requirements. We want a minimal network surface, a limited set of capabilities, permissions, and available resources as to avoid any one tenant causing problems or maybe even snooping where they should. One of the primary functions of the outer platform APIs is to automate the provisioning of the sandbox whenever a new tenant signs up for the platform. And if we manage the environment well, then the next part should be simple enough. We get to run our meaningful workloads for our users. When indirectly opening up what can run on your cluster to a large number of people of varying skills and backgrounds, problems at this layer are bound to arise. In this presentation, we're gonna cover techniques to catch problems as they arise and avoid classes of errors. And with that, I'm gonna hand it over to Christian. Thank you, David. All right. So I'll talk to you about how to go about updating your cluster. Though it can feel a more mundane and routine part of your deployment process than say your platform deployments, cluster upgrades require equal presentation preparation and attention to detail. Your cluster is the foundation upon which your platform is built and therefore can have and lead to unintended consequences when that foundation changes unexpectedly. It's therefore important to have a plan before, during, and after any upcoming cluster upgrades to identify potential regressions, test and monitor for those regressions and recover from those regressions if necessary.  Such a plan is equally important whether you own your cluster scripts or if they come prepackaged for your use like in our case. And a good cluster plan includes the following steps. First, listen for upcoming cluster upgrade dates or deadlines and prepare your team to have a resource or resources on standby for potential failures, rollbacks or hotfixes. Ideally, these support resources have been set aside already as part of standard support rotation and have been prepared with this action plan in advance. Second, make sure you review the change log and any relevant documentation for the upcoming upgrade and identify known breaking changes and suspected points of failure. Third, prepare for the upgrade. If you have identified definite breaking changes, make a plan to implement the necessary changes before the update deadline. It's important to review upcoming changes well enough in advance to make these preparations. Second, if you suspect any changes might be problematic or cause failures, ensure your tests and monitors and alerts cover those potential points of failure so that they can be quickly identified post upgrade. Four, test, monitor and alert for regressions. During and after the cluster upgrade, ensure that your entire test suite is run, including integration tests, end-to-end tests and performance tests. These are powerful tools and identifying regressions quickly. No test suite is a complete picture, however, so it's equally important to monitor your logs, performance metrics and application health wherever possible using whatever tools are in your arsenal. Five, after the upgrade, conclude there are no regressions introduced into your platform and you can safely sign off on the cluster upgrade as a success in the current environment. However, if regressions have been discovered, ensure you communicate your discovery and determine whether you need to roll back and deploy a fix. Only after a successful assessment of your current environment should you elevate your cluster upgrade to your next higher environment. Six, performing the elevation. It is crucial that any change to your cluster or your platform that will ultimately end up in production begins in your lowest possible environment. Only after explicit approval in a lower environment should any deployment be elevated to the next level. It's therefore critically important to allow sufficient time before your upgrade deadline to fully test these cluster upgrades in a lower environment. Now, I will explain a situation that happened to us during a previous cluster upgrade and I'll explain how we learned the hard way, how we needed to implement such a plan as I laid out in previous slide. So back in March of 2020, after several smooth cluster upgrades, our lack of proper planning came to light with a regression that was introduced in our QA environment after a routine cluster deployment. After this deployment over the course of a few days, some of our users began reporting high error rates in the form of 502s and 499s coming from our API. These higher error rates were not caught by our alerts, nor did we have sufficient monitoring dashboards in place to help identify the source of these failures. After triaging and assigning a lead to this issue, as well as informing stakeholders and clients about this issue, we began investigation. Due to our lack of foresight to plan for issues like this, this investigation did in fact have to be done manually and touched many aspects of our ecosystem that many of us were only tangentially familiar with, unfortunately. After concluding, a regression was not introduced within our platform code. Our team struggled to identify the source of these errors, let alone the root cause or solution. And all that we could conclude was that due to the nature of our HTTP error codes we were seeing, network traffic was severed somewhere along the way. Though, after some thorough investigation, we discovered logs from our ingress controller, which pointed to it as our source of failure. Further network testing on the ingress controller confirmed a high rate of connectivity problems between it and our server pods. Further investigation along with our network administrators revealed an undocumented regression in our CNI plug-in, CILIUM, which was blocking inter-node traffic for pods attached to certain outdated network policy definitions. Our ingress controller was attached to one such policy and after updating the policy, network traffic returned to its former working state. This trial by fire introduced our team to aspects of our platform ecosystem, which were previously unfamiliar and revealed to us many new potential points of failure worth testing and monitoring. For example, the image on the right summarizes all the hops, skips and jumps our network takes, our network traffic takes to get from our client to our server and back again. As we locked a consolidated view of our system, we had to start at both ends of this flow and test and inspect each stage for network issues manually. This involved running commands like NSLookup to confirm DNS resolution and at points even running simple curl commands from within our ingress controller pods to test connection to our downstream systems. This process was unsustainable to say the least. Ultimately, we concluded that the network error originated from the connection between our ingress controller and our server, but we determined it's worth testing, monitoring and alerting on every stage of our network flow. Since then, we've since made efforts to consolidate our logs from each stage into a dashboard on Splunk, as well as monitor each stage for health and performance metrics using tools such as Newerlick and CloudWatch. We've also revitalized our knowledge transfer sessions to teach our team about the various supporting actors in our overall platform ecosystem. So let's analyze our experience with this particular cluster upgrade. What went right? What went wrong? How could we have improved our experience with a proper action plan like we described earlier? Starting with the positives, to ensure customer questions and issues were promptly addressed, we have built a multi-layered support system and this support system worked as intended in the resolution of the issue. We triaged and prioritized the issue, assigned a lead to resolve it and effectively collaborated using Slack and Zoom to troubleshoot with our team and with our cluster administrators. Simultaneously, we kept the impacted customers and leadership informed on our progress. Additionally, despite our setbacks, we were able to identify the root cause of failures and iterate quickly to deploy the fix and before the cluster hit our production environment, we were able to resolve the issue and continue as normal. However, despite our quick turnaround on this particular issue, we did expose ourselves as insufficiently prepared for cluster upgrades in general. A few glaring issues can be identified from auditing the upgrade process from the perspective of a platform team. First, while we were reviewing the release notes for this prospective cluster upgrade, we failed to identify the Cilium version update as a potential source of failure. Correctly identifying this risk would have narrowed our investigation significantly and since we have taken care to call out any version upgrades coming down the pipeline to investigate first in case of detected regressions. Second, our testing, monitoring and alerting suites proved insufficient to notify us of any failures outside of the scope of our immediate platform components. We only discovered this particular issue after being notified about degraded performance from our clients. And when investigating, we had to cobble together logs and metrics from disparate sources. A consolidated testing strategy and monitoring suite would have identified this regression quickly and we've since begun consolidating long streams into centralized dashboards. Third, before this incident, many of our development teams members had a tentative grasp at best on some of the systems in our platform ecosystem outside of our immediate platform components. A firm understanding of these systems would have resulted in a more confident and robust debug process. We've since ensured that our team has several SMEs of our platform ecosystem and beyond. And we also do regular knowledge transfers to elevate the rest of the team. By addressing the above gaps, we've conducted subsequent cluster upgrades with confidence in our game plan. I'll now hand it off to Cruz to discuss observability. Thanks, Christian. So turning now to observability, I think it's really helpful to frame any discussion or any work around observability in terms of the target outcomes. And so for us, there are really two outcomes that matter and all of our observability work drives towards enabling these outcomes. The first one is probably familiar to you and that is minimizing our time to restore and that time to restore captures how long it takes to bring the service back up whenever an incident occurs. And it's also helpful to think of that journey in terms of two separate stages. First is the stage where you're actually waiting for your on-call engineer to detect that there's an issue. And so that can be measured as the time to detect. And then once the on-call engineer knows about the issue, there's another delay as they figure out how to restore the issue or rather how to fix the issue is at our issue to fix. And so that diagnostic process, we can kind of capture as the time to repair. And so by thinking about time to restore in those two categories, it helps direct our investments to the most valuable work. And so in the case of alerting, alerting is there to help minimize that time to detect. It helps us to learn about issues before our customers come and report them to us, as Christian mentioned earlier. And then we have other levers to pull to minimize the time to repair. So specifically that's things like having a run book so that some common remediation steps are easy and apparent to the on-call engineer and also having very fine-grained application performance monitoring or APM. And that also looks like having distributed traces and even metrics that can point to very specific parts of your stack so that you can easily see where the issue likely is. And so by thinking about the time to restore in terms of these two components, it really helps us to optimize and focus on the most valuable instrumentation efforts and even efforts that may not involve instrumentation such as writing run books. And then another really important outcome is to maximize the legibility of the system. And so we can think about legibility as a measure of how readily the system can be read and understood by anyone on the team. If you have a situation where only the experts know about this one piece of the ecosystem, then if that is where the incident is occurring, then you have to get that expert on the phone. But if you've done a good job instrumenting the system and centralizing those signals to be viewed in one or a few places, then suddenly everyone can kind of be an expert. And in that sense, we've really improved the team's ability to respond to issues and to understand what's happening in the system. And so we can also think about our dashboards as opportunities to sort of answer some known questions of some frequently asked questions, but even that has limitation. And ultimately we want to have opportunities to ask any question of the system with logs that are really rich with context and traces, which also can show the interdependencies between the system and help us even diagnose very subtle issues with high resolution traces. And so in our system, as mentioned earlier, because we're living in this layered world, we have a couple of unique challenges. So the first one is our shared responsibility model. Our MLIS team is going to own the compute infrastructure and the platform APIs that ultimately help orchestrate our workers or whether our workflow jobs that our users are running, but then the individual users, those tenants, they have their own code that they have to attend to their settings, certain resource specifications. Obviously they're writing the logic that could go wrong. And so oftentimes it's been difficult to parse through the root cause of an issue and know whether or not it's ours to solve or if it's a user to solve. Or in some cases it's even up to the cluster administrators to resolve certain issues. And so keeping that in the front of our mind has been really important as we instrument the system. And then another unique challenge, and this is sort of specific to the way our platform is orchestrating our users' workloads is we have very short-lived pods. And in some cases, the pods that are most problematic, the ones that our users want to debug and dive into further, those are the pods that are only living for a few seconds, if that long. And so it turns out that there are several tools in the observability of the ecosystem that rely on a pool-based mechanism. And so we'll learn how that has become problematic in some cases to sort of outline a few of these tools that are operating at different layers in the stack. I think that this diagram is useful yet again for illustrating core observability plays at each level. So at the cluster level, we really do rely on our cluster to provide some of those foundational capabilities of collecting metrics and logs and traces and shipping those to our tools of choice. So in our case, we're using the Fluent D, David said, to collect the logs on individual hosts. And then those logs are ultimately forwarded to Splunk using the HEC plugin. Even there, we have two different projects. The HEC plugin is this project separate from the Fluent D project. And so there have been cases where we have to dive into issues that may originate from the HEC plugin and then others that seem to be related to the Fluent D, David said, pod. And then apart from Fluent D and Splunk, we have a New Relic operator that's collecting infrastructure utilization metrics and it's essentially providing that infrastructure view in New Relic. And then in addition to the metrics collected by the New Relic operator, we also have Prometheus collecting metrics, not only from other applications in the cluster, but also a lot of these plugins that we rely on. And those metrics are also consolidated in New Relic so that we can see them side by side and debug issues as needed. And I can even speak specifically to the Ingress controller that Christian mentioned earlier. We rely on the IngenX Ingress metrics that are exposed and collected by Prometheus to see into that pod, because otherwise there wouldn't really be a good way for us to understand what's happening inside of that, inside that pod or that application because we didn't write that and we didn't instrument it. And then the layer that we actually have control over is the platform APIs. And within that layer, we're implementing some very fine-grained instrumentation with New Relic APM. And that allows us to have distributed tracing, as well as error reporting. And that error reporting has been really valuable for detecting very specific errors with our clients code. And we're always leaning into improving the granularity of the errors that we're able to see in New Relic. And it's this layer that we're actually pointing all of our monitors for alerting. We've learned that alerting on anything that our on-call engineer actually can't resolve has been really frustrating. And it's been hard to actually understand what issues are worth interrupting and beginning to triage. And so we've really just tried to focus all of our alerting efforts on this platform API layer. And that's really helped reduce toil and reduce churn on the part of the on-call engineer. And then moving on to the tenant sandbox, we have several guardrails, including the resource quota construct. And those guardrails help us to prevent those tenants from exceeding certain resource limits. And that is ultimately the best approach to preventing those tenants from having too big of an impact on other systems that are running alongside them in the cluster. That's a much better approach than simply alerting on high resource utilization and requiring someone to take an action. So we're able to leverage that resource quota construct to essentially provide that guardrail by default. And then also in addition to the logs and metrics that we capture from our platform APIs, we're also capturing all the logs from those tenant namespaces. And we're making sure that each tenant has the label that they need to be able to pull the logs and see the logs that are relevant for their workloads. And then lastly, within the actual jobs themselves, we have application logs that our users are going to be writing and they understand best. And we're labeling those and surfacing those in tools like Splunk where the users can diagnose issues related to their application code. We don't yet have metrics and traces and structured logs throughout those batch jobs that we're pursuing those opportunities right now. As mentioned earlier, there are a couple of interesting considerations when you have short list pods. So for instance, with Prometheus, you might actually not scrape that pod before the pod dies. And so there is a chance that your pod metrics wouldn't be presented or they would be presented in an incomplete form. And then for the observability and the implementation of the individual API calls using things like open telemetry, we know that there's a non-zero performance impact to that interpretation. And so we're trying to find ways to measure that impact and quantify it. And then ultimately give our users an opportunity to opt in and maybe even control the granularity or the sampling rates so that they can essentially make those trade-offs themselves. And so that's sort of the state of open telemetry and those more advanced mechanisms of tracing within our individual batch jobs. So now I'll hand it over to Trevor who will sort of explain a recent issue that we had with logging and how we resolve that in our platform. Thank you, Cruz. I would just like to dive into one entertaining case study I will affectionately dub my logs are missing. So as earlier mentioned earlier, our SRE team maintains an enterprise cops repository providing our clusters with several conveniences such as Fluent D. This comes with a seemingly handy cube Fluent D operator self-described as a Fluent D config manager with batteries included config validation, no needs to restart with sensible defaults and best practices built in. Based on the theme of verifying defaults as well as the studies foreboding title, the question becomes, which of the defaults was not so sensible for our environment? And I ask that because verifying the defaults has been a cause of several logging pains, especially the default resource specifications. To answer that question, we'll look at how a log message is built using this pattern. So the log starts its journey and the data scientist's job, hoping to tell the world about the looming issue in prod just as all little logs do. The log is written to the file system and scraped by a Fluent D file source. Next, on a schedule, the log router queries the cube API for the pods metadata and showing that the logs author is able to find it. Finally, the enriched log is sent to Splunk to be united with its engineer. By default, the log router queries the cube API every one minute, which is fine for long running pods such as web servers and the like. However, in an environment with many moving parts, which logs would you suppose people are most interested to find? The ones with errors. It is well known that errors can dramatically decrease a pod's lifetime. Sometimes by so much, the pod is long gone by the time the log router asks for its metadata. This leaves our lonely log without any labels. Now, logs without labels are not impossible to search for. With so many logs in our Splunk index, we found that logs without labels provide little more value than no logs at all. In fact, some of the configurations users add a sleep greater than a FluentD's refresh interval just to avoid this issue. Of course, another option is to configure the FluentD query to query the cube API more frequently, but some jobs have a sub-second lifetime. So there's a limit how often you can do it. Thus, we're moving towards a structured logging approach to remedy these missing logs. By enabling the Kubernetes downward-facing API, the pods creating the logs can also retrieve the Kubernetes metadata required to make them searchable. As shown here, we can more reliably provide meaningful logs by entirely avoiding the fallible enrichment step the FluentD operator offers. In short, our logs themselves are created batteries included. Also, as a last note, when it comes to logging your logging, it is helpful to enable FluentD metrics. In our case, Prometheus sends FluentD metrics for the number of errors in the cube buffer length to New Relic for alerting. This lets you keep tabs of even the logs you missed. Enough of this use case, though. I'm handing it back to Christian. Thank you, Trevor. I will now be discussing how to isolate your tenants' compute within your cluster and your platform. And I'll discuss how the machine learning as a service platform does so. So at its core, the machine learning as a service platform provides its clients, known as tenants, the means to author, manage, and execute their data workflows. Given the unique and open-ended functional capabilities of a workflow, each running instance of said workflows considered its own application. Thus, as the trusted host of such applications, the machine learning as a service platform needs to ensure compute isolation in order to prevent tenant workflows from gaining access to other tenant workflows and data. And if complete isolation is not achieved, there is a high risk of data exposure or denial of service. Therefore, machine learning as a service platform uses Kubernetes namespaces to implement a multi-tenancy model for compute isolation as namespaces provide a means to house or isolate groups of resources within a cluster. In this use case, namespaces are used to house tenants running workflows. Every tenant has their own namespace and tenant workflows run in their designated namespace only. Isolating tenants to their own namespace allows us to administer tenants individually with their own configurations, limits, and permissions. Primarily, we use namespaces to ensure least privilege access and to manage resources on a tenant level. For example, using network policies and role-based access control, we can configure least privilege network and resource permissions respectively, allowing us to limit which services our tenants workflows can communicate with and which resources they can modify. We can also prevent denial of service and resource hogging by setting resource quotas and metering resource consumption, which we'll discuss in more detail later. We apply all of these configurations and permissions on a namespace level to apply the tenant workflows across the board. Now, namespaces are a powerful tool for administrating and organizing your tenants in your platform, but in practice, you're going to run into some overhead when it comes to configuring and deploying said namespaces. The primary consideration to tackle when administering namespaces is determining the minimum provisions needed by the fewest entities to meet your namespace administration requirements. As with all cluster-scoped resources, the cluster-wide nature of namespaces necessitates due diligence when assigning permissions to create and modify them as a misconfiguration could, in fact, open a security gap which could expose all namespaces in a cluster. Therefore, it can be simpler and safer to offload the work of namespace management to a separate service dedicated for such a purpose. A Kubernetes operator could potentially fit this use case. Operators, which offer extensions to Kubernetes that define the deployment and management of custom resources which extend the vanilla Kubernetes API can be used for namespace management. In this case, the custom resource would consist of potentially a single definition file which wraps the definition of a namespace and any additional resources which live within that namespace. The operator would then manage the deployment and maintenance of this custom resource and therefore the namespace and objects defined within it. It's therefore a good idea to investigate if any such Kubernetes operators exist which fit your namespace management needs as it is possible that there exists a well-defined operator within your organization or an open source which you can use to create and configure your tenants and namespaces and resources. By offloading namespace management to such an operator, you're ensuring proper segregation of duties and least previous access for your application. If you choose to follow this pattern, consider the following when choosing a namespace operator. First, as these operators deal with a critical cluster-scoped resource and therefore can have a cluster-wide impact in case of failure, discuss the permissions this operator needs and the impact on existing namespaces in your cluster with cluster administrators and any other applications on your cluster before deploying such an operator to your cluster. And second, research the level of developer support this operator has. Whether it's open source or in-house, you should see how frequently the operator is contributed to and whether it has any critical outstanding issues in its repository and whether any developer team exists to offer integration or bug support. We had to consider these items when deciding between two such operators to conduct our namespace management for our platform. These operators were the hierarchical namespace controller and open source operator and embark, a namespace operator within our organization. When we were exploring operators to manage our tenant namespaces and resources, two promising options presented themselves. First, as stated before, the hierarchical namespace controller. This is an open source operator which establishes a hierarchical relationship between namespaces, allowing for the creation of the concept of a child namespace to which you can propagate resources or configuration items from a parent namespace. In our case, this paradigm would allow us to create tenant namespaces as children of our applications main namespace to which we would automatically propagate standard network policies, roles, role bindings and other resources automatically. Changes to these resources and configuration items and the applications namespace would then automatically reflect in the child namespace. Our second option, embark, is an operator developed within our organization which alternatively defines and operates a custom resource called a super namespace. This super namespace object lets you define a namespace and any resources within it, like network policies, roles and role bindings within a single YAML file. Deploying or modifying any of those items can be done by simply modifying the super namespace definition itself and deploying that object. The operator handles the rest. This option would have allowed us to configure all of our tenant namespaces from one single standard template and deploy a super namespace per tenant which would thus deploy their namespace in all of their resources. A promising option embark was only lacking one feature and that was the ability to configure custom additional resources on a individual super namespace level. By default, you were able to configure objects on an operator level that would apply to all namespaces in the cluster but was missing the feature to do so on an individual super namespace level. If we contributed to embark to fill this feature gap, both operators from a functional perspective would have provided a viable solution to host our namespace management. However, after taking into account the considerations mentioned in the previous slide, even with that feature gap, embark stood out to us as our only single viable option for namespace management for the following reasons. First, the hierarchical namespace controller lacked an official Helm chart to install and maintain it on our cluster, instead requiring installation through a tool called crew. As crew is not a supported tool in our enterprise clusters, we would have had to take the time to create and maintain our own Helm chart for this operator. Now, in the long run, we were considering taking this challenge on and contributing it back to the open source project, but we did hesitate given our internal deadlines and our relative experience at the time with Helm. And given the following concern, which I'll state now, we certainly did not want to make any mistakes when creating the Helm chart for this operator. And that is because of the following reason. This was the main issue we had with the project. And that was that the hierarchical namespace controller has a very broad impact on the rest of our cluster beyond just simply the tenant namespaces we wanted to create using it. By default, the operator placed webbooks on every single namespace in a given cluster, which would therefore trigger admission controllers on upon any namespace modification. That would be for tenant namespaces, our platform namespace, or any additional namespace that exists in our cluster for other purposes. Therefore, in the case of the operator failing or going down, this could have had a potentially cluster-wide impact on namespace creation or modification and could have potentially blocked those actions altogether. And this would include non-tenant namespaces necessary for other platform operations. As a best practice, we wanted to minimize the scope and impact of any operator we install in our cluster. And this control over non-tenant namespaces proved too big of a risk for us to take on, especially without a standard helm chart for us to install. Embark, on the other hand, had no such issues, establishing no admission controllers or webhooks and having no impact on namespaces not managed by the super namespace custom resource. In other words, the only namespaces that Embark would have touched would be those which we were creating for the purposes of our tenants. Because of these two reasons, ultimately our team chose Embark to manage our tenant namespaces. And we were confident in that decision having weighed the impact and level of support for each operator. So if you are in this situation in which you want to manage tenant namespaces and you want to offload this to an operator, certainly consider the same things that we considered for that purpose. I will now hand this off to Patrick to discuss rate limiting and resource management. Thanks, Christian. In order to reduce the impact one user can have on the overall system, implementing rate limiting per client on our APIs is an important step we can take. The goal is to put limits in place so one client cannot push the service over and cause outages for all other users of the platform or cause a response times to fall beneath what is defined in our SLA. The biggest consideration from implementing rate limiting is at what layer in the application stack to put the limits in place. The API gateway, ingress controllers and the app layer were all taken into consideration and their pros and cons were weighed. Not all of our traffic goes through the API gateway, something at this layer would not catch all traffic possibly leaving us exposed down. Rate limiting a ingress controller layer was considered. We already used the Nginx ingress controller in our architecture. So we explored its rate limiting offering but with its complex setup, difficulty to monitor objective requests and inability to limit by anything besides IP, we did not pursue this option. Other ingress controllers such as traffic were explored but similar pitfalls and needing to add an additional layer to our architecture led to us beginning looking at limiting on the application layer. Since our API is written in Python, we explored the rate limiting packages that exist and found the offerings to be what we were looking for. ASGI rate limiting is the package we are currently testing with its simple implementation, rule-based rate limiting and custom authorization function. We found a solution that is easy to maintain moving forward and does not require major architectural changes. With the amount of jobs running in our cluster, we have seen some situations that required manual cleanup of pods and hanging states and other resources that stuck around. Kubernetes offers a TTL. This only applies to pods and jobs in a finished state. The Kubernetes de-scheduler offers a way to configure max runtime on pods. This is just a hard limit and could lead to preemptively terminating workflows that do not meet our criteria to clean up. This led to a design of a cleanup nanny process deployed in our clusters. Using Kubernetes cron jobs, we have a job schedule that checks multiple criteria such as logs of a pod to determine if a workflow is still running. The job terminates resources in our cluster and then issues status updates to our database if applicable. To reduce the load on these cleanup processes, we deploy a cleanup cron job in each of our tenant namespaces. This also helps reduce the blast radius if one cleanup job fails rather than one cleanup job handling the entire cluster. This also helps tenants stay within their resource quarters. This nanny process that handles jobs and pods has been generalized to handle other parts of our system as well. We did face unintended consequence of this cleanup job where it was actively hurting our cluster's health. The cleanup job identified failed pods to delete. However, the job remained active and it continued to spin up failing pods. As this cleanup job deleted pods, it was also resetting the failed pod count for that job, never letting the job fail. We resolved this by not cleaning up the pods directly, but by just letting the job fail and cleaning up the job itself. The delete on the job propagated to clean up the respective pods cleaning up everything. With the implementation of this nanny process to clean up resources in our cluster, we saw a saving of over a million dollars in compute spend per year. Now I'll hand it back to David to wrap up. Awesome, thanks, Patrick. Yeah, so to wrap up, we've laid out the requirements for our sample platform and we took a bit of a tour of all of the organizational processes and technical considerations. We found important in building and maintaining this platform. For those building higher order systems on top of Kubernetes, we really implore you to spend time upfront toward establishing these types of processes for maintaining a production grade cluster and its upgrades, having full observability throughout your stack, ensuring least privilege access. And finally, when building platforms that pseudo extend the Kubernetes control plane, be mindful about what your services actually exposed to your end user. Impart those safeguards to avoid the eventual non-ideal user that we'll call them. And with that, thank you for watching and that's it.


Hello everybody, it's not over sold back here with another video and Today, I thought I would talk about things I've learned in 2018 And it wasn't a lot and I'm gonna be totally honest with you I probably haven't like fully comprehended all of these cuz I'm still a dumb bitch, but um, you know, I I feel like I Feel like I got some of these things down at least and I got exactly three of them because that's the amount of Speedpaints you're gonna be watching. So I hope you guys enjoy So the first thing I learned this year was living my life on my own productive schedule And what this basically means is that when I first graduated high school It was the most dreadful thing I had ever been through of that year at least um And that might sound like nothing But when I graduated high school, I had no idea what I was gonna do Everybody was telling me that I needed to go to college and I didn't like I felt in my heart that I didn't want to go To college. I didn't need college So I didn't but the bad thing about not having college was that somebody else was not going to create my schedule for me Being at home trying to work on my own and trying to create my own sort of little thing Really requires some time management skills, which is something I didn't learn in high school They never actually taught me how to manage my time well aside from Like actually scheduling me, but that's not really teaching you you know because they're doing all the work They're doing all the scheduling and I figured as long as it got done It would I could do it at whatever time I wanted But something I quickly learned when I had first gone into my vacation or just like my Full term not being in school anymore was that I had to create my own schedule everything I had to Do had to be done within a day. Otherwise. I wasn't productive every single task had to be created and That's not something I was good at at first So I would sit down on my bed for the entire day and just scroll through Tumblr Twitter whatever social media that I was using at that time and I just I never did anything with my time Which is why I wasn't productive for a little while. I think even in July. I went MIA That you'll actually see I'll talk about that later on Because I I just didn't do anything and I didn't think I needed to do anything And I wasn't used to doing anything outside of what I really had to do So just I wasn't productive And so what I had to do was I kind of had to make to-do lists for myself and make sure that everything on that to-do List got done and make sure that I kept myself up to my own standards Which is why I still post once a week as opposed to how I used to post like never Because I actually had to keep myself doing things and the way that I learned how to do that Was just to instill some sort of force in me that made me do things and I still don't know what it is Maybe it was a sheer panic of like my parents disapproving of me or my entire family disapproving of me because I didn't go to college And I wasn't gonna be anything if I didn't actually hustle So I had to try and so graduating really kicked that into gear and it took away the the time schedule that school gave to Me and made me do my own The second thing that I learned how to do in 2018 sort of I guess I'm still not perfect at it Was keeping my creative flow So I was talking about how I went basically missing in action in July I didn't post anything and so I actually made this graphic a poster right here I actually made this graphic where I showed all of the artwork I did once a month and I think it's supposed to show how you improved except I didn't really improve I just kind of drew and hoped for the best but I mean I Didn't know how to keep myself drawing and that's something that's also evident in my Instagram if you were to go on my Instagram I didn't post some months.  So you really got to keep on top of yourself If you're gonna do something like that. So in order to keep my creative flow Um, I had to tell myself that I was excited about doing whatever I was doing Like for example, if I wanted to draw, I had to tell myself like, oh, yeah, it's gonna be fun drawing Come on, Lonnie. It's gonna be you're gonna have a good time drawing. Let's draw. It's gonna be good And in that way, I was sort of able to trick myself that What I was doing was exciting and fun and then I would want to do it Because the biggest thing for me was getting stuck on social media because I Felt like laying in bed and scrolling through social media would be way more fun than ever doing artwork Which was my biggest problem is that I wasn't doing artwork because I preferred laying in bed But if I told myself that it was gonna be fun and that I was gonna have a good time doing it Then I could keep myself going which is I think kind of why I even finished october Was because half the time I was telling myself. Oh, yeah, it's gonna be fun Then product is gonna be exciting drawing today is gonna be exciting And I was able to sort of trick myself into working and being productive and I still sort of use that now So that I can continue to post monthly or weekly. Yeah weekly. I post weekly. I don't post monthly. Trust me guys But um And that with that, you know, I I stopped not posting once a month I think I post at least four times a month now if anything because I do a once a week sort of thing But hopefully it's gonna I keep myself on this because I do want to go forward with an art career And that means I gotta stay on top of myself, you know, it's gotta it's gotta be consistent. It's gotta be consistent, you know This one this last one isn't completely related to art It's just kind of something that was a huge thing for me that I learned back in 2018 or you know in 2018 since we're still here Um, I had to learn how to calm my overactive brain and be less anti social So I didn't talk to very many people and I didn't have a whole bunch of friends because I was too afraid to talk to them I always thought that they never that they didn't like me as soon as they met me that they thought I was weird That I was being too overbearing and so I would keep myself from a lot of social interaction because of this so I I would like get in conversations and halfway through the conversation It would go silent and awkward just because I couldn't figure out what I was gonna say and I felt like every next word needed to be calculated when it really did it So something big that I had to learn was to just like get out of my own head and realize that people don't hate me as much as I think they hate me and Keep myself from getting that social anxiety and just talk with people and realize That people are probably more concerned about what they are saying more than what you're saying And a huge thing for me was intimidation By pretty people and not that they were trying to intimidate me or that they were intentionally being really mean to me It's just I was really intimidated by pretty women especially because I always felt like I was lesser than them that I just wasn't as pretty and I shouldn't have been talking to them And maybe that's like an old way of thinking but like my little stupid brain was like Yep, that's the way we think now pretty people are not to be messed with So I would also hold myself back from talking to a lot of like influential people like I knew some people that had done big things and I didn't want to talk to them because either they were successful or they were pretty And that also kept me from a lot of opportunities, you know And that's something that I also had to work through by just forcing myself to talk to these people Whether I wanted to or not and holding a conversation with them And what I found was that it actually wasn't that bad as soon as I got out of my own head And I got out of me thinking that they hated me all the time. It sort of got a little bit better And I'm still guilty of this. I still hold myself back from a lot of social situations But I've gotten a lot better with it and I talk to a lot more people now and hopefully I'm just I just seem less like an awkward turtle around them like before this I was probably less pleasant to talk to you because I was just so nervous But now that I'm less nervous now that I don't care as much as what people think of me It's made it a lot easier to communicate and probably a lot easier for people to talk to me or you know, I hope So these were three things that I learned in 2018 guys and I hope you guys enjoyed And let me know what you guys want to see me draw in the comment section below And this is the last video guys that I will be posting um Until 2019 so I will see you guys Next year Not next year as in like I'm not gonna post for 12 months next year as in guess what next month is 2019 That is so wild 2018 went by so fast So I hope you guys enjoyed this video and thank you guys for watching. Love you guys. Bye


Good day everybody. Dr. Sanjay Sanyal, Professor of Department Chair. So this is going to be a demonstration of the spaces around the liver and the peritoneal ligaments. This is the right time to show you because after this we will remove the liver. So we have removed every structure from the abdominal cavity and we can see the liver here. First, let's start with this ligament that we can see here. This is Fatsiform ligament. The word Fatsiform means sickle shaped. Therefore, it is a triangular shaped peritoneal fold. One edge of the triangle is attached to the liver here and this is the one which demarcates the liver into an anatomical right lobe and an anatomical left lobe. The other end of the Fatsiform ligament is attached to the under surface of the antriabdominal wall and that is we can see it here. And the third margin of the Fatsiform ligament is free and running in the free margin of the Fatsiform ligament is this additional embryonic structure. This is called the round ligament of liver. The round ligament of liver is the remnant of the umbilical vein, the obliterated umbilical vein which in fetal life carried oxygen from the placenta to the fetus. The next structure that we can see here is this is the lesser momentum. More specifically, the hepatogestric part of the lesser momentum which came from the lesser curvature of the stomach and gets attached to the fissure for the ligament and venosum. We have just kept it here. Now let's take a look at the liver itself and for that I have lifted up the right costal margin to show you the liver and we can see this is the anatomical right lobe and this is the anatomical left lobe. I am putting my finger under the diaphragm and the anatomical right lobe and this place. This place adds similarly this space. These two spaces are referred to as the sub-phrenic recess meaning under the diaphragm. This sub-phrenic recess communicates with the general periodontal cavity. However, as I mentioned just a little while back, this is the falciform ligament and this falciform ligament separates the sub-phrenic recess into two parts, the right part and the left part. The next space that I want to draw your attention to is this space on the visceral surface of the liver. This is called the sub-hepatic space. This is the one which communicates again with the general periodontal cavity and we can see clearly the sub-phrenic recess also communicates with the sub-hepatic. If I were to put my finger further deep inside from the sub-hepatic space and it's going in right now and I'm lifting it up here to show you that space. This is known as the hepatorenal recess also called the Morrison punch. This is present on the right side between the right anatomical lobe of the liver and the right kidney. This hepatorenal recess if I were to put putting my pushing my finger at one point my finger does not go any further. So therefore the posterior limit of this hepatorenal recess is the posterior layer of the coronary ligament. Just like the posterior limit of the sub-phrenic recess is the anterior layer of the coronary ligament. Hepatorenal recess is one of the two most dependent parts of the abdominal cavity when the patient is lying down. So therefore if there's any abnormal fluid collection it tends to gravitate in the hepatorenal recess. The other most dependent part of the abdominal cavity is the pelvis which we can see here and we can already see liquid fat collected there. Just to reiterate these sub-phrenic recess, the sub-hepatic space and the hepatorenal recess they all communicate with each other and with the general period of the cavity. Now I'll draw your attention to this ligament structure here. This is the left triangular ligament. It is formed by the fusion of the anterior leaf of the coronary ligament and the posterior layer of the coronary ligament which we cannot see here. Anterior layer of the coronary ligament is formed by the splitting of the phalseiform ligament. We are supposed to see a right coronary ligament which is faintly visible on the right side here. That is also formed by the fusion of the anterior layer of the coronary ligament and the posterior layer of the coronary ligament. But we cannot see the posterior layer of the coronary ligament. Now let's take a look at the visceral surface of the liver. We can see this is the fissure for the ligamentum venosum and the round ligamentum liver. So this is the one which has demarcated the liver into a left anatomical lobe and a right anatomical lobe. On the visceral surface we can see this is the gallbladder. So this is the gallbladder fissure. When we look further we can see these three structures here. One here, one here and one here. These are the structures of the extra hepatic portal triad which were enclosed in the two leaves of the hepatodeodontal ligament and they are all entering the liver in this region here and this is referred as the portahepatus. So what are these three structures? This is hepatic artery proper. The hepatic artery proper is the continuation of the common hepatic artery. Common hepatic artery makes an upward curve and at the point of the curve it gives up this branch. This is the gastrododontal artery and after that it becomes known as the hepatic artery proper. This enters the portahepatus. The next structure that we see here, this is the portal vein which is formed by the union of the superior mesentric vein and the spleenic vein and the third structure that we see here, this is the common bile duct which is formed by the union of the cystic duct made from the gallbladder and the common hepatic duct. So these are the three structures which enter the portahepatus and the margin of the portahepatus, we can see remnant of this structure here. This is the peritoneum of the hepatodeodontal ligament which is attached to the margins of the portahepatus. My instrument has got under the extra hepatic portal triad and if the extra hepatic portal triad had been enclosed in the hepatodeodontal ligament this case would have been known as the epiploid forabinum, Winslow and my instrument now has reached the place which was the lesser side or the omental bursa and similarly the omental bursa has got also a superior recess where my finger is gone in and that superior recess is also demarcated by the posterior leaf of the coronary ligament. Omental bursa also communicates through the omental foremen with the hepatorenal recess. We can see this portion of the liver here between the fossa for the gallbladder and the fissure for the round ligament of liver. This portion of the liver is referred to as the quadrate lobe of the liver because it is roughly quadrangular shaped. Quadrate lobe of the liver forms part of hepatic surgical segment 4. So this is what we can see in this particular view. After this we were going to remove the liver and before I conclude I would like to draw your attention to the consistency of the liver as we can see it here. Here if you look the liver is firm in consistency and this is the inferior margin of the liver which is also quite firm. But if I were to look at the visceral surface of the liver you would find that the liver has got abnormal texture and it is soft in consistency and my finger is dipping in very easily and here also it is dipping in very easily. So obviously there is some pathology in this portion of the liver which we shall determine once we have got open the liver. These are the points which I want to demonstrate to you coordinating to the structures around the liver including the spaces and the ligaments. Thanks for watching Dr. Sanjay Sanyal signing out. David over the camera person if you have any questions or comments please put them in the comment section below. Have a nice day.


You may never hear this about the narcissist in another video This is something you may never hear in another video It's something that you may not want to accept But it's the truth and it's something that you need to recognize Many videos about narcissists on YouTube Talk about how important you are to the narcissist But the truth is You're really not that important to them You're not that significant It's an illusion They're giving you a deceptive appearance They're giving you a false impression You may be in a relationship with a narcissist And you may think you mean something to them You think you're something significant But it's just an illusion You will never be anything important to them Because they just see you as something temporary They see you as something that serves them for a limited time Something to sustain their existence You're just a toy to the narcissist And eventually they get bored of the toy They no longer take care of it They no longer give it any attention They just forget about it Because it no longer has the same effect on them It no longer excites them They're not that interested in you Because the narcissist is constantly distracted by other things They're always looking for the bigger better deal The next shiny new toy Which is why they give other people more attention Than they give to you They treat other people a lot better They always treat the people closest to them the worst Because it's very easy to get rid of them It's very easy to insult them and put them down Because you no longer have any use to them Which is why they feel like it's so easy to replace you Because now they're so familiar to you That it feels normal and usual They're comfortable with you They don't regard or treat you as more important than other people They can replace you just like anyone else Because they see you as something temporary They're not planning to keep you around for a long time At some point they know they're going to get bored of you Or that you're going to figure them out And that is why they don't put much effort into you Because they already know you're not going to be around for a long time And they don't feel any need to change that themselves So they have to find someone else They can't help themselves They have a compulsive desire for excitement and adventure They need constant attention They need to have something new They need constant praise and admiration Because they enjoy the effect that it has on them It makes them feel better about themselves So they have to keep trying to find something newer and better Which means that whoever is with them Is destined to be discarded at some point Which is why you should never think that you are anything significant to the narcissist You should never think that any of the times you shared Was something significant But this is something that is very difficult for most people to comprehend Because you might think that after everything you did for them After everything you put up with How could it not mean anything to the narcissist? But it didn't mean anything You were just a substitute for the next person To give them what they need To make them feel better about themselves To make them feel alive They need to have different people to make them feel like they exist The only reason why they will stay with someone for a long period of time Is because that person is still giving them what they need And they are willing to put up with the abuse But even when they are with one person They will have other relationships on the side They cannot survive without having multiple relationships at the same time Because they are addicted to people They are addicted to the attention they get from them So you were never anything significant to the narcissist You were never the only one in their lives It was never something exclusive It wasn't restricted to you and only you You shared them with other people the entire time They never belonged to you They were never yours You never had anything So you never lost anything You were just wasting your time with the narcissist Because you were expected to have something meaningful You thought you had something that was worth investing in But you were just wasting your time Because you were just a substitute for someone else If someone is really committed to you They are going to be careful not to inconvenience or harm you They are going to be considerate of you But narcissists are very immature They don't want to talk about anything deep or meaningful They don't want to solve problems If someone really cares about you They are not going to let something get blown out of proportion They are not going to let it go too far But the narcissist will And then they will come back like nothing happened But if you were so significant and important to them And they really valued you They wouldn't let it go that far And that is how you should know That you're not significant to the narcissist You're not important You may think that you've worked hard in yourself And you should be worthy of appreciation and respect But for the narcissist This isn't their first rodeo They've been around the block a few times They've experienced this before They've seen people rise and fall So you're just another brick in the wall You're not special to the narcissist You don't really mean anything to them But you could be special to someone else You could mean something to someone else Someone who recognizes what it took for you To become who you are Someone who understands your worth Because a person like that Is not going to mistreat you And they're not going to let you go so easily Thank you for watching I hope this video resonate with you Please like, comment, share and subscribe Thank you for watching My PayPal link is in the video description Coach the Inquiries You can email me at coach.net.com Thank you for watching And I'll talk to you soon


If a grass fire starts and you live next to grassland, walk two streets back from the fire. If you live two or three streets away, it's safer to stay where you are. Grass fires are unlikely to spread into built-up areas. To find out more, go to emergency.vic.gov.au


a sut y gallwn yn gyntaf yn fyddion. Mae'r oedden nhw'n gweld fy modd o'r ymddych chi. Mae'r oedden yn gyntaf yn gyntaf, yr ymddych chi'n fyddio bod nhw'n gwybod ymddych chi'n gwybod a'u bod yn effaith efallai o'r gwaith i gyfnodol i gyd-feydd rydyn ni'n agedeit, ac yn gweld gyda'r defnydd. Mae'r oedden nhw'n gwybod i'n gwybod i gyd-feydd i gyd-feydd yr ymddych chi'n gwybod. Rydyn ni'n gobeithio'r llunyddau, ond ganddo'r fforddi, oherwydd, ond, oherwydd, y ideaf yng Nghymru yma o'r prif o'r ffordd am y cyfnodd. Rwy'n gwybod hynny'n mynd i'r ffodd cymorth o'r ffordd o'r Llywodraeth, o'r Ion Llywodraeth, o'r Llywodraeth. Yn cael ei ddweud, mae'r cynnwys yn dweud yn gweithio. Efallai, hefyd, mae'r definitiad Alunau a'r definitiad Pead i'r Llywodraeth, felly y morhau rwy'n gweld mynd i, ond yw'n neud yn fawr o'r sgol o'r ddafyn i'n holl yn dweud. Felly mae'n gwrdd gennych i'r ddafyn o'r ddafyn i'n ddafyn i'n holl yn dweud. Rwy'n rhai bod ydych chi'n bryd yn fawr o'r hyffordd. Mae ydych chi'n ddweud o'r ddarlodd hynny o'r drwng o'r llwyth o'r llwyth o'r ddarlodd o'r drwng o'r rhaf, iawn ar ddefnyddio cewddio'r dwdoedd. Mae'r pwysig yn deilleg. Roedd ateb yn hyn, ond rydyn ni'n gwybod fan hynny. Roedd yna'n busgyn i meddwl wath chi'n meddwl i adegol a achos i chyffrwyr a phobl agaf ymddangos i melodig yma? Roedd yna didentified gyda'n sylfaen o dd wickedid i chi, rydyn ni'n cael ei chael hyn yn ystod i ni'n dod. Rydyn ni'n gwneud i mi nad eich yw? Roedd, Fyndedyn I'm known as Seferthornig pretty much all I've ever wear. I've got since the funeroses, since the funeroses are related to their meaties, where I define as functions of question, imposter syndrome and lots of other things that I don't wish to label myself as unnecessary. I've been a back-end developer in Truthless since 2009 and stopped specialising in winter 2011. Last week, Burnout was in 2016 and my last major case of imposter syndrome was yesterday when I was updating this talk. So, why am I here? Well, I want to help promote better-looking health. It's too important. I dislike seeing people suffer and leave our community and by that, I don't just mean the local community, I mean developments or I mean the technical community as well. I came up with these tips and tricks when I was last suffering from burnout and I wanted to share them with people because there was too much imposter syndrome in our community. Please note I am not a medical professional. Your mileage may well vary with these tips and tricks. And also some of them are mutually exclusive and by that, I mean they may well help your burnout but they will not necessarily help your imposter syndrome or vice-versa. So what can I help you with and what can't I help you with? Well, these tips and tricks are designed to kick-start recovery of burnout and hopefully prevention of future relapses. I say recovery in air quotes because I don't believe you can fully cure imposter syndrome or burners. I believe you can only recover and prevent further relapses because if you're prone to burning out or throwing yourself into something or prone to feeling of your imposter, you're not necessarily going to ever change like that. It may well be part of your nature. What I can't help you with, well, can't solve all your problems. Some of them maybe be rooted in something that you'd experienced before and need professional help with rather than talking therapies rather than necessarily something that I can help show. You've also got to want to help yourself in the same way as a recovering alcoholic has to be wanted, has to want to help themselves or a recovering addict to some description. Some of this advice will not work for you because it requires you to be in a good place. For example, if I avoid a relapse, I talk about wanting to go to events. If you don't want to leave your house, that's not going to necessarily work for you. So what did I try? I tried therapy, chemotherapy, cognitive behavioural therapy, etc. I tried putting myself out there, which in most cases inspired me to speak. I met my driven heroes and mentors. I tried meditation. I tried buying stuff and other first world problems. Other first world start solutions rather. I tried to stick my head in the sand. I found a new hobby. I tried spelling up all the words again. What works, well? Therapy, it helps, helps both of them. Putting myself out there has helped with my burnout, but it wasn't so great for my hospital syndrome. Meeting my driven heroes and realizing they're human, that helped massively with my hospital syndrome. My new hobby helped with my burnout, but getting a hospital syndrome for that. And attending events such as through and for come, that's brilliant for me to burn out. I guess I don't think you'd buy it all, but it's not great for my hospital syndrome. I tried changing job. My contract ended and wasn't renewed. So I took advantage of being able to look around and try other things within technology. I'm not suggesting that you quit your job, but if the opportunity arises, take a look around and see what else is potentially available within something that you feel that you can provide for yourself and your family. And I explored me more about that a day. I also looked at rest days and mental health days that helped massively with my burnout. What didn't work? Well, meditation didn't work for me. Your knowledge may vary, but it didn't work for me. Buying stuff and sticking your head in the sand, I would hope it's self-explanatory while those didn't work, but they didn't. Everybody comes about what worked. Therapies worked really, really well. I tried multiple times, including cognitive behavioural therapy, more traditional counselling. And I also found that sort of... I also found that therapy, which has worked particularly well for me, isn't a case of like you're on, staying on the counter show, and going to end up doing all sorts of things for entertainment. The idea behind it is similar to that of cognitive behavioural therapy in terms of it's trying to be wide and brave to think differently about certain problems. But the idea behind the treatment therapy is you throw away the unwanted thoughts or the negative thoughts that you're experiencing. And therefore, if he's different to other talking therapies, it's privately funded. This is the biggest problem with therapy. I'm lucky enough to be able to report it. Other talking therapy may be available on the NHS when you are or whatever health service you have in other countries. But they do tend to be fairly limited and it tends to be self-funded. In allowing me to see the work for the trees or work out with the problems I needed to resolve first, because certainly when I started this in the midst of... towards the end of my last burnout, I was in a position where I wasn't even sure what the problems were. It is very slow going. I'm three and a half years in as of this month. I'm still going, but I'm nearly there. Hopefully by the summer I'll be down to a sort of maintenance. A level of attending therapy sessions. What else have worked? Well, put myself out there who's not that speaking. That worked really well for me, but it's probably the most difficult to do for the impossible scenario perspective. It's had more than my burner because it's allowed to focus on different things. Even though there we didn't save a figure of influence in this case for them. I've attended events like Cripple College of Dublin. I've spoken of local user groups. We are very excited. Cripple camps through the Bureau, Cripple College of Down, and regional groups as well. I've got future plans with this as well. I'm not planning on stopping. I plan on showing more camps, potentially in Stash or French, which has people of reasonable standard. I was intending to submit to Cripple Cants Day this year, but due to corona I'm postponing that for next year. Future plans include presenting today via Zoom. I find that putting myself out there really, really helps that I have to do it for the discomfort for it to work with my burner. So the idea is I have to try and present to a bigger audience. That helps me. I think that is just the kind of way that I am. It may not help for you.  I found a new hobby. I decided to go into the weightlifting and general fitness, because that was also going to help my health problems. I started weightlifting April last year. I went from struggling to lift a 20kg barbell to regularly deadlifting under 100kg, aiming to lift twice my body weight by the end of the year. It helped my CF as well as my mental health. My lung function was around 38%. It is now topping out at about 55%, which is great, because lung function, is going the other way. I thought that would go down, because it is unusual. It is talking to love to cheer. But it wasn't easy. Burnout has kind of happened. It has passed a syndrome and definitely happened. It is also part of searching for fitness. I also tried running around part running during last year, just before my 36th birthday. It isn't for me though. I mentioned this as much, because it can show that you do have to try, you do have to search to find the right thing for you. It is also going to non-drupal eras. There are a lot of people with this fibrosis in the delivery, but it is giving me non-drupal eras, which has helped me because it has shown that other people in other spheres have passed a syndrome as well. I also have many more ideas to do with burnout. Rest days or mental health days, it was a concept introduced to me through fitness. Training schedules often include one or two day breaks. The idea is you have lighter rest days or full rest days. I thought I had difficulty going through my mental health as well. So I had light and full rest days. A light and mental health rest days might be one where I just don't do any work, but continue sort of a normal day. A full rest day I would either just stay in bed all day or watch Netflix all day or sit on the sofa and do very, very little. It was validated to me at least by Boris because that really, really helped him. It's a similar kind of thing of what I do mainly when I'm taking it quite as long at home. Meeting when I end home with a group of heroes, people say don't meet the heroes. I say different because working with people such as these, we've got Stella, Mark, and Nick and myself, working with people such as these helped me because I can learn from them. I can see how to work with support teams. I can see to work differently in terms of back-end development. I can see how to work with front-end teams. I can learn things from that. I saw that they were human and I saw that they had it too. You know, they frequent family houses. Particularly Mark. Mark has told me more about front-end than I need to know, and I did want to know that. He inspired me to contribute to CORE and at last check I got 7.8 million to CORE, mostly thanks to him. Rebuilt my confidence massively when he asked me to help mentor him in back-end work. He frequently referred to my texts after me, helping me to realise that I do know things which helped certainly to the right of the hospital syndrome that says I can't do this, nobody will listen. I've seen people go about and implement things that I've suggested and they were non-drillful heroes. Most of the staff, they hold them. They're still heroes to me. I can't meet these people in real life because part of the people will say they're very easily. It does have an effect on their mental health but thankfully it hasn't had an effect to what might go back to the hospital side of things. People like Ben Murch, Sophie Grace Holmes, Joshua Jones they may not be anything to any more than a title that says progress as well but it shows that it helps show the other side of me that I've developed exists and it's how many people burn out giving new techniques giving new ideas to try. As I've said, it isn't about showing people showing you guys what you know, people are very inspirational. It shows that there are other people out there they have a hospital syndrome they have tips and tricks to use the ideas, some of the ideas surrounding rest steps have taken Ben validated by Ystyn's podcast with Joshua and also Sophie Grace Holmes again through podcasts and videos on Instagram the idea of rest and the idea of training schedules etc. It helps validate that my tips work for me because I've taken the same tips same ideas that I've used to beat my burn out in a hospital syndrome with Drouville the development that they've worked in the Fitzwil a bit of self abomination that hopefully it will show that where they're worth they can work for you. In the same way as I've highlighted one particular person I've worked with and asked, this is Ben he's a Thor of Collegory so he was a lot more of a Thor of Collegory had a lot of hands in my PT he has a system of roses and he's a constant source of inspiration to myself and others The Thor of Collegory came about with something called Thor's Day where he dresses up as Thor and does various parts of his treatment dress as Thor to try to encourage young children to see how he does that and also not just as if I wrote it he's won a multiple awards current meetings that I'd really like to and he's encouraging me with many things in general and fitness which is helping everything for me he's also helping me and others too because since he's given me I've been able to pass on to others and again seeing them implement it helps me think, oh maybe I've done something about this lifting arc So moving on to what didn't work well meditation didn't work for me motorists etc my memory just doesn't seem to work that way I thought more about how do I fix why I'm 71 where that's at the trouble that makes please don't think I'm against meditation and mindfulness I'm not, I have simple work for an awful lot of people it's just my mind is not quite enough or it's not in the right place maybe at the moment it was just that I tried mindfulness on the other I'm going to give this a try during the social distancing that I'm certainly going to have to do over the next few weeks I would hope that buying stuff is self-explanatory a swipe of work but just in case it's an optimal fear of death from needing to work more to work out how I was going to pay that death back it's very widely discredited there's a method of coping but if it works for you then great, I just don't think it should be you know, try ignoring it please just don't it should be fair, it's mandatory but ignoring it won't make matters worse it could lead to potentially other mental health difficulties such as depression foreign someone you can talk to it doesn't necessarily have to be a therapist it doesn't have to be professional health it could be a mentor of work it could be a friend it could be a family member that finds somebody that you think is important ignoring it is not a solution for anything things that have worked for other people that they've suggested to me or I've not mentioned for very specific reasons mindfulness on the other hand I'm going to give this a try because I'm going to give you a video out there showing the other views of mindfulness on the other showing the other views of mindfulness on the other it's just something I've not yet got the chance to try I've just gone to dress classes I've just gone to dress classes at my local gym things have gotten away improving your sleep and sleep hygiene improving your sleep and sleep hygiene now I haven't mentioned sleep too much most of me because for me at least when I have problems getting away and I don't have a sleep hygiene because of that I do keep waking up etc but yeah certainly it's known that sleep and I've experienced it as well good night sleep because I've walked a lot of problems things can look awful up there up to a good night sleep so better sleep hygiene regular bedtime regular get up time sleep until midday one o'clock when you wake up at 9 10pm at 9 should help a lot dietary adjustments and having a more healthy diet cutting out large amounts of additional sugar etc my diet is very specific for myocystic fibrosis therefore I go to Tames to mention this too much however I have started eating eggs eating more whole foods etc I wouldn't say I've gone down sort of a cleaning eating route I still have a good briar I'm serving a lot more than the sort of the plant that is rude as of yet although I am cutting out as much meat from a pre-inspected and I feel a lot better for it but I still have 4000 calories a day that is slowly decreasing and my health improves but 4000 calories a day is not a particularly sustainable diet to most of the people talking to someone that is into their best now I've not really tried this in as much as I've got my therapist at the moment but I am looking around to following the people that I've been talking to in a different manner talking to my friends more etc so hopefully as things will go on and it's time yet to be here for example I will have a support method because it's not necessarily a therapist and seeing the therapist occasionally comes on I've never really been able to completely escape support through therapy mostly because of my physical health problems I'm always looking for other ideas partly to be able to to help spread the word for more people but also because it may help me it may help me further I may find that silver bullet whilst it may not cure my plastic syndrome or it may not stop me burning out together may adjust my routine so much that they become mine and nickels so the way to go from here what ideas can we have in this conversation well I want to produce ideas in our community and develop a discussion around what I've spoken to people and through the association about what we can do as a community what they can do in terms of promotion to try to help with mental health signs things and those are hopefully ongoing I've created a Slack channel mental health because I can't do this alone but also I am only one person I can only have certain ideas so I've created it for a development area for mental health tips and tricks that are able to process improvements actually there is also a wellbeing channel on there which may be more appropriate as all of yourself obviously a volunteer is in any channel Slack or known volunteers are not necessarily professional lawyers or professional culpers and medical trained personnel so it will be a volunteer your line it will vary but I'm hoping that if we can have some sort of central location we can discuss these things it will be great I'm looking at putting pages up from Rhymstod Twrwgol and have ideas for the locations that you can go to in the state of the UK or Irish people who have ideas for Ireland French people from France etc because just because the Samaritans for example exist in the UK doesn't mean that the French version is called Samaritans speaking of charities it is a WordPress charity but they are designing to focus on health within the community called WPNUP I'm looking at what they're doing and I'm going to reach out and say look with the community we have got everything like this can we help you guys can we look at helping the community in ways you have to work as a community there is also the open source mental health initiative or open source mental illness which are a charity that works towards helping those women in the open source realm so here are some links I am sharing this presentation on get up so you won't be able to get all your links don't worry too much if you can't call them down it's WPNUP.org the open source mental health initiative and there is also a post that's in the home.com on down in the last couple of days and I hope it won't come back because there was some very nice information on that about the trip for arriving hospital syndrome it's run by an American who has had one experience with the hospital from a professional level so it looked like it was a very open resource but as I said it came on down in the last few days so hopefully it comes back so we don't need to get to the trying at a mental stage in terms of what can help you as an individual so for example the experiment I mentioned that I have explored myself that was partly the idea of the kind of mental health and fitness what do I like trying there is new hobbies what did I like the most did I like drawing even though what do I like what do I like reading et cetera looking at this more for a hobby perspective rather than necessarily looking at it from a perspective of well I don't know how to read I don't know how to talk I don't know the pencil but that doesn't mean anything to that and I don't just mean there from a postures and growth perspective I need to ease the case that I have no or very little artistic talent from a drawing perspective my artistic talent lies elsewhere so on to the part that is probably a little bit more difficult to do over Zoom questions, suggestions I'm not too sure how we can do this but we do sort of have technically 15 minutes so I've been trying to stimulate some discussion over this I'm just going to finish the last couple of slides and then give it a try so contact if you want to contact me to look at all of this and speak about it further keep the conversation going here's the details for me again presentation will be shared online so don't worry too much if you can't get these things down I'm relatively online if you search you can also search today or just ignore the literary talent or the Australian who is a co-reader terrorist and finally if you ever feel useless someone invented a protective cover for the hockey invented to retell ok so I'm going to suggest we now look at trying to do questions and answers I'm not sure how best to do that so yeah I think we just sort of go from here and say if people have questions and if John could repeat the name of the microphone and I will do the best to answer them any questions comments anything nope ok well in that case please feel free to to reach out and yeah but hopefully we can take this conversation further and we can look towards helping other peoples burn out all the possible syndrome