Live from San Jose, it's theCUBE, presenting Big Data Silicon Valley, brought to you by SiliconANGLE Media and its ecosystem partners. Welcome back to theCUBE. I'm Lisa Martin with George Gilbert. We are live at our event, Big Data SV in downtown San Jose, down the street, from the Strata Data Conference. We're joined by a new guest to theCUBE, Sastri Mallotti, the CTO of Foghorn. Sastri, welcome to theCUBE. Thank you, thank you, Lisa. So Foghorn, cool name. What do you guys do? Who are you? Tell us all that good stuff. Sure, we are a startup based in Silicon Valley right here in Mountain View. We started about three years ago, three plus years ago. We provide edge computing, intelligence software for edge computing or fog computing. That's how our company name got started, is Foghorn, particularly for IoT industrial sector. All of the industrial guys, whether it's transportation, manufacturing, island gas, smart city, smart buildings, any of those different sectors, they use our software to predict failure conditions in real time, or do condition monitoring, or predictive maintenance, any of those use cases, and successfully save a lot of money. Obviously in the process, we get paid for what we do. So Sastri, GE populizes this concept of IIOT and the analytics and the high, sort of the new business outcomes you could build on it, like power by the hour instead of selling a jet engine. That's right. But there's, and actually Wikibon and David Fleur did some pioneering research on how we're going to have to do a lot of the analytics on the edge for latency and bandwidth. What's the foghorn secret sauce that others would have difficulty with on the edge analytics? Okay, that's a great question. Before I directly answer the question, if you don't mind, I'll actually even describe why that's even important to do that. So a lot of these industrial customers, if you look at it, we work with a lot of them, the amount of data that's produced from all of these different machines is terabytes to petabytes of data. It's real, and it's not just the traditional digital sensors, but the video, audio, acoustic sensors out there, the amount of data is humongous. It's not even practical to send all of that to a cloud environment and do data processing for many reasons. One is obviously the connectivity, bandwidth issues and all of that, but the two most important things are cybersecurity. None of these customers actually want to connect these highly expensive machines to the internet. That's one. The second is the lack of real time decision making. What they want to know when there is a problem, they want to know before it's too late. We want to notify them here is a problem that's occurring, so they have a chance to go fix it and optimize their asset that is in question. Now, existing solutions do not work in this constrained environment. That's why Foghorn had to invent that solution. And tell us actually just to be specific how constrained an environment you can operate in. We could run in about less than 100 to 150 megabytes of memory, single core to dual core of CPU, whether it's an ARM processor, an x86 Intel-based processor, almost literally no storage because we're a real time processing engine. Optionally, you could have some storage if you wanted to store some of the results locally there, but that's the kind of environment we're talking about. Now, when I say 100 megabytes of memory, 150 megabytes, it's like a quarter of a Raspberry Pi, and even in that environment, we have customers that run a dozens of machine learning models, if you're not talking about. Like an ensemble. Like an anomaly detection or a regression or a random forest or a clustering or a k-means, some of those. Now, if you get into more deep learning models, like image processing and neural nets and all of that, obviously need a little bit more memory, but what we have shown, we could still run one of our largest smart city buildings customer elevator company, runs in a Raspberry Pi on millions of elevators, right? A dozens of machine learning algorithms on top of that, right? So that's the kind of size we were talking about. Let me just follow up with one question on the other thing you said. Besides the, we have to do the low latency locally. You said a lot of customers don't want to connect these brown field, I guess. Operations technology machines to the internet and physically, I mean, there was physical separation for security. That's right. So it's like security. Bill Joy used to say security by obscurity here. Absolutely. It's security by like. Physical separation. Yeah, absolutely. Physical separation. Tell me about, I was actually coming from, if you don't mind saying, last week I was in Saudi Arabia, one of the island gas plants where we deployed our software. You have to go to five levels of security even to get to that. Multi-billion dollar plant and refining the gas and all of that. Completely offline. No connectivity to the internet. And we installed in their existing small box, our software, connected to their live video cameras that are actually measuring the stuff, doing the processing and detecting the specific conditions they were looking for. That's my question, which was if they want to be monitoring. So there's like one low level, really low hardware low level is the sensor feeds. But you could actually have a richer feed, which is video and audio. That's right. But is, how much of that then, are you doing the sort of inferencing locally or even retraining? And I assume that since it's not the OT device and it's something that's looking at it, you might be more able to send it back up the cloud if you needed to do retraining. That's exactly right. So the way the model works is particularly for image processing because you need, it's a more complex process to train and create a model. You could create a model offline, like in a GPU box, an FPGA box and whatnot. Import and bring the model back into this small little device that is running in the plant. And now the live video data is coming in, the model is inferencing the specific thing. Now there are two ways to update and revise the model. Incremental revision of the model, if you could do that if you want, or if you can send the results to a central location, not internet, they do have a local in this example, for example, a PiDB, an OSSF PiDB, or some other local servers out there, where you have an opportunity to gather the results from each of these different locations and then consolidate and retrain the model, pull the model back again. Okay, the one part that I didn't follow completely is, Yeah. If the model is running ultimately on the device. That's correct. And perhaps not even on a CPU, but a programmable logic controller. It could. Programmable-ized controller also typically have some notion of a CPU there as well. These days, most of the PLCs, programmable-ized controllers, have either an ARM-based processor or an X86-based processor. We can run on either one of those too. So, okay, assume you've got the model deployed down there for the local inferencing. Now, some retraining is going to go on in the cloud where you're pulling in a richer perspective from many different devices. That's correct. How does that model get back out to the device if it doesn't have the connectivity between the device and the cloud? Right, so if there is strictly no connectivity, so what happens is once the model is regenerated or retrained, they put a model in the USP stack, the low-tech, right? USP stack, bring it to the PLC device and upload the model. Oh, so this is sort of how we destroyed the Iranian centrifuges. That's exactly right. Exactly right. But, you know, some other environments, even though it's not connectivity to the cloud environment, per se, but the devices have the ability to connect to the cloud optionally. They say, look, I'm the device that's coming up. Do you have an updated model for me? Then it can pull the model. So, in some other environment, it's super strict where there's absolutely no way to connect this device. You put it in the USP stack and bring the model back here. Other environments, device can query the cloud, but not cloud cannot connect to the device. This is a very popular model these days because, in other words, imagine this. An elevator is sitting in a building. Somebody from the cloud cannot reach the elevator, but an elevator can reach the cloud when it wants to. Sort of like a jet engine. You don't want the cloud to reach the jet engine. That's exactly right. Jet engine can reach the cloud if it wants to when it wants to, but a cloud cannot reach the jet engine. That's how we can pull the model. Okay. So, Sastra, as a CTO, you meet with customers often you mentioned. You were in Saudi Arabia last week. I'd love to understand how you're leveraging, engaging with customers to really help drive the development of Falcorn in terms of being differentiated in the market. What are those kind of bi-directional, symbiotic customer relationships? Like, how are they helping Falcorn? Right, that's actually a great question. We learn a lot from customers because we started a long time ago. We did an initial version of the product as we begin to talk to the customers, particularly that's part of my job, where I go talk to many of these customers to give us feedback. Well, my problem is really that I can't even do, I can't even give you connectivity to the cloud to update the model. I can't even give you sample data. How do you do that modeling, right? And sometimes they say, you know what? We are not technical people. Help us express the problem, the outcome, give me tools that help me express that outcome.  How we distinguish ourselves in this process from the traditional cloud-based vendors or traditional data science and data analytics companies is that they think in terms of computer scientists, computer programmers, and expressions. We think in terms of industrial operators. What can they express? What do they know? They don't really necessarily care about. When you tell them, I've got an anomaly detection data science machine learning algorithm. They're going to look at you like, what are you talking about? I don't understand what you're talking about, right? You need to tell them, look, this machine is failing. What are the conditions in which the machine is failing? How do you express that? And then we translate that requirement, or that, into the underlying models, underlying well expressions, values, or CEP expression language. So we learned a ton from user interface, capabilities, latency issues, connectivity issues, different protocols, and number of things that we learned from the customers. So I'm curious with like some, more of the big data vendors are recognizing, you know, data in motion and data coming from devices. That's right. Some, like the Hortonworks data flow, NIFI has a mini file component written in C++, really low resource footprint. But I assume that that's really just a transport. It's almost like a collector, and that they don't, it doesn't have the analytics built in. It's exactly right. NIFI has the transport, it has the real time transport capability for sure. What it does not have is this notion of the CEP concept. How do you combine all of the streams and everything is a time series data for us, right, from the devices, whether it's coming from a device or whether it's coming from another static source out there. How do you express a pattern recognition, pattern definition across these streams? That's where our CEP comes in a picture. A lot of these seemingly similar software capabilities that people talk about don't quite exactly have either the streaming capability or the CEP capability or the real time or the low footprint. What we have is a combination of all of that. And you talked about everything's time series to you. Is there a need to have sort of a, an equivalent time series database up in some central location so that when you subset, when you determine what relevant subset of data to move up to the cloud or on-prem central location, does it need to be the same database? No, it doesn't need to be the same database. It's optional. In fact, we do ship a local time series database at the edge itself. If you have a little bit of a local storage, you can down sample, take the results and store it locally and many customers actually do that. Some others, because they have their existing environment, they have some cloud storage, whether it's AWS, Microsoft, party, it doesn't matter what they use, we have connectors from our software to send these results into their existing environments. So you had also said something interesting about your sort of tool set as being optimized for operations technology. So this is really important because back when we had the net heads and the bell heads, it was a cultural clash and they had different technologies. They sure did, yeah. Tell us more about how selling to operations, not just selling, but supporting operations technology is different from IT sort of technology and where does that boundary sort of live? Right, so typical IT environment, you start with the boss, who is the decision maker, right? You work with them and then they approve the project and you go and execute that. In an industrial, in an OT environment, it doesn't quite work like that. Even if the boss says that, well, go ahead and go do this project, if the operator on the floor doesn't understand what you're talking about, because that person is in charge of operating that machine. It doesn't quite work like that. So you need to work bottom up as well to convincing them that you are indeed actually solving their pain point. So the way we start with, rather than trying to tell them what capabilities we have as a product or what we're trying to do, the first thing we ask is, what is their pain point? What's your problem? What is the problem you're trying to solve? Some customers say, well, I've got yield, a lot of scrap. Help me reduce my scrap. Help me operate my equipment better. Help me predict these failure conditions before it's too late. That's how the problem starts. Then we start inquiring them, okay, what kind of data do you have? What kind of sensors do you have? Typically, do you have information about under what circumstances you have seen failures versus not seen failures out there? So in the process of interrogation, we begin to understand how they might actually use our software. And then we tell them, well, he is using our software to predict that. Sorry, one, 30 more seconds on that. So the other thing is that, typically in an IT environment, because I came from that to have been in this business for 30 plus years, IT, OT, and all of that, where we don't right away talk about CEP or Expressions or analytics or machine, and we don't talk about it at that.  Is it analytics, is it machine learning, is it something else, and what is it? So that's kind of how we approach the problem. Of course, if sometimes you do surprisingly occasionally run into very technical people, for those people, we can right away talk about, hey, if it is analytics, it is machine learning, it is an expression of all of that. That's kind of how we approach it. One thing that's becoming clearer is, I think there's widespread recognition that this data intensive and low latency work to be done near the edge. That's right. But what goes on in the cloud is actually closer to simulation and high performance compute if you want to optimize a model. It's not just train it, but to maybe have something that's prescriptive that says, here's the actionable kind of information. As more of your data is sort of video and audio, how do you turn that into something where you can sort of simulate a model that tells you sort of the optimal answer? Right. So it's actually a good question for more experience, right? There are models that require a lot of data. For example, video, audio, that. There are some other models that do not require a lot of data for training. I'll give you an example of both customer use cases that we have. There's one customer in a manufacturing domain where they've been seeing a lot of finished goods failures. There's a lot of scrap. And the problem there was, hey, predict the failures, reduce my scrap, save the money, right? Because they've been seeing a lot of failures every single day, we did not need a lot of data to train and create a model to do that. So in fact, we just needed a one-hours worth of data. We created a model, put the thing that we have reduced completely eliminated their scrap, right? There are other kinds of models, other kinds of model of the video where we can't do that at the edge. So we require, for example, some video files, or simulated audio files. Take it to an offline model, create the model, and see whether it's accurately predicting what they're based on the real-time video coming in or not. So it's a mix of really what we're seeing between those two. Okay. Yeah. Lucestry, thank you so much for stopping by the cube and sharing what it is that you guys at Foghorn are doing, what you're hearing from customers, how you're working together with them to solve some of these pretty significant challenges. Absolutely, it's been a pleasure. I hope this was helpful and, yeah. Definitely, very educational. We want to thank you for watching the cube. I'm Lisa Martin with George Gilbert. We are live at our event, Big Data SB in downtown San Jose. Come stop by forager tasting room, hang out with us, learn as much as we are about all the layers of big data, digital transformation, and the opportunities. Stick around, we will be back after a short break.


Good afternoon everyone. Thank you for joining. We're going to go ahead and get started in just a minute. We're going to give everybody another minute or two to get logged on. So tight. We'll start in a minute. Thanks. Good afternoon everyone. Thank you for joining this important webinar today. I am Jose so guard deputy director for the office of nightlife at the mayor's office media and entertainment. For those of you who don't know, we are a liaison between the city and the nightlife industry of businesses workers performers and patrons. We work to support the nightlife community navigating city bureaucracy, improve quality of life and community relations, promote safety and harm reduction and elevate nightlife culture. If you have any questions or issues about matters related to this webinar or anything else related to nightlife, please always feel free to reach out to us at nightlife at media.nyc.gov, or you can follow us on socials at NYC nightlife. So today's webinar is part of a new series of courses we created called night school or nightlife industry training and education, which will be held both virtually and in person. This is a series to share resources and trainings for owners workers and patrons addressing how to best engage with city agencies while opening and operating nightlife businesses proactive harm reduction and bystander intervention, quality of life issues and much, much more. You can find out more information at nyc.gov slash night school. That's n i t e school and we will put that link in the chat. Today we're very happy to share with you some financial management resources and services from our partners at the Department of Consumer and Worker Protection and their Office of Financial Empowerment, as well as their partners at Bedford Stuyvesant Restoration Corporation. And we know that so many New Yorkers working and performing in nightlife that can be challenging to find time between your gigs and shifts to manage and make sense of your finances. And so today we're very excited to connect you with our partners who are providing free one on one professional counseling to help you do just that. And in just a moment they'll be sharing a presentation with an overview of tools and tips that can help you get started. Before I introduce my colleagues to conduct the training, just a few quick housekeeping notes. First, please use the Q&A feature in zoom to let us know your questions throughout the meeting. And after the presentation, we will have some additional time for Q&A. There will also be some questions for you to answer in during the presentation, when we welcome you to use the chat feature for that as well. This is also being recorded and live streamed to Facebook and a recording will be available to share with your staff or other colleagues who would like to view the training in a later time so please share with anyone else you know who might be interested in this information. And again, you can always visit nyc.gov slash night school that's NITE school to find information on other scheduled trainings and webinars as part of this series. Without further ado, I'd like to introduce my colleague first from the city's Office of Financial Empowerment. Arielle Munchek-Eleman is a program officer of financial counseling and coaching there and will kick us off with some additional background before our colleague Erlene Green, the manager of financial counseling and coaching programs at Bed-Stuy Restoration, gives today's presentation. Thank you both so much for being with us today. Arielle, take it away. Thank you so much Jose. I'm really excited to be here and excited to see so many attendees. One of the reasons that I'm really excited about this webinar is I actually was a nightlife worker myself before I joined the Office of Financial Empowerment for several years I was a drag performer and a barista. And unfortunately I wasn't in New York City so I didn't have access to some of the really awesome free public resources that exist here, which we'll learn about in just a minute. But yeah, if you have any questions about financial empowerment services in the city, feel free to put them in the Q&A. And I'm really excited to turn it over to Erlene Green, who as Jose mentioned is a manager of financial counseling programs at Bedford-Sivison Restoration Corporation and actually a financial counselor herself as well. So Erlene, I'm going to share my screen here with the presentation and turn it over to you. Wonderful, wonderful. Thank you so much, Arielle. And welcome everyone to today's webinar on financial management. I'm excited to be with you today so let us just get started but let me just give you a little more information about restoration. The first community development organization with a 56-year track record serving residents in the borough of Brooklyn and our Center for Personal Financial Health is like a one-stop multi-service hub where we promote economic self-sufficiency and financial stability for families while providing for them an opportunity to move upward in their finances. And, you know, we are like a pipeline sort of of services, including workforce readiness, training, job placement, benefits, screening and enrollment, and of course, financial counseling, employment and home ownership services. And we help our clients to develop assets, right. And so I'd like to say that our services help our customers along their financial journey where we meet clients where they are and we help them to navigate through the process of becoming or through the process of being financially vulnerable to financially coping to eventually being financially stable. So that's just a little bit about who we are in terms of restoration. So we can now go to that third slide and get started. And you will see that, you know, nightlife has a unique sort of circumstances or opportunities and challenges I should say as the slide says. So I'd like to just take a moment to ask you to respond to the questions on the slide. Ariel talked about his experience and now I'd like you to kind of just share a little bit about your experience in terms of some of the challenges that you may have faced. And also, please share with us any of the, any financial topics that you feel would be helpful to you moving forward. Okay, you could type that in the chat. Thank you. So, while we talk about while you may be putting some information in the chat about some of the challenges that you face as a nightlife worker. I just want to also talk about the fact that there are some chat opportunities also that exist in nightlife I was just reading some information from the mayor's office of entertainment and media, and they did an economic impact on nightlife. And it revealed that nightlife supports over 299,000 jobs, resulting in 13.1 billion dollars in employee compensation so that was good news for me, because clearly, nightlife workers are growing faster, or if not as fast as the rest of the economy in New York City so you know there's the data to support that. And in doing that we know that such as because it's a growing economy that we can dedicate more time and support to manage this industry and help nightlife workers to overcome some of those challenges. So, let us go to slide five. So, this is what we are going to cover today, understanding your relationship with money financial management strategies how to make the most out of your asset, how to minimize your liabilities. And then you will also learn how to access access the confidential free one on one financial counseling session at the financial empowerment centers. Okay. So personal financial management what it is. Okay, so personal financial management. I think we are on the slide before this one, Ariel. Yes, there you go. Thank you. So personal financial management, as you see is the process of planning and budgeting for how your money is saved or spent. It's managing your personal finances in a way that will help us to set goals to save save for various reasons like retirement. And it is a crucial, it's critical and our ability to effectively manage exactly what it's coming in in terms of our income and what is going out in terms of our expenses, and how to save and how to invest as well. And so this is especially important when we have variable income, as many of us do in nightlife work right because if you are a nightlife business owner without sound sound financial management you're likely to make the kind of mistakes that can cost you your business. Or that can cause you to even mismanage your finances resulting in minimal and not maximum profits. It can also cause you to mismanage your finances and cause you to not be able to pay your employees on time I have been in situations like that as well. But if you are an employee practicing financial management principles enables you to take control of your finances to save and to plan for the future. So we're going to go to the next slide which depicts three major components of personal financial management and the first one is a budget. The second one is goals and the third one is savings. So when we talk about budgeting budgeting is simply a spending plan to help us to balance our income, our expenses, and our financial goals for a specific period of time. And so a budget will help us to track our spending and ensure that we are living within our means. It's a way of helping us to keep control of our money so that our money doesn't control us. When we look at budgeting, we look at, again, the need to total up all of our income right, many of us have income from various sources so you total up all of your income from various sources, whether it's from a full time job part time job, whether it's portfolio income or interest income, or even tips that you receive right, you total all of those up for the month, and then you minus all of your expenses, your fixed expenses, your variable expenses, and even periodic expenses and so what's left over after you deduct your expenses total income is known as cash flow right so in the end you can have a negative cash flow which means that your expenses exceeds your income, or you can have a positive cash flow which means that your income exceeds your expenses, and you have some money left over to do some other things right. But depending on how much you have left over, you know you could create a plan to, you know, to save or to invest or to make some kind of major purchase right, but I want to also mention that when you do your budget, you can also find yourself at a place where you are breaking even right so to break even means that you are spending everything that you actually earn, and that there is nothing left over. And so you just have enough money to meet your expenses and this is really the epitome of what is known as living from paycheck to paycheck right, or living on an earn and spend treadmill and I don't know about you, but I have definitely been there and done that right and not looking forward to, to going back to that place. Okay, so that sums it up for the budget. So let's look at smart goals. When you create a viable budget, we, we give ourselves the best chance of being able to save and achieve our goals. So let's just look at the smart goal component and exactly what it is. So we know that a goal is a destination or something that we want or need, which will require us to take some kind of action to achieve it. And I, I always indicate when I tell that people always say that it is definitely different than a dream. People use dreams and goals, synonymously, but they are really different goals is an object of effort is something you have to work toward a dream images and thoughts and visions and you can have them but you don't have to put any effort toward achieving them right. So, when we talk about a goal we think about and we, we think about it providing goals providing a sense of direction, motivation, and in some kind of clear focus. And it helps us to kind of clarify our importance in terms of not an importance but our priorities, I should say, okay. And so when we think about smart goals, it is used to help us to set goals in a way that is viable right. So it's an acronym smart is an acronym, it means specific measurable attainable realistic and timely. And we incorporate these criterias to help us to focus our efforts and increase the chances of us achieving our goals and so I would what I want to do now is just really just give you some idea of what the the acronym stand for, because it's important when you sit down and write these smart goals. So the specific is when your goal is well defined. It's clear is on the ambiguous. For instance, when you sit down and write your goal, saying I want to spend my spring break at Daytona Beach versus I want to do something over the spring break right. The first one is more specific is clear. It tells you where you want to spend your spring break because knowing where you want to spend your spring break is going to inform all of your following decisions measurable the M is when your goal is capable of being measured it answers the question of how much do you need to accomplish this goal. For example, I need $120 for my share of the gas hotel and food for the week versus I'm going to need some money for the trip right. So you want your goal to be measurable. You want to understand how much is going to cost achievable. The goal has the ability to be achieved it's not something that is impossible right. So for instance, I'll say $15 a week from Thanksgiving until spring break or, or I should say versus, I'll win a weekly radio talk show contest to get the money to pay for the trip right. With that second one there are no guarantees it's a long shot right so because of that it may not be achievable, and it's unlikely that it will be achievable right so your goal has to be achievable. Realistic it has to be real to life it has to be relevant, and within your reach it has to be real practical, right. So for example, I plan to drive from New York to Miami, Florida. 20 hours using for drivers versus I want to drive from New York to Miami in eight hours right that is not possible realistically that is not possible right. Unless you're going in the airplane of course but it's going to take you a lot less time but I'm talking about driving specifically. And then we have time bound right which is a clearly defined timeline, including a start date and a target date. The purpose of the timeline is really to help you to create a period of duration, because a period of duration will give you a sense of urgency, so that you don't prolong your, your goals so that you start working toward it and the goals can be short term within six to one year it can be long term one to five years, and it can be five years right. And so, often individuals, or businesses will set themselves up for failure by setting general and unrealistic goals right goals that are vague and have no sense of direction so we want to avoid doing that. And so, you know, again the smart goals set us up for success by making our goals focus right it helps push us further in the direction that we want to go and it helps us to organize and to be able to reach our goals. And then we have savings right and this is another component or a principle of financial management and when it comes to saving there's no one thing that that we should, when it comes to saving there is one thing that we should acknowledge about ourselves and that is whether or not we are or we consider ourselves to be a saver or a spender. So that's a good question that you can ask yourself which one am I am I a saver, or am I a spender you can put that in the chat to are you are you a spender are you a saver. And knowing the answer to this question is important for several reasons. The first reason is because it gives us a good way to determine our relationship with money. The second is that it speaks to our attitudes in our actions regarding money, and it can also help us to change some bad habits and to replace them with with some good ones right. And so here's what we know about a spender. If you are a spender. You know, we we love money for the things that it can buy right. And we would also prefer to have something concrete intangible, like a car or trendy gadgets or technology over having something as abstract, such as savings right. And so, and we know that that savers are life savers really right. Because, you know, they, they are the ones that we borrow from from the most part right and savers create a fortune in the bank very quickly. But while still living a comfortable life. You know, and while still being on a tight budget. My brother is that way. And we used to call him a penny pincher we used to call him cheap but as I learned more about finances, I learned that he is frugal and he is responsible with his money. So, so, and, and you may very well be someone who is not a spender or saver but you can be both right. And because you're both you may have, or you may do a really great job balancing between the two right balancing between being a spender and a saver but the balance here is really for us to acknowledge where we fit in, and then determine if where we are is where we really want to be right because the question is, wherever I am, is it working for me or is it working against me is it moving me towards my goal towards my desires and plans for the future or is it moving me away from it right. Am I able to say for that house that pension, my children's education that vacation, my children education that vacation or even medical expenses right so I think for for nightlife workers. You may be faced with the challenge of inconsistent income and maybe lie more heavily on tips and other compensations but I, and that might be a reason why you may not put so much effort into savings but I want to encourage you to to really save savings a priority because it's not really how much you save from every paycheck that makes a difference it's really how consistent you are with saving right. And consistent helps us to develop good habits of savings therefore when you are able to save more, you will be more willing to do so because you have already developed the habit of saving and remember, you know, responsible spending and to live within our means and to live within our means in order to do that we must spend less and and then make and save as much as we can right. Okay, so we're going to go to the next slide slide seven and this kind of ties everything together as a way for us to really understand and control our finances right. So, here we have a list of things that we can do. And one of them I'm just going to put a little emphasis on some of them not all of them reduce monthly spending and really this is about budgeting right what we already talked about because in order to reduce our expenses. We need to track them and to determine what we can reduce and or eliminate from our budget, in order to put more money back into our budget, so that we can use that money for other priorities right. So, then we have paying off debt and this is also related to spending right because when we have little to no debt, we get to keep more of what we earn, then we'll be able to do more with our earnings so this is a really important step to take to help us to control our finances and then exploring ways to invest our money right. This is related more to setting smart goals right understanding our destination or or plans for the future and making our money work for us and not against us and so this is really about allowing our money to grow and to get a good return on our investment. It will allow us to to buy assets or financial products that will eventually appreciate and not depreciate right. And so I want to just take a minute to to to talk more about investing because it's we are in a time now. Where the idea of investing is is it's called it's it's closely correlated to our eye to what will will we like experiencing right now right, and that's inflation. So when we invest, we believe that we are buying an asset that will generate a return over a period of time so the main reason we invest is really to to grow our money. And of course with with investing there are no guarantees and most investors are better suited for the long term, rather than the short term and when I say the long term I mean for retirement or purchasing a home. Something like that right, but there are no guarantees. So why do people choose to invest in addition to not at the exclusion of but in addition to savings right. And so the answer it has to do with the increase really of the cost of living over a period of time and this is known as inflation right and. And again we are really feeling the sting of that these days. So while savings is is good for the short term they typically don't generate the return beyond the level of inflation right. In fact, they often lag inflation. Excuse me so that can mean that our money is not just standing still and in our savings but it can also be losing its purchasing power over a period of time and so the goal of every investor should be to outpace inflation so that our money can generate a solid real return. And that's the return after inflation right so while most investments again don't guarantee. Don't make any guarantees they generally offer the potential to beat inflation and that's what we're looking for the kind of investments that will be inflation so saving and investing really go hand in hand and they can. And they both have their place in our financial management plan. Okay. Another important aspect of financial management is understanding our relationship with money. And so this particular slide. Ariel you can move to the next slide please. Okay thank you. So when we took when we talk about relationship with money. We're looking at and this is interesting that I want to preface this slide by saying that when we think of relationship. We often think of interactions with people right. Our life experiences teaches us that relationships can be healthy. It can be unhealthy and it can be toxic right and and usually when we think of it in terms of that we think about it when we are in relationship with each other. However, we really consider whether our relationship with money can have the same effect or impact or could be the same right. So why is having a good relationship with money important. It's important because it's really a part of our overall well being right. Being financially well includes having a relationship with money that makes us feel satisfied and content and not stressed out all the time you know. I've read so many reports that said money is the leading cause of stress for adults right and an unhealthy relationship with money can cause to. It can result in depression, divorce, bankruptcy, and so on right so what we want to do is now let's just look at the difference between healthy and unhealthy relationship. So under unhealthy relationship we see an example of that would be when you spend more money when you when you when you're spending money because it but when you're spending money it makes you feel guilty. Sorry. So, when you spend money it makes you feel guilty. When you spend money. You spend more than what you make. And then you have to borrow money to take care of your responsibilities on healthy relationship is when you spend money carelessly without giving any thought to how you're spending your money. And without even planning to spend your money. It's when you're overwhelmed with credit card debt. And, and although that is the case you just keep digging and digging yourself deeper and deeper and dead.   Right. And so that's that that has to do with keeping up with the Joneses and we'll talk more about that later. So when we're talking about a healthy relationship with money. An example of that would be when you monitor your spending and have control over it. So can you go to the next slide please thank you. And then you use money to achieve your goals.   And that we are reasonably content with our relationship, because we feel in control of it, and we feel that it's doing for us, what we desire for it to do. And the good thing is that when we can learn to focus on changing unhealthy money habits or relationships with money. We can reinforce a healthy one as well with our money right. So, where we are today is not necessarily where we have to be tomorrow, we can make choices to be in a healthy relationship with our money, and to effectively manage our money and the way that will help us to plan the kind of future for ourselves that we want to have. Okay. Let's go to slide 11. And so we're going to talk about values and how that impacts our relationship with money. And so our upbringing and experiences and values are often what determines the kind of relationship that we're going to have with our money. And so a question that I like for you to answer in the chat is, what are your values when it comes to your finances, and how do your values factor into your decisions around money. These questions are really important. So I would really urge you to, to answer that if you can't do it now, then just think about it and see what you come up with right. So our relationship with money is formed in many ways right it's formed by our observations and the messages that we see and hear that is related to money. And according to a PBS report, children can understand basic concepts about money, starting as early as three years old and by seven definitely. And I can attest to this because I used to do investing and financial training with young people in junior high in public school, and they were able to get those concepts and understand them right. But our early experiences with money, such as witnessing arguing in the household about money or being defined by money. It can trigger a range of emotions such as anxiety, resentment or feelings of elitism right that we may carry with us through life. And those experiences and those emotions can help shape and they usually do help shape our relationship with money. And based on our values. Many can make us money can make us feel insecure. It can make us feel secure to right. And as we provide not only for ourselves but for our family, and even our joy can come from the sacrifices that we make with money and money can also, you know, make us feel insecure when we don't have it right. When we don't have the things that we want. It can affect us negatively and it may encourage some envy or maybe even corruption or maybe even some resentment toward those who may have a little more than we do. And it can also position us to be overly socially competitive right by feeling the need to really keep up with the Joneses. I had a pastor that and I love the way he put this when he talked about keeping up with the Joneses. And he says we spend money that we don't have to impress people that we don't even like right so given people a full sense of impression of ourselves, and it only serves to keep us deep in debt and causes us to, to keep less of what we earn right. So, let's let's be real about our money at least to ourselves and let's respond to what we have accordingly right. So, let's see. I think we can go to the next slide. Okay. So, when we have a healthy relationship with our money we are willing to be honest with ourselves and to fully understand our financial position right so some of us don't want to know what our financial position is really because we have a, a, a, although we have a good sense of it. We really don't want to look at the true reality of where we stand, because we may not be in such a good place right. Sometimes we ignore the financial numbers, which really tells us about our financial well being. And so it's important to know our numbers. And when we talk about knowing our numbers, we're talking about knowing your cash flow we talked about that your work, your FICO score, your credit utilization and your income to debt ratio right. And so, when we're talking about cash flow we can go to the next slide. This is simply, we're going to go. Yes, thank you. So we're going to go to, so we're talking about when we talk about cash flow. Again we're talking about money coming in and out, not only of our household but out of our business as well right for month to month. And whatever's left over is our net cash flow just like on the budget sheet right. So we look at what's coming in and we look at what's going on and we look at what's what's left over. And if you have a business the best way to keep track of your cash flow in your business is to run a cash flow report. And this report shows the cash you receive, and the cash paid out to show your, your, your businesses cash position at the end of every month. It's not going to be a need for a business not only to do this at the end of every month, but to keep track of your cash flow on a weekly basis or even on a daily basis to really stay on top of it and to track it to help you to make day to day decisions right. Next slide is our net worth. And this is the combination of what you own your assets and what you owe, which are your liabilities. Right. And so, in our financial counseling sessions, we use a balance sheet to help clients to determine their network. And so as you see in this graphic here it reflects an unbalanced scale. And in this case the person has more assets than liabilities which is where we should all strive to be right, we want to have more assets than our liability. So, our network is just the value of all of our assets minus the total of all of our liability so knowing this number again can help us to make financial decisions about our future and about our financial stability. Right. It's helpful to help us to stay on track. And I'll just talk about briefly just ways that we can increase our network on that work we can pay off our debt, our credit card debt. This is the best way to grow our assets right and increase our network. Because it will eliminate all of those high interest credit card debts and loans that we have right. Another way is to build an emergency fund. This is going to also help us to grow our assets and increase our network because creating an emergency fund is going to help us to stay financially afloat without having to rely on other money. And especially high interest accumulating high interest debt from credit cards or expensive personal loans when we have an emergency fund, we can go to that to that fund to pay for the emergency as opposed to putting it again on a high interest credit card. Or take out a personal loan for that. And we know the interest rate today on both are extremely high right so we want to stay away from that. And then another way is to consider maxing out maxing out our retirement contributions taking advantage of these accounts and keeping as much invested in our retirement as possible especially when your employee, your employer is contributing. You want to at least max out on your specific contributions and another way is paying off your student loans and invest as well okay paying off your student loans and get into whatever investment opportunities you can. The next one is your FICO score we know that this is an important number because it helped lenders to assess how well you manage your financial obligations and this number is determined by credit agencies based on the information from my credit report. And it's made up of five different components, 35% is based on our payment history. So that's why we want to make sure that we pay as much as we can, and always pay on time, 30% is our total amount that is old, and that is reflective of our debt to credit ratio and or credit utilization ratio. They're both the same but this is a reflection of how much credit we are using for example, if you have $2,000 and outstanding debt, but you have $10,000 and available credit line right. That's only a 20% debt to credit ratio or credit utilization ratio, you divide the, the, the 20 the 2000 by by 10,000, and you get the percentage right, and we will show you more about that toward the end but using. This in mind that when you are using your available credit, if you use more than 30% of your available credit limit, it is going to cause a reduction in your FICO score is going to decrease it right. So whenever you're using your credit, try to use less than 30% of it so that you can get the maximum FICO score right, and we know that our FICO score ranges from 300 to 850. And this is really an important number for our financial success right so if you have a very good to excellent score it can open up doors to loans at the best rates. You can get higher credit limits, you can it can open you up to drop opportunities it can give you access to additional credit for major purchases. And so your credit worthiness is an advantage to you and it can save you thousands and thousands of dollars in the end credit utilization ratio that's what we just spoke about. So again, your total credit utilization is the sum of all of your balances divided by the sum of your credit card limits, and you again want to keep it below 30% below 30% because anything above will negatively affected. Okay, so it's the 800 which is your balance in your 4000 that's your credit limit so you're using 20% and vice versa, you would be using for 40% in the other example, right. So, let's go on to the next slide. And here we talk about. Let's see, let's talk about ways to manage our assets ways to manage our assets. Okay, there was another slide in there and didn't see that one. Okay. So here's a deck to income ratio, which is the same as the credit utilization so that side didn't necessarily have to be in there so forgive me for that. So we're going to go to ways to manage assets and there are several ways. First of all, identify your assets know what your assets are. You know, keep them written in the book keep them on a spreadsheet know how to access access them immediately when you need to assign value to your assets what are your value your what are the what are the values. That your assets carry at this point and even look at an appreciation value as well. Ensure and or protect your assets you don't want anything to get in the way of, of you losing your assets or it not being protected so that's important that you do that. Understand your assets and your taxes right your liability and create a plan to leverage your assets so when we when we manage assets well. We can we can use it to accumulate more assets right which will bring us greater value and add to our growth portfolio right now. Expanding access assets, expanding assets is important right and we can expand it. For instance, if you are really wanting to put $20,000 to work investing in real estate right. You can put all of that in one basket but we've learned right how important it is to diversify right so we could instead use the $20,000 put 20% down on five different properties of the same value. And at the same time $10,000 profit per house but we can actually use again that $20,000 to leverage that money and purchase more assets right and so this is just something to think about when you are considering. Managing your assets not considering hopefully you will right but when you're considering creating a plan to manage your assets and again this is a way for you to leverage your assets use what you have and capitalize on it so that it can create more assets for you. Okay. And then the other slide is ways to manage your liabilities and these are just several ways that we already discussed, but minimizing our liabilities simply means increasing our numbers right making sure that our numbers are financial numbers are working for us. So, we want to increase our cash flow, our FICO score and network, and, and we want to be in a position to increase our assets so all around if we are managing our liabilities. Effectively it's going to be a win-win situation, and it's going to create the road of financial stability and wealth for us right. So the one point I do want to emphasize here is for, I would, I would encourage you to seek the advice and help from a financial counselor right we're here to support you and to help you to develop strategies for savings, increasing your FICO score, paying down your debt, addressing collection accounts and your student loan debt, and all of this is at no cost to you. We also have community partners who we can refer you to who also free of charge will support you in other ways such as our partnership with the foundation for financial planning where you can meet with a certified financial planner to discuss investment opportunities, retirement planning, insurance, taxes, estate planning, and etc. We also have a partnership with the New York legal assistant group and they help our clients to navigate consumer debt collection accounts and especially when it's in civil court. So, you know, speaking with the financial counselor can can really benefit you and we can definitely help support you to support you on your journey to financial freedom or financial stability whatever the goal is for you is different for everyone. So, I think that is it for us. I'm going to turn it over to Arielle at this point. Arielle, you can take it away. Thank you so much, Arlene. It was really fantastic. We have here just a few highlights of the various consumer protection and consumer support agencies that are available to you as residents of New York City. There is the statewide, the Department of Financial Services, there is the federal FTC, the Consumer Financial Protection Bureau has tons of resources that are extremely useful. And then of course we have the New York City Department of Consumer and Worker Protection where I work and restoration where Arlene works. One thing I want to highlight here is the Financial Empowerment Centers which Arlene mentioned a number of times. These centers offer free professional, confidential and one-on-one financial counseling to anyone who is 18 or older and lives or works in New York City. There's no income restrictions. There's no immigration status restrictions. We have services both in person and over the phone. We have over 30 sites right now operating across the city. So if you want that face-to-face interaction that can be so helpful, there's lots of options to do that. And collectively, over time, we've helped New Yorkers reduce their debt by, I actually believe this is almost at 100 million right now if it hasn't already passed that and build more than 11 million in savings. So as you can see, you can schedule an appointment by going to nyc.gov slash talkmoney and then you click the book and appointment button. Or if you're more comfortable scheduling via the phone, you can call 311 and schedule that way. Just say that you're looking for financial counseling. Thank you so much. We do have time for just a few minutes for questions now. One thing I do want to highlight is Erlene's contact information. Erlene is one of our most experienced managers and counselors. If you would like to meet with Erlene specifically, you can give her a call or email here. But because Erlene works for Bedford-Seifers in Restoration, you know, their services are only in Brooklyn. So we have, you know, other services across the city as well. So if you have any questions, you can feel free to put those in the Q&A. I answered one already. Erlene, I guess, do you have any regarding the question that was asked? Do you have any particular advice on managing a budget with multiple streams of income? I personally will say is, you know, obviously a budget with multiple streams of income or with variable income month to month is definitely more complex than a budget where, you know, if you have a salary job where you're getting a consistent amount every month, but it's still really helpful to create kind of a budget perhaps like with several ranges. So, you know, if you know that the there's like a lower limit to how much you might make in a month and upper limit based on say how many tips you get or how many gigs you have that month, you know, adjusting kind of having several different versions of that budget with the same items is one way that you can, you know, manage that kind of variability or multiple sources. Thanks so much, Ariel. I while we're waiting for any last questions to come in. I'm wondering if I can just ask you to read out the other answer you gave about the other resources your agencies providing just for the benefit of folks who might be viewing the webinar at a later time. Absolutely. So there was a question asked about tax preparation and about advocating for untimely pay or rather against untimely pay. So there are, I'm, I do not work in the Office of Labor Policy, but there are a number of laws that are New York City specific regarding, you know, what your, your pay schedule has to be how quickly you can be paid, or you have to be paid after particularly a freelance gig. I put the link in the Q&A but I'll put it in the chat as well. If you search, if you're watching this and you don't have the chat, you can just search DCWP, that's the Department of Consumer and Worker Protection and Labor, or DCWP and taxes, and those will bring you to the agency pages on those topics. And what I will say is that they're, the laws are very industry specific so depending on what kind of workplace you work at, the rules around timely pay around, you know, at what kind of threshold of freelance gig, certain kinds of protections kick in on will vary. So I definitely recommend looking at those web pages to figure out what applies for your specific, you know, employment situation. That's really terrific and because we are just about a month out from the tax filing deadline for those of you who are still looking to fire your taxes, please do check out DCWP's free resources for helping many New Yorkers prepare and file their taxes. You may be eligible so we really encourage you to check that out. Next thing I'll also add about taxes, the New York City specific free tax program, the income, there is an income cut off, unlike the financial empowerment centers. I believe that cut off is $56,000 a year. But if you make up to $73,000 a year. You will be eligible for the IRS's free tax prep program. That's kind of unlike these private tax prep quote unquote free tax prep applications that you know companies. I probably shouldn't say any specific names but certain companies advertises free that actually hit you with fees after you like enter all your information. So if you make up to $73,000 a year, there are fully free publicly available programs for you. You know, if only only use those other private companies if you make over that limit. That's really helpful advice thank you and we do encourage everyone to to check out both of those links and I'm seeing no additional questions in the q amp a and we're just over. One PM cut off time so I would like to thank both Ariel and early so very much for putting together this really helpful and informative webinar. And for everyone who has joined us today we really hope you've been able to take some helpful information away from this session, and that you will be able to take advantage of the resources and services from the financial centers as well as some of the other offerings by the Department of Consumer Worker Protection and their partners. So thank you very much, everyone for attending today. As always feel free to contact the Office of Nightlife via email or social media. That's nightlifeimedia.nyc.gov or at NYC nightlife gov on social media. We will leave the meeting up just for another couple of minutes for you to be able to grab any links or other items from the chat that you may need. So, thank you very much everyone and have a wonderful day. Thank you all take care. Bye bye.


Coach we know the Suns are talented but from the net's perspective What were the biggest most glaring issues that stood out in your performance the biggest biggest issue was the Suns? they You know they dominated us in every fashion they were They were unbelievable. We were obviously subpar and not up to our standards And when you have that kind of disparity you can you can lose by 30 on the road. So You know these games happen in the NBA. I don't There's two things. I don't love that. It's this early in the schedule and I don't love that we gave up, you know 140 points or whatever. Those are those are two two big issues for us and obviously We got to get it fixed We'll work on it tomorrow. See if we can bounce back as a defensive communication positioning What do you think part of it? I think part of it was they were really good They have a five-out system that does great ball movement great A great Distribution I think they had 37 assists they really kind of spun us around and something we got to look at But real credit to them Like I said, if we're subpar And they're an elite level that that's what happens in this league. You're you're you're gonna you're gonna lose by 30 especially on the road so You know, we'll take the responsibility, but I'd give them credit first Guys go down big early and then the bench really kind of gets you guys back into the the mixtures What would they start doing? Well that they'd really got that run for you defend them better The energy was better the engagement was better the You know understand what the game plan was was better every everything was better. So that was our only real stretch Now one thing we got to learn how to do when we're not making shots like tonight You get to learn how to get stops even when you're not making shots and we're missing and missing and they're coming at us and and It's tough when you're missing all the time, but we have to be more resilient than then again given up a hundred and thirty three points They scored on ten straight possessions What what is the problem that keeps recurring with the starting in it? Do you think yeah? No, we'll have to look at it. It's just not a strong First-quarter defensive team. We've we've given up 30 points. I don't know how many times in the first quarter It's just we've something we got to look at Had two rough games Is there anything you kind of point to that that he's going away?  Can you do it? How many how many games? How can you back it up? How can you back up for back, you know for good games and and he's had a couple games whereas? Not not playing like you usually plays, but he'll get it back He says these games happen in the NBA. Are you trying not to overreact over react because I think we've competed hard now No, no, we can you know, we're team. I like our team. I like our process Not gonna overreact to this to this. We've competed played a lot of close games They were on point we were far from on point and that that's kind of the story of the game So that includes any changes in the starting lineup as well Obviously you can beat it overall, but I don't I'm not not ready to go there after after one You know one bad game so going back where you send it the issue with the starters, you know after miss shots Not being locked in defensively to get those was it specifically with that with that starting unit Did you notice anything else there with the starting five? Yeah? I mean like I said, we miss shots and then we're not resilient enough to get stops even when when when You know when we miss shots that starts with starts with our transition defense wasn't good But you know, you see the stat.  We have to do better and we will A few games ago when you guys were down big that you used so much energy to tie it up and rally back that That's why you kind of struggled to hold the leaf from those that you get down to when then they have that big run Is that similar? We're just you use so much energy coming back that you might not I mean, that's a great You know that group that got us back, you know, we kept them and they still you know, they they stretched the lead again so You know, I don't think that was the case and that Overall, we did not have the juice that we needed the energy wasn't there but the energy When the energy is not there, you also have to execute your game play You got to rely on that and we we just so we weren't there physically mentally And those those are some bad combination A particular message that you delivered to them after the game You know, it's not up to our standards, you know, I forget to score forget the Sun's net standards our performance standards We weren't we weren't close and that's everybody that's a coach is to that's that's not just a player So, you know, you just got to accept it and and and fix it and and raise your standard standards


Hello, welcome to our talk on ML Ops for ground operations of the ISS. We are really excited to be here, especially excited that so many of you made it to this talk at such a late hour, aren't partying yet down in the exhibition hall. Before we start talking about ML Ops, we'll give you a brief introduction about ourselves and how we got here. I'll start. So my title nowadays is data scientist, but I haven't done any data scientisting in quite some time. Instead I've been a data engineer, data architect, something like that. So basically building all of what we are going to talk about as part of a larger team. I used to be a physicist, did a PhD there, got a lot into data analytics there, also like running bigger clusters and so on. I've been a data science consultant for some time after this. That's where I picked up wearing a jacket like this. I just don't look like that usually, but I'm a long time UNIX nerd, which really helped me out with this because I actually had no previous exposure to Kubernetes at all before. Obviously, I knew what it was, but had never really touched it. But without that 20 years of UNIX experience, that would have been a much harder journey. Hi, I'm Samo. Similar role as Christian, slightly different background. So my background is in life science, so I have a PhD in computer-aided drug design. So I spent quite some years working in pharma and after that in consulting before joining this company. Also have decades of experience with working with open source and Linux. And before embarking on this project, I had some experience with Kubernetes, but that only from convenience of the cloud. So this talk will be about, we will just give you a high-level overview of the use case, but we'll then mostly focus on our approach towards designing and building this MLOps platform and obviously on-prem without the cloud. We'll talk about major components we used and how we decided for them. We'll touch automation, resistance storage, networking, logging and monitoring, and then finally we'll give you some tips and tricks and some useful tools that we discovered during this journey. So what we won't be talking about is any Qflow details, running pipelines, ML jobs, things like that. We also won't go into any details about anomaly detection, root cause analysis, and other algorithms we're developing. And also fortunately, we don't have time to go into any details about International Space Station. So the work here is sponsored as part of this project called KISS, and the project is about AI for International Space Station. So what we're getting is we're getting telemetry data from International Space Station and we're trying to develop algorithms for anomaly detection, diagnostics, and finally also reconfiguration. So if something goes wrong, also the systems can then be reconfigured and amend the problem. They are four project partners. So two German universities, then Airbus and as just the AI. So the universities are focused on the models, while Airbus and us, we are mostly focused on the platform itself and also Airbus provides us with data. And just to give you a rough feeling, so there are like five, six people not working like full time on this platform. So still a brief info about International Space Station, so the data we are getting is actually from Columbus module, so this is Europe's first permanent outpost in orbit, and it's actually a space laboratory. So it weighs 10 tons, takes up to nine tons of payload and it's seven meters long, and you can now here see it also within the International Space Station, which in contrast is 440 tons, weighs 440 tons, so it's a really massive thing. Let's have a closer look. So being a laboratory, it has 16 racks for experiments and infrastructure, but primarily as mentioned, so we are interested in sensor data, in telemetry data. There are thousands of parameters recorded in one heart sampling and this ends up in like 10 gigs of data per year. And why is this important, you know, working with data, having models and so on, is because like microgravity really introduces some unique challenges, so due to microgravity you don't have convection, so convection doesn't work, air is not mixing and you might get local pockets of CO2, which is obviously not good for astronauts, so that's why it's really crucial that ventilation always works, so we have to monitor this. But being this sensitive data, so it's not allowed to go on public cloud, so we kind of had to develop this platform so it runs on all project partner sites, so this means bare metal clusters, corporate data centers, but we're also still using cloud, but only for development. Another requirement was that we can only use open source components and end users are actually data scientists and aerospace engineers and not necessarily software engineers. So yeah, we quickly realized that leaving compromises means losing a lot of comfort, so we suddenly had to take care of installing and configuring Kubernetes, taking care of storage, IAM, and all those other things. So but as you do nowadays, we started with the workshop and we collected some ideas what this platform actually should have, what kind of functionality, building blocks and so on. So we came up and quite soon that we're going to use Kubernetes, also explains why we're here, because it really covers a lot where it's a really flexible platform. Then we quickly also decided to go with GitLab for our Git and NCICD needs, and then when it comes to data science part, we also quickly zeroed in on Qflow. So those are three key pillars for us. And then some other components also followed quickly. So just to guide you through those key pillars, so it's GitLab, why? Because it's open source, it takes a lot of boxes when it comes to source code, CICD, and so on. And for Kubernetes, so we had to pick a distribution and we took a while to evaluate some distributions and then we decided to go with MicroKate. So the reasons were because it's really easy to deploy, works mostly with distributions. It supports single node deployments, but also multi node deployments. It's still a CNCF certified distribution, and it was also very nice for us, so it has a lot of add-ons, which actually get you started really fast, so we had things up and running quite fast. But still if you want, which you also did, you can replace those add-ons once you need more flexibility. And then finally, Qflow, it's a complete data science workbench. It's Kubernetes native, it's a mature and stable product, and it's also actively developed, so it's an easy pick. So very soon we also realized that we'll have to do some automation, so as it says in Kubernetes values, heroism is not sustainable, and it's especially important for small teams that automate as much as possible, so we have different targets for deployment and so on. So yeah, and we also had to learn a lot, especially initially, so it really made sense to immediately encode that knowledge into code. So our platform can be completely deployed, so the deployment is completely automated, so we are hitting all those different environments. By automation, so it enables quick iterations, so it takes really one hour for complete deployment from VMs to things running on top of Kubernetes, including Qflow. It's reproducible, it's also scalable, so it's easy to add new nodes, deploy new environments, and so on. Key components for deployment are for us Ansible, so this is our tool of choice when it comes to infrastructure as code, so it's very nice, so you work with VMs, it's implemented in Python, supports templating, and what's also important, it's agentless, so you only need SSH access, you don't need any dedicated demons or control node. Then the next one is GitLab CICD, so again, you define everything in VMs, it's very flexible, you can build software packages, containers, and then in the end also deploy to Kubernetes. And finally, we also decided to use Customize. So when you start deploying things from Kubernetes, you very likely will actually start with Helm because a lot of software is packaged with Helm, it's actually easy to deploy, but there are also online heated discussions, which is better, Helm or Customize. And those two tools are actually quite different, so one is imperative, supports templating, but the other one is declarative, but actually so Customize is also built in Kube control, works with Plain Emils, it's very minimal, and you work with Overlay and Patches. So we quickly realized that Helm was not flexible enough for us, so we decided to go with Customize because we needed to customize more things that you normally have available in that value.tml. So how we do it? So we have a separate repo for all the manifests, and we try to get official manifests, if those are not available, we render Helm Chasl, it has a template and everything goes into this repo. Then on top of this, we have Custom Overlays to do changes we need, and then in the end we simply use GitLab CI CD to deploy, so Kube-CTL, apply-K, and things are deployed on our clusters. Also a lot of software comes with Operator, so if you have this option, go for that, it's going to make your operations easier. And what we also do, we have a Custom Operator to handle dynamic changes, so for example if there are new users and so on, actually an Operator kicks in and takes care of some changes there. And for this, we actually just went with Shell Operator, because it's really easy to implement stuff in it. Okay, so the next big thing for us was storage. So storage, as you know, was initially not really well supported in Kubernetes, so Kubernetes was not developed for stateful workloads. But yeah, nowadays it is supported and you will need persistent storage. And there are two big types, so you have block storage and object storage. And first, we had to kind of wrap around our heads about how all this works, and just to give you a quick rundown. So you have persistent volume claims, you have persistent volumes and you have storage classes. And persistent volume claims, so those are the claims requested by POTS, they are namespaced and based on that claim, there is a persistent volume which is created either manually or dynamically and that's a cluster resource and that's then mounted into a POT. And behind the scenes you actually have storage classes which depend on storage engine or provisioner and storage classes actually define how your application and access mode works. So for our block storage, we decided to go with OpenEBS ecosystem. So we are using OpenEBS hostpad and Mystore. So Mystore is for high availability deployments because we also do single node deployments if needed. Mystore provides application but needs minimum three nodes and it's quite picky when it comes to hardware performance. Yeah, and this is how our storage classes look like. So we have two Mystore storage classes and one OpenEBS hostpad. And final storage component is Minio. So this is the one that provides S3 compatible storage for us. We went with it simply because it was already included with Kubeflow but we did update it to a later version but newer versions are under AGPA license so that might be a problem for some organizations but for us it is fine and works well.  Thank you. So let's talk a bit about how do we actually get users into the applications we host on Kubernetes. And here we use Istio. We didn't actually choose that by ourselves really if we're honest but that's also shipped via Kubeflow. Small team, we can't do everything ourselves. We can't then also rip that out of Kubeflow so we decided to go with Istio as a service mesh. You can see some examples so we use that for routing also. So that, for example, Slash Minio gets you to the web interface of Minio. We also use this then for authentication which is also very nice. So if you don't know that then if you are not logged in as a user that will automatically redirect you to our identification provider which will come to in a second. But this doesn't really help us yet how do we actually get users into our cluster because as you can see at the moment still is everything just running into the cluster. Originally we first looked at something called Metal LB but that seemed to be at least for now a bit too complex for our small team. So we did something like very simple. We just use EngineX as a proxy and have our users access EngineX. This obviously isn't very resilient because if that one host machine goes down everything's down our users can't access that anymore. So what we're really doing at the moment is something more like this. We have several host machines all with an EngineX proxy on top of it and then we do two different approaches depending on the deployment. So either we actually just give users really three different URLs. So in case something goes down which doesn't happen very often our users are at the moment are technical enough so that they can just select something else. Obviously this won't go for the long run when we really run models in production. This won't be a solution. Then we might have to look into something like Metal LB again. But for the moment this especially together with DNS failover works pretty great and we don't need that on every single node. Sometimes we have like then bigger deployments or add some more nodes which then don't come with EngineX on that. While we're at the topic of networking let's quickly talk about TLS. We can't go too much into detail here but as probably all of you know that's important and also important to get right. But on the other hand it's also hard. And so some people might be just tempted to say effort we trust our users. Let's disable that. We would say don't do that. And if you've been to any of the security talks here that opinion will probably be have been confirmed there. So what do you do instead? You need to learn about TLS unfortunately there really is no way around that. But then if you dive into it it's really not that half after all. Some topics you might want to pay special attention to is A how does certificate signing work and then also how do you distribute those signed certificates to your hosts. We use Ansible for that as well. And to distribute that needed at different places. So like on GitLab but also then on our Kubernetes nodes. And then you need to think about how do you get your certificate signed or by whom. So you can do that self signed which we just use for our development classes which we spin up well daily if not hourly. But then we also use let's encrypt for example or have those certificates signed by the root certificate of the organization we are currently deploying to. Which could be Airbus or one of the universities. Unfortunately this is about the time we have to go into TLS. But as I said important topic. What we also already somewhat mentioned is authentication. Here again we use DEX because DEX comes with Kubeflow. And well I've seen some people on the internet who actually managed to rip out DEX out of Kubeflow and use Keycloak instead. Which we would have liked to do as well because Keycloak there's just a lot more information available online like more documentation. They're more, well it's easier to connect it to something else. So we would have liked to use Keycloak because then we can also like connect it to Grafana for example or whatever other applications we're using. And then on the other end for example use Azure AD or GitHub as an authentication provider or whatever. But luckily that can really just be connected pretty easily. So that Kubeflow itself keeps continuing using DEX. But DEX then talks to Keycloak and then you can that way hook into whatever authentication provider you want to do. That works then pretty well out of the box. A note on Keycloak, at least the version we are currently using is not really Kubernetes native yet. So configuration is done via JSONs which is pretty ugly. So currently what we're doing is we're configuring it with a web interface exporting those JSONs and then loading them back into Keycloak. There are new versions available that use an operator and also then custom resources and custom resource definitions. We haven't tried that yet but it looks a lot easier to handle in the near future. Then let's talk quickly about how do you find out what's going on in your cluster. Especially so I've been to that exhibition hall down there and like every second booth seems to offer some observability options. We are currently not using something as fancy as that. So we went for Prometheus and Grafana also just because it's like a very easy, lots of documentation available, lots of people use it, resources available for almost everything. So for example, for Ksurf, we would use for model serving. It's well integrated. You get like pre-configured monitors and dashboards for almost everything you can think of. So that is really very nice and we are pretty happy with that. We currently use this Kube Prometheus project to also monitor our cluster itself because also we live on premises and so we don't get that by our suppliers. And that is also very nice because it also comes more or less pre-configured. You just need to make sure to use the right version that fits to your Kubernetes version. So we can highly recommend that as well. It also supports that metrics API that some software needs for example, Kubeflow, so full featured solution. We're very happy with that. We use a second instance of Prometheus to do application monitoring just because we tried to mess as little as possible with that Prometheus instance by Kube Prometheus. And then we also separately installed Grafana mostly because we wanted a newer version of Grafana. So later version of Grafana come with something like some Krabi builder. So if like me, you don't know really the Krabi language that Grafana uses, you also have pretty easy access to get whatever you want. So for us that looks like that. Also then logging, logging is also very important. Perhaps even more than monitoring. We use something else from Grafana there, Loki. Also very happy with that. And the users here are not only cluster admins, but also the end users. I'll quickly give you an example for that. So in Kubeflow, if you start a job for example, it's also a pot. And if that pot then gets cleaned up after some time, your locks are gone. But perhaps you would just check on a Monday morning what happened to my job. Job failed, you would like to see the locks not there anymore in the Kubeflow user interface by default, but then you can very quickly find it in Loki. We here use also like a bigger set like PLG which is short for Cromte, Loki, Grafana. But also we don't use that Grafana there just because we want one unified interface. So that looks then in total like this for us. And if you've been at the talk before, so we also use Postgres, actually the same operator that Karen talked about before. But also there we don't use that Grafana that comes with that, but use just one Grafana instance to just get all the data we want in that instance. So one big component we haven't talked at all here is really Kubeflow and we're not gonna go into detail here. Obviously we really like Kubeflow, otherwise we haven't decided to use it as like the main machine learning ops component of our platform. And our users also really like Kubeflow, especially like the pipelines I just mentioned, perhaps after a bit of an initial hesitation because it's a bit more complicated than just using a Kubeflow notebook now they really start seeing the value in that and are excited about that. But what's also very good about Kubeflow is that it's very flexible. So if it doesn't exactly fit to your use case, you can very quickly make it fit to your use case. But on the other hand, that flexibility might also mean that the defaults are not exactly where you need them to be. And so that means Kubeflow can be modified rather easily, but perhaps also should be modified to actually get that out of what you actually want. Some examples, so limits for example, for users are not there. Like on one of our very first deployments, like a data scientist found like a feature in those pipelines to start concurrent pods, did like 1,000 pods at the same time, worked fine, tried 10,000 pods and it turns out like et cetera didn't like that too much and that brought the whole cluster down. So since then we've had some users not done by hand, but by that operator that we wrote that someone mentioned earlier. Then there are quite a lot of default containers there, but they also might not be exactly what you're looking for. But also that's pretty straightforward to get like your own default containers which you can then also expose to your users that they can get like something that's a default perhaps for your organization. Stuff like access to those Kubeflow pipelines isn't also there by default like POT defaults, but we also supply that and would recommend you do that as well. In general, the documentation around Kubeflow pads isn't the best. We probably could contribute more and we just started with that. But what is very nice is like the community there's a Slack channel and people are really very, very helpful. So if you have any problems reach out there I can only recommend that as well. We're coming to more or less the end of the presentation and before we go we would like to leave you with some tips and tricks. So as you can see, we had quite a bit of fun with mid journey there. So all those like pictures are generated by that. Some ideas on debugging. So we also use Kube control locks and exit a lot, but then also we like K9S and Lens very much to have a look at what's currently going on in your cluster. Then we can highly recommend a YAML rally data for your IDE. So at least some and I are both using something from Red Hat called I think YAML language server, which is part of that LSP ecosystem, which means like more or less no matter what editor you're using, you can use that to get your YAMLs validated on the spot. Then we really like JQ, YQ to pre-process some YAMLs perhaps even before we do deployments. So make something fit the environment variables we set and so on. So then reminder check your hardware requirements. So especially Maya store really wants like low latencies for example, working requirements might also need to be checked something that we didn't do from the start. So microk8s for example, doesn't come with RBAC enabled from the start, but you need to manually enable that. We started without that and then enabled that later that led to some issues, which would have been like much easier to fix I guess when we had that from the start. We already mentioned Shell operator, but also a tool we would like to recommend is Bats, the bash automated testing system, which we use to test the whole platform in itself. Finally, like one of my least favorite bugs we run into is that if you have huge pages enabled on your host machine, which we needed to where we run a Maya store, then Postgres might have some issues if you don't explicitly give it huge pages like you can see on the right. So if you have issues like that, have a look, at least we didn't find the error messages there too helpful. And I think it's a bit of a discussion who's fault it is, if it's that's Postgres or the Linux kernels fault, but it seems to get fixed. So this brings us to more or less the end of our presentation. Let me quickly summarize. We showed you how we built an ML Ops platform, what components we choose, not all of them obviously. So there's like lots of topics we didn't touch at all like secrets management, like how do we do upgrading and stuff like that, but not everything is possible like in 30 minutes we have. Some key takeaways, we already mentioned the first one, automate as much as possible, especially if you're in a small team, like one day invested now might save you like a week or more later. So we would say definitely to do that. Then don't be afraid to asking for help on Slack channels and so on, but then obviously also don't forget to give back later if you found out something or if you can help somebody else. And then so for me, I always liked a lot to just experiment with stuff, but like if you dived into something, at least for me, new like Kubernetes which is really such a big ecosystem. It also makes sense to pick up a book or something and learn a bit about the basics. To conclude, building something like this with a very small team, especially on premise is hard and can't be done in four weeks. It took us something like six months I guess before people were really productively using that and we haven't figured it out completely yet. We are still like iterating, improving and so on. And I think we will also be doing that for quite some while. But the good news is if you really commit to it, it totally can be done even with a really small team. Before we come to the questions, I would like to remind you that this was a group effort even though it was only some of me talking. Lots of more people involved. Some are also here, I've already spotted. So if you want to come afterwards and chat about other topics or any of that what we just mentioned come by, they're probably also like hang out. Also, some of those people really know a lot about space. So if you were here for the space part and were really like a disappointed that we didn't go into that, we can also make that happen afterwards. Thank you for your time. And please don't forget to rate this talk. Thank you.


Turkey, mashed potatoes and other delicious foods were on the menu for the brainer community meal and thanks to a special group of people it wasn't just for those who made it to the American Legion that got to celebrate with the community Clayton Castle has that story It was a packed house today at the American Legion and brainered for today's community meal bringing together people young and old but before the 1130 dinner special deliveries were made to those who couldn't make it out to the Legion on this Thanksgiving day we have the police department the sheriff's department the fire department and we get each of their trucks and cars together they get to bring their families along with so that's a great experience for the kids and then they go out into the community so people can not only see that they're there in the good in the bad times but they're there in the good times too for some first responder families volunteering their time on Thanksgiving is an annual tradition it's uh it's really it's amazing to be able to have my family do this and instill those traditions into the next generation delivering meals as a way for the first responders to give back and meet the community that they serve on a daily basis well that's part of the reason I got into doing firefighting was to give something back to the local community and this is just another great opportunity for me to do that back at the Legion the community meal was packed as early as a half hour before the meal was served marking one of the biggest crowds in the event's history I do believe it's going to be the biggest here because already we have 50 more to go orders than we did last year and again it's not even 11 30 and we're packed already so I'm hoping that we have enough food to go around and for those who made it out to the Legion they were served with some good eating this year's the most the most turkeys I've ever ordered so we cooked 40 turkeys this year 15 cases of mashed potatoes it just seems like a large large amount so it's been pretty awesome the american legion eagles elks and vfw all located in brainard put on the community meal reporting in brainer clayton castle lakeland news any monetary donations collected at the community meal were donated to the salvation army food shelf if you've enjoyed this segment of lakeland news please consider making a tax deductible contribution to lakeland public television


So the General Scheme Maneuver for Bokenville II was to establish and operate expeditionary advanced spaces across the island of Oahu. We conducted an air insert up to Dillingham airfield on the north shore of Oahu, secured that key maritime terrain, and then flowed a follow on forces such as a gator radar into that maritime terrain, established and then operated that expeditionary advanced space. From there we secured and operated that EAB for approximately five days, and at that point we then conducted convoy operations down to Barber's Point. So having that gator radar there really provides not only the Marine Corps and the MLR, but really the joint force, a forward sensor on the leaning edge inside the wedge that can actually sense and make sense on behalf of the joint force to cue targeting, to cue follow on forces. The most difficult part of the training was really identifying all of the friction points that are associated with securing and operating that expeditionary advanced space. There's a lot of things that you just can't account for in planning, really having to think through and execute all of the finer level details, all the small unit leaders that have to do small unit leader tasks in order to make the overall mission successful. The main difference between Bokenville II and other exercises we've done to date is that this is really the first time the MLR has taken the MAG TAF together. Task organized, put boots on the ground and conduct an EABO. Getting the team together to do the detailed planning are things that we've done before, but this is really the first time that we had the opportunity to go from planning all the way through execution. This was the first time with the entire team across the MLR getting an opportunity to get after that problem set. And the large reason that we were successful in Bokenville II was because those individual Marines were actually involved. They were engaged. They stayed engaged. They did their job. That was the main reason why we were successful with those individual actions at the small unit leader and individual level.


Item No. SCP-153 Object Class, Euclid Special Containment Procedures Specimens of SCP-153 are to be kept isolated in acid-resistant containers, measured no less than 10m x 10m x 5m, partially filled with sewage and organic material, at Bio-Research Area-12. Every four hours, both the level of organic content and the structural integrity of each container must be checked. Under no circumstances are these containers to be connected to any internal or external plumbing system. Specimens of SCP-153 almost certainly still exist in the wild. Any reports of people mysteriously disappearing from showers and bathtubs must be investigated immediately. Agents must be equipped with infrared and ultra-violent sensors to circumvent SCP-153's camouflage. Agents should be brought alive to Area-12 if at all possible. Description SCP-153 appears to be a species of nematode, groundworm, that can reach up to ██ meters in length and ██ centimeters in diameter. Specimens of SCP-153 live in sewers and drain pipes, and while specimens of SCP-153 can live off of most any organic material, they prefer them to consume animal tissue. As the SCP-153 is screed a highly acidic substance similar to ██ from their mouths and gullets, and they have the ability to camouflage themselves both outside and in. SCP-153 is developed a hunting technique where a specimen will swim up a shower or bathtub drain, dissolve the drain cover with acidic saliva, and camouflage his gullet to look like a standard drain.  The entire process usually takes less than ██ seconds. Addendum 15301 Specimens of SCP-153 are to be kept isolated from each other except under controlled breeding conditions. Dr. Kobalyskia Addendum 15302 Incidents in ██ suggest that specimens of SCP-153 have started attacking people through sink drains, toilets ██ researchers are investigating how such an apparently simple animal can develop such relatively sophisticated hunting techniques. Addendum 15303 Dr. Kobalyskia has received approval to start a breeding research program with SCP-153 with emphasis on genetic engineering. The program ██ as well as a potential method of organic waste disposal. Addendum 15304 A controlled release of SCP-153 instances has been proposed as a potential containment method for stalling the spread of SCP-2717. Approval Pending