Hello, hello, and welcome to another Stack Rocks Office Hours, the second of the year. I'm super excited to have a special guest, Christian Hernandez on. We're gonna be talking Argo CD, security best practices, supply chain security, all the good stuff, all the hot topics, and we're coming off of some pretty exciting news, actually. So thanks everybody for joining. Before I get into it, Christian, introduce yourself, welcome to the show, and let us know what's going on in the Argo CD side of the world. Yeah, yeah, so thank you, thank you, Mike, Michael. I never know what to call you, actually. I've asked you before, but you said either one, so I'll stick with Mike because it's less syllables. Yes, so thank you for having me on. Yes, my name is Christian Hernandez. I am a technical marketing here at Red Hat. Focus on OpenShift GitOps, which is our supported offering of Argo CD. So yeah, so I've been here at Red Hat for almost eight years now, so it's been a while. So kind of funny background how I got here at Red Hat. I was like some Red Hatters, I was a customer and I was poached. I was actually the, so fun fact for everyone, so only for your show, because I haven't said it on my show yet. I was actually the second OpenShift SA hired in North America, so if you believe that, so for your Red Hatters, I'll give you a moment to process that. There was only two OpenShift SA's in North America at the time, so I was very busy. My background's basically mostly operations, systems management, that's kind of where, special place in my heart, right? So Andrew Sullivan, I talked with him a lot because that's kind of, I see Aida Aida with him a lot of things because he's in the operations. Ask of it, later on in my career, I actually moved into more like SRE work and so that's where kind of the first taste I got of CI CD and application delivery and keeping the lights on sort of thing, so. Yeah, and so going from being the second SA in North America to now, basically helping run GitOps to the point where Argo CD, 115% uptake in production in the last year according to the CNCF survey, so that's one of the main reasons why I want to have you on. You know, it's such a big background that you have and honestly that project is moving extremely quickly so I would love to, for anybody out there who doesn't know Argo, maybe break down the Argo project a little bit into its subgroups and what's going on in the space. Yeah, so when we say, so when people say Argo, they mainly mean Argo CD, but Argo is actually the project name because Argo CD, so the actual Argo CD, the actual CD part is, that's the actual name of what we use. So Argo is a project, right? An umbrella housing project for a tooling that was created add into it. So the Argo project has this history add into it. Into it actually, they were consulting with the company, Appalachics, and then they basically said, these guys are great, we're just gonna buy the company and move them in house. And they ended up doing that.  So one project being Argo CD, another one, very, very popular is Argo workflows. And there's other like Argo events and I'm missing one, the one that works with Istio, I completely forget the name. Oh man, they're gonna beat me up if I forget the name. Rollouts, Argo rollouts. Argo rollouts, right? And so Argo is basically an umbrella project created by Intuit, open source by Intuit. It was kind of their strategic plan to enter the open source community to basically take all the tooling they were using in house and open source it. And so, so yeah, rollouts, thank you. I knew I was gonna, someone was gonna help me there. So yeah, so basically Intuit said, hey, we've done a lot of cool things that helped us deploy our applications out. Let's just open source this stuff. And of course, Red Hat being Red Hat jumped on it saying this is great software. So I am, so being technical marketing, I am a member of the marketing committee at the Argo project. So that's kind of where I spend most of my time doing things like ArgoCon and their marketing type of thing. So we are planning another ArgoCon later of the year. So we'll look for that announcement. And again, I'm a big fan of the software. And so there's been a lot of, I do have one commit with Argo CD, very small commit, but I do have a commit. So as an end user, I do try to improve the software as much as I can as time permits. Yeah, that's awesome. And especially it is such an important space. And you mentioned it was donated by Intuit and it's going to be a graduated project soon. I think that's the goal, obviously. Yeah, the goal is. It's the one that looked like. Yeah, so the goal, so it was donated to the CNCF, right? So it is an incubating project. And they're actually looking for graduation. There's actually one more hurdle that they need to clear for graduation. But that, the expectation is graduating by the end of this year to be a fully graduated project in the CNCF. We'd love to see that announcement come. Was it October in Detroit? For a cute moment. Hopefully October in Detroit. Yeah, North America, yeah. Detroit, we can, you know, have fun in Detroit. That would be a great, great announcement. So we're, you know, touch wood, knock on wood. However you say, I know you're Canadian. I don't know which one you say. The American version or the English version. Yeah, we just do it with a little bit of an accent, you know? But yes, that would be amazing. And with such an uptick and obviously supply chain being so hot, I did obviously want to bring you on because there are some anti-patterns that I see with Kubernetes, especially, you know, in deployments, right? So I figured it was a good use case to bring you on and chat. Argo CD had to configure it securely. Yeah. Maybe some example pipelines that are good for demonstration to say, hey, like these are hardened pipelines that are sort of best practice, right? Yeah, yeah.  And like GitOps is essentially the cornerstone, right? Of the software delivery pipelines, right? Because the whole automation that takes place, you can use things like Argo CD for some of that, you know, supply chain, you know, building security first in that whole process is very important. And so, yeah, I'd like to kind of go over some of the patterns that you can use Argo CD with. You know, I messaged you saying like, I kind of want to go over like this Goofis Gallant, the sort of idea. So I don't know if any of you remember Goofis Gallant, this was, maybe I'm aging myself, I'm not sure. But it's kind of like the kid that does the bad thing versus the kid that does the good thing, like be like this guy, don't be like this guy, sort of thing. So I do kind of want to, so, you know, full disclosure, everyone, I actually set up kind of like a little demo, but I didn't actually really test it. Risky. So risky, always risky, it should work, right? Cause I've done, you know, things like this before, but we'll see if the demo gods are happy with me. But I do kind of want to go over kind of like, when someone gets started with GetOps, some of the, with OpenShift GetOps, which is Argo CD, some of the things that people automatically do, and maybe you should probably think twice about before doing. So I'll share the screen there. Yeah, that'd be awesome. Let's share it out and let's start with the basics. Oh, starting right on the command line, eh? Okay. Yeah, I don't mess around. I just go to command line right away. Let me, do I need to make it bigger? Is that fine? Yeah, I think it's okay, unless somebody in the chat wants to comment. Yeah, just, yeah, if somebody in the chat just let me know. Yeah, and welcome all the chat comments to everybody who's watching any questions, Argo CD related, security related, more than happy to call out live. Well, yeah, or if I'm doing something wrong, let me know, we like it all. So I have actually already installed, if I do OC GetOperators, I think for a, I already saw the, I already saw the OpenShift GetOps operator, right? And so normally what people would do, get routes on this guy, let's do, what was the name? Oh yeah, I almost forgot. But the OpenShift GetOps, right? And then I can get that route and go over here. And sorry, real quick for anybody, warning, potential security risk ahead, that's. Yeah, I use self-sign certificate. I trust my own self-sign certificate, so that's fine. You probably shouldn't. Anybody in the chat, we can see all the chats. So wherever, whatever platform you're on, throw it in there and we'll call it out. Thanks, Bjorn. What, the, so one of the things that OpenShift GetOps with you is like that integration with OpenShift, right? And so you can either use the Argo CD, ODIC connections, right? Or the OpenShift authentication, right? Obviously I recommend using the OpenShift authentication. So you have that single source of where people can log in, right? And so, which makes it convenient, right? And so here, I created a user. Let's see if I can remember the name. That's not it. All right, so this is the standard ODIC, right? And so they'll come in, they'll log in here, and they'll try to create a new app. So let's create a new app, my test, let's. So here's another thing. I should make this a little bigger. And this will come into play later, but. The default project. So there's a project, right? So this is where, especially with OpenShift customers, they get confused here. They see project here, and they immediately associate it with projects in OpenShift or Kubernetes namespaces. They're completely different. So talk about an overloaded term. I think projects are overloaded term, but this is, so here, Argos CD has its own concept of projects. So whereas in Kubernetes slash OpenShift, the project and namespaces, right? One project associates to one namespace, whereas in Argos CD, a project could spend many namespaces, right? A project is basically a housing for your application, your application can spend multiple namespaces, but for the sake of this here, we'll do project, we'll do a default sync policy, right? You can do automatic, you can do manual. We'll do manual. And you can change that after as well, right? The sync policy? Yeah, the sync policy, right? So you can do manual, you can actually do, let's do automatic. Actually, I think if I do actually have a YAML for this here so that makes it a little easier to read. Yeah, which is nice because all of that configuration that you just went through, you can save, correct, for Argos. So even though you're going through in the user, in the UI, you can have everything as code as well, right? Yeah, so here, this is the application, right? So this is kind of the YAML view of what I was doing in the UI, but to kind of say, hey, you know, I'm gonna deploy some Quarkus app, right? In this namespace, you know, this is where it trips up people, namespace, but then you have project, these are different things. So like in the namespace inside of Kubernetes, and these are my parameters and these are the versions, right? Kind of the standard thing for a helm chart, right? And if I can jump in, the reason for the project is also so that admins can help group developers that are working on multiple applications. Really like that's the use case for it because you logged in as a developer but there is sort of an admin view as well, right? Yeah, so the, and so that's kind of like where I was getting with projects is yeah, so like you'll have to set up some sort of RBAC because when someone comes in and wants to deploy this application, for example, I do an OC apply, if I spell it right, right here. Oh, look at that. Here it'll do, it'll try to sync. This will probably, this will fail because by default, OpenShift GetOps doesn't give the Argo CD service account access to do anything outside its own namespace. So. Which is the hardened OpenShift platform getting in the way a little bit, right? Yeah, yeah, yeah. So this is, and this is kind of like the OpenShift default for almost everything. And which is, it's kind of like the argument you always have with people is like, OpenShift is hard, I like, no, OpenShift is secure. You know, you're used to like being able to do everything.  If I do OC, just get SA, OpenShift GetOps. BNF user brought up a good point too about the projects. So that you don't have an overload when syncing. So you can break up your applications in the smaller. And it's a smaller, yeah, finite. So you don't have these massive microservices that will crash if you try to sync them all at once. So the instinct, right? And if you do OCADM, add, oops, I see in policy, I see in policy, add a cluster role to user cluster admin. This guy here, well, you don't need a namespace because you're doing cluster admin, right? Oh wait, I do need to provide those. So now if I, you know, I do that and I do terminate. So this is, because it's never gonna finish here. Oh, now I have to go to the sync, sorry. All right, terminate that. So if I design the, to terminate instead of, like refreshing the sync status? Yeah, so you have to terminate or else it'll never, it'll. Gotcha. I mean, you could technically wait for three minutes and it'll do it again. So, so this, so this way here, so notice two things, like one, I logged in as a developer, right? But since I gave cluster admin access to Argo CD, it doesn't matter what OpenShift thinks I am, right? So like just because I logged in as developer, it has nothing to do with the access that Argo CD has. So this is, so I wouldn't say this is necessarily bad, right? So like going back to the supply chain conversation, it's like if you're doing a supply chain delivery on an application, that may be fine to give cluster admin to your Argo CD or whatever get ops controller you have, but then you need to go in, you need to either trust it or don't trust it, right? So you need to either absolutely trust it and not let anyone log in to OpenShift, right? Or Argo CD because Argo CD has that access. And you can give Argo CD access per namespace basically, right? Into the Detroit to deploy and sync. So yeah, it's not necessarily insecure. It's just more of now there's another layer of a possible misconfiguration if somebody's not paying attention. And you give developer admin permission by accident through Argo, right? I think that's the worry. That's the worry, right? And then Argo CD has its own RBAC configuration. So let's go to my second cluster, the Mr. Gallant here. So this is why you see the multiples. So here, let's go developer, except of the same account. Here, like the same thing, right? Another like, I guess not bad or another anti-pattern I see is that like here I can go here and I can go to app details and I can do edit. Oops, not their parameters, edit. And I can say, hey, I want to change this from two to like three. Then save this here. So it'll actually deploy that pod. So within the UI. So I think this is what they call click ops, right? So like for me, from like being a get ops guy, like I wouldn't do this. Like I would actually check in this YAML and keep this in version control. So even if I edit this in the UI, it'll just revert it back to the known state, right? To the state that I said. So that's like another kind of anti-pattern that I see. If you're doing get ops, if you're not doing get ops, then it's fine. It's just automation. But if you want to go towards get ops, it's an anti-get ops pattern. Yeah, it's, you want to get to the point where everything is checked in as code and then configured into your environment, especially when you start working with developers. You don't want developers going in and clicking changes in Argo, correct? I think that's probably the biggest pattern you want to stay away from. So this is what this cluster, this user is going to do, right? So they're going to deploy an application here. So this is an application. This is not a Helm chart, but this application actually, if I do a tree of, let me clear this, that of this guy here. I have that, is this it? Is this it? Yes. I have the same Helm chart, but then it's checked in, right? Also another thing, which you brought up, like you can give Argo CD access specifically to specific namespaces, is that I annotated this namespace, right? And so by default, Argo doesn't have, like I said, doesn't have access to deploy anything to other namespaces. You can actually go and granularly add those in after the fact. Yeah, that actually ties in really well because Rockhound had a comment and they asked, so where would I basically limit our backs for devs in Argo, right? And so you would, go ahead. Yeah, so you would do that. So like I have here, so I have other applications, like the price list application, which is an app actually when you came on to my show, you scan that image. It's still bad by the way, but I haven't gone back and updated it, so don't scan this with the Stack Rocks. But this is an application that I'm deploying into an app project, right? So here, so this is kind of, this is where you define your Argo CD application.  So I gave it a name, what resources it's allowed to use. So you can actually even say this project can only deploy secrets, for example, right? Right here, since I want to deploy everything, I just kind of put asterisks there. Destinations, right? So you can then say only to this server, because you can manage multiple clusters with Argo, to this specific server, to this specific namespace is the only project, right? Source repos meaning like if you have different Git repos, you can list them like so like this app. So you can get pretty granular as you see here, just within this repo, only deploy it to this specific cluster in this specific namespace and only these specific resources, right? So you can get pretty granular and on top of that, you can assign roles to them, right? Yeah, that's awesome. And so just a kind of like quick background on roles and this always trips up people. There's only two roles in Argo CD, full admin and read only. And so like you can either give people the world or nothing, right? And then the reason for that is that everything else is granular, right? Let me make this a little bigger. So everything else, you have to define yourself. So for example, and then this is basic crud. So like it's kind of like a ACL crud, create, update, delete sort of permissions. So for example here, I'm gonna give the role developer. So whoever is the role developer, the ability, this is, it kind of like I'm speaking backwards. So the object is applications, you're able, you can get that. And within, for all applications under the price list, application, the app project, you can, sorry, in the namespace, in the project price list, you can get all applications, right, if you're allowed. So I always read it the other way around. I'm allowed at all applications in the price list namespace is essentially what, and then you basically have to do it for each one, right? So it's really granular. You have to say, I can get it, I can sync it, and I can list them, right? And so you're saying, what's nice about this is, and you gotta think that probably operations isn't doing all of this work.  But overall, I know that they can't break out of that one namespace because I've carved out that project, right? And this is like their little island, right? And that's their bubble. And so like you have your main operations that's kind of like, you know what, here, I'll let, because you don't want to be sending all these emails to developers on applications, I just kind of carve out those bubbles, and then you have different teams with different access and they can coexist and get as fine-grained as they want to get, right, with permission. Yeah, yeah. Which is really useful. It's really useful, right? And it has that aspect of delegation. Yeah, you give the power to the developer and the development teams, right? Yeah, yeah. And there's different ways of doing that, right? Like this is one way of doing it. Another way of doing it is have multiple Argo instances running because you can just, because it's an operator, right? So you can install the Argo CD operator to like the cluster operator to like manage the cluster, but then say, hey, developer, here's your own private version of Argo CD. That's another way of doing it. A lot of teams do it that way. Yeah, and you can, if there's an issue with having Argo on the cluster, you can also tell the operator to basically with a taint or toleration that Argo was not gonna run on this cluster too, right? That's just one of the built-in great functionalities of operators. Before you move on, BNF user had a question. How do you handle the state change with image name? I know not using latest tag isn't best practice, but using say, shop becomes impossible if you want to use it all as infrastructure as code. Yeah, so for GitOps, it's, and I don't know for what your opinion on this is Mike with respect to security, but for GitOps using floating tags as an anti-pattern because someone can always just update the tag, right? So like, because of GitOps, the idea is that what's in Git is reflected on my cluster. So if I'm using a floating tag like dev and someone changes, just force pushes an update on the image, like then there's a, then the application updated without me knowing, right? And so there is a project. It's within Argo CD project, they have like Argo CD labs. So it's guys like their sandbox way. There's a project called Argo CD image updater, which the idea behind it is it takes a lot of that guesswork out of how to update, manage, maintain those images. Yeah, to me, it also depends on how you're going to implement it if you're implementing continuous deployment, right? Where it's an automatic update, just understand that there's security implications behind having any sort of automation make changes like that. For me, I personally would rather there be some sort of manual process, automate everything up into the point where somebody's saying, yes, this is the tag that we want to go with, but really there should be some sort of, even though you can't just automate. Yeah, you need some sort of gate and you need some sort of tagging process that people follow and documented well. Yeah, I'm a fan of versioning. So I may be like stepping back for what I said. I personally don't use the Shaw, I use versioning, right? And so it's more like a process sort of thing where it's like, I use V1 and then when I deploy again, I use V2, when I play again, I use V3. That could technically be considered floating tags. When I say floating tags, I don't mean like environment tags, like dev, this one's tagged as prod. I mean like actual versions. Sometimes I'll like the get commit, I'll chop off like the last six digits or whatever and use that as a tag. So it corresponds to a commit that I did. But yeah, so a lot, so part of what the Argo CD image updater does is you can have it do a PR, right? So like you can have it like, either write back directly to get or do a PR for you. So I think a lot of people have done the PR as a gating mechanism. That's awesome. Because yeah, you wanna update to the newest version so it creates a PR, basically just updates the tag to pull the newest image version and then goes through that reconciliation process. Yeah. So, cool. Which is cool. Sorry, I disrupted the flow a little bit. Yeah, yeah, no, this is all great conversation. No, I love it. I do wanna point out though here. So, I did say there's only two roles, read only in admin. You might be saying, hey, Christian, I see you have role developer here. So like what constitutes a developer? Like how, where did that come from, right? So they say, this is policy. This is what the policy applies to. This is the object. This is what you can do to the object within whatever project and whatever your, so what is, you know, where does that come from? So, I once said that get ops is kind of like rat droppings, right? So there's, they're like all the configurations everywhere. I think that's Kubernetes in general. So they actually, let me pull that back up here. If I do, there we go, let's do an edit while we're, so that way I can use VI. So here in the operator, this is where I define it, right? So here in the operator, it says, all right, I'm gonna define a group. What's the name of your group? Well, the name of my group is developer and the role I'm assigning. So this here, this is basically, I'm just creating that definition. So whoever is part of the group developer gets assigned this role. What can the, what can this role do is then define within the project? So I'll let that simmer a little bit because it's really, really confusing. It took me a while. I can barely explain it now, but it's the recap here. I'm defining whoever is part of the developer group gets assigned a brand new role that I'm creating. So I'm creating this right now called role developer. What can that developer do? It depends how you define it in your project. So just kind of two ends of it. How that, so then how does that come through, right? How does that, let me go back here. If I go to user info, you'll see that I have that this user info is came from developer. This actually comes from OpenShift. So if I do a, oops, but OC get groups. That's really, you can, it says that I'm part of the developer group, right? So that's kind of how you tie it all together. You have three things, right? So you have the OAuth, SAML, whatever you want to do, right? I'm using HT Password. You can use SSO, whatever, right? How that group comes in inside of OpenShift, you can then tie it to Argo CD here. And then from there, you can tie that into your RBAC. And so, and then you can do this multiple times, right? So that you can, if you have an admin, you can create an admin group and they can do whatever you tell it to. So you can get mix and match that sort of thing. Yeah, and having just, having the two sort of separated RBACs, let's say, like having the ability to create a role in Argo and then the roles on the cluster itself allows you to get such a fine grade approach because Kubernetes RBAC is notoriously very annoying to work with, especially if you're getting into the weeds on YAMLs. So it's nice to be able to group it all together and then just say, here, Argo, you handle deployment within this and then we'll handle everything role-wise within a user interface that can then also be declarative too, right? Which, to me, that's awesome because, like, it's a year operations team. You can go and work with one team to create a template that then can be shared out to say, hey, here's a default of, let's say, senior engineers and junior developers or something like that. Default permissions in Argo, take it and expand on it to fit your use case a little bit more, right? So you have that capability, which I think is awesome. Yeah, yeah. Actually, so this is proof here that I didn't, I didn't test this. I know what the error is, right? It says resource quota is forbidden. So like, I'm not actually allowed to set a resource quota here. So this is, actually, I might file a bug on this, I'm not sure. Hold on, desired minus if you do. Oh, I know what the problem is. Cool, we can do this live. We'll do this live. So remember when I said, let's go back to, where was I? Yeah, there we go. Remember when I said that, that I can specify, I can annotate specific namespaces for it to allow to deploy to. So it looks like I forgot to annotate this namespace here. Notice I do open ship get ops. Look at, I push it, get status. Let's do this here. It might be just how they are back is set. And so for those who are watching, in order for our go to have permissions over the namespace, you add the annotation, right? And then it will, our back will automatically update so that our capability to push to that namespace, push changes for that namespace, I should say. Oh wait, I have to go over here and do this. So I have what they call an app of apps pattern. Unable to deploy revision. Yeah, so this is my R back, right? So I'm logged in as developer. This is my R back coming into play because I can't actually sync this here. It won't let me, because I'm a developer, right? So. Awesome, and it tells you that, right? That it's a permission issue? Yeah, it says permission, yeah. So if you were to log into it, min and hit sync, it would sync. Yeah, so let me log out. Let's do that. Put me on the spot, let's do it. I don't know what my password is.  So let's actually look at the R back. So I see here admin. Oh, I forgot to mention, you can set a default policy, right? So default policy, I set to read only. So if it can't find the user, so if I didn't define it here on lines 83 and 84, I'm pointing at the screen like as if you can see me. If you can't, if you don't define like lines here 83 and 84, if you don't define it, you can have a default policy to say we'll read only, right? So you could be a catch-all. Yeah. So I do have admin set there. So let's do this here, keep admin. So you find some of the top mistakes most of the time are misconfigurations between the two R back policies. Because so that's one that comes to mind. Another one that comes to mind is get repository syncing. That can be challenging. I think it's more of what you were writing about at Conway's Law, right? It's maybe trying to force the application to mirror your organization, but you might be trying to force the functionality too far. Yeah, yeah, it's, yeah. So it is Conway's Law.  Like it honestly is, it's like your organization isn't allowing for these processes, right? And so. Sometimes for a reason, right? Like it's not. Yeah, sometimes for a reason, yeah. Yeah. Because I've seen this happen a lot, especially with GitHub Actions, for example, when they first came out, right? They were so powerful in a sense to just be able to push and that everybody was giving access to GitHub to like push directly into a Kubernetes cluster, which it can be a little bit scary. It could be a little scary. So I know what this error is. I know it's basically, if I do OC Git, I thought I was going to run into this problem. If I do OC Git, it's role bindings, I think it is. GitOps. Oh yeah, I can do it in the namespace. Let's do in priceless, that's what it is. We have this guy here, Argo CD server. So the problem is, I'll just do this live. This guy here, the role is. Ah, the classic role and role binding dilemma. Yeah, that's right, I messed that up. This is exactly what I was saying about mess around with RBAC in YAML format, right? So anyways, the reason why my app isn't syncing, even though I have the annotation, is because the annotation is a pre-described set of, it's basically a role, right? It's a pre-described role. So when you annotate that, the backend controller sets all this up for you so you don't have to worry about. One of the things that it doesn't involve is actually, if you notice here, if you look at the resources, there's no resource quota, right? So like if I wanted to, I can go here and I can add resource quota, that would be a part of core, do you know? Questions for the audience. Oh, that's good. Yeah, we're putting the audience on the spot and I gotta go look at the Kubernetes documentation now. Okay, resources. Sometimes it's good to see other people debug. Okay, plus, no, so it is part of core, resource quota is part of the, here we go. So anyways, that's how you find out. I don't know if all you know that.  You'll see that pod is part of the core API. Horizontal pod autoscalers is actually part of autoscaling API group. Anyways. Cool. There's a whole fact there for you here. So here, API groups. It says all though, doesn't it? Maybe it groups. API groups get patch delete or secrets and config maps. Why is API groups there twice? There's one, so one is for, you get patch delete but not create, looks like. Gotcha. Let's just do this. That works. I'm not, I always have to look our backup so this may not work. I'd actually be surprised. Again, let's go back here. I'm admin, so I can sync it. Remember when it told me I couldn't sync. It can't sync now. Let's try this again. Let's see if this works or it errors out again. I may be putting it in the wrong place. In the namespace price list. I may be putting, let's try one more time. Oh, I know it's from operators. It removed the, it's helping you. It is helping you. That'll never go away. So the way around this would be to create a custom resource and attach it to that. So. This is kind of a good feature though because obviously if you have a hardened cluster with set features and you're trying to push something with no resource quotas, right? That is insecure. That is a good catch. That's one of the biggest issues I find sometimes.  Because they want their applications to run and then all of a sudden it's a big memory hog. That's what I call an umphest. Yeah, umphest. You get an umphest. Kubernetes has done a lot of work on memory and actually auto scaling based off memory. There was some features that came out in the last couple of releases. But I remember what two or three years ago when you just get the out of memory all the time and the cleanup for those was not really succinct in Kubernetes. It's a huge challenge. I'm very proud of this fact. If you Google GitOps dash examples I'm the first hit. Oh wow. So there you go. Google celebrity over here. Yeah, I'm gonna try to find another app here that'll... So let's do this live. Let's do this from scratch here. So this one looks pretty good deployment. This has the namespace though. I don't want to, that's always a problem. I could use customize but we literally be here an hour for me to do this correctly. So I'm trying to find something that is not on as a route. This one is overlays. Oh man. I did find it slightly funny that a lot of the conversation has been about RBAC but misconfigurations. I don't know if anybody watched the OOSP top 10 that just came out in 2021. I think the number one excuse was misconfigurations just in general in the cloud because you have so many applications, so many different permissions now that getting the cluster user project permissions correct without allowing for too much permission is a big challenge because you get into these situations where sometimes you're not quite sure exactly how to debug it and so you just log in as admin and hit sync, right? Yeah, yeah. And that's exactly what every human would do and so you need to have these processes in order that are actually, well one, useful, two, easy. There's the whole level of security of if an update means Chrome comes down and it comes down for five minutes, well I'm less likely to hit the update button, right? Yeah. So we need to get rid of those anti-patterns, get it set up properly and make sure that people are aware of their permissions, right? Yeah, and this is actually, like you said, an actually like a good thing that like even me as admin can't sync this, you know, good news bad news with operators, right? Like the operator, you know, switches that back, you know, whether or not you're able to set resource quota, that should be up to your organization. I think the operator should allow it, but resource quota is I can see the argument for not letting I understand why it's the default. The default is to not allow people to update resource quotas. Eventually you can get to the point obviously where you can say, hey, this namespace has this total amount of resources allowed, right? You can set that at the namespace level, but what's really cool I found is, like we're talking about Argo CD for developers, but Argo CD for your operations teams would be awesome too, right? You have a good repository, you have, you know, your operator declarations and yamls and stuff like that and you can make changes and then sync and update the cluster in a version controlled manner too, right? Which is a lot, so I, you know, you and I both deal a lot with our field here at Red Hat as well as customers and I've seen that pattern a lot where kind of like where I mentioned before where there is a admin Argo CD instance and that's kind of like the overarching instance that sets up the cluster for multi-tenancy. So you can still have multi-tenancy. I think that's where OpenShift shines a lot. There's a lot of tools for multi-tenancy and so we having, being able to still leverage the multi-tenancy and still have like total control is something that's really powerful, something you can do with Argo CD, right? So like with Argo, you have the cluster Argo and then you have maybe tenant Argos, I guess is what, you know, you know, you have those default, let's say cluster setups, maybe some of them are devs, some of them are tasks, some of them are greenfield deployments or something like that and you can use ACM or like ACM as a policy engine to go and then deploy those new clusters too, right? So everything has code for operators and developers and then- Yeah, common patterns. Yeah, common patterns and so, and then it's nice too because you get your security experts to go in and say, okay, well, here are the hardened defaults, like these are the things we can change on, these are the things we're not gonna compromise on. You know, maybe in this cluster we can allow a little bit more flexibility because maybe it's all stateless, right? Yeah, yeah, what another, I think like I'm thinking more of also just like the process so like Argos City will keep your cluster in sync, right? But like how things get into the cluster is also an important thing.  Like how did those images, you know, that with all those vulnerabilities, how did they end up on my cluster, right? And essentially it's just like my process is broken, right? The process of, you know, you can have, you know, the drift detection is fine, but like what if the declared state is bad, right? Like what if I'm running an image? Like how do I, you know, how did that get into my cluster? And I think a lot of, you know, we, at least I think more time should be spent on the CI part, right? On the pipeline part where it's like, hey, you know, you can do a cube linter, right? For your YAML and then you can do an image scanning and depending on your criteria, you can, you know, allow to promote that. Like you talked about gating, gating I think is very important as well. Yeah, the, there is a lot of time spent on runtime detection, drift prevention, right? Those are sort of your brake glass things, right? If Argo, let's say is automating something and it gets to the point where all of a sudden something, you know, crashes or there's a vulnerability that was introduced because of an automatic update, it's nice to have a tool to automatically notify you to say, hey, by the way, there is a serious vulnerability like a lock for shell that's in your, in your cluster in a container and something that is close to something important, right? Maybe a storage device. There's also the difference between something that's stateless that doesn't really have high impact, right? And it has maybe a bash capabilities. So that's a little bit of a different situation, right? Cause maybe it's not publicly available. It's very, it's a, maybe a developer tool to diagnose things, low impact, especially a dev cluster. So there's how we adjust and look at things in the cluster and then how we adjust upstream is the biggest part. And if you're, if they're the security team that just says, hey, by the way, this container is not, it's too vulnerable for us to run the cluster. Like you have to go and fix it. That's not really useful for the developer, right? Yeah. Realistically, it should be sort of cordoned off. You should go into the process upstream with Argo CD or some, or your CI process, whatever you have set up and say, hey, these are the packages that we're not gonna let through. So you just need to update and, and we'll notify you if anything like that happens, right? And honestly, if you can do it in an automated way, that's awesome. Like something like ACS does this, imagine having eight teams and you need to push log for J. I don't need, I can send out one email to all the teams and I can just enforce the policy to say, hey, you have to update anything with log for J to the new extension. Yeah, and it doesn't, it doesn't pass, right? And so, you know, then that's, I hold the supply chain developer. Like I always think about it, like, I think, I don't know who said this, like Henry Ford, right? It's like an assembly line. Something is bad. I just knock it off the assembly line, right? And, you know, in, you know, the cars don't get built with some, with a bad part. It's essentially, that's, that's the idea. Someone in chat actually, Luis said, better one Argo CD per cluster. I actually agree with that. So you'll never get, hear me fighting over that as a get-ups purist. If I can get up, get on my get-ups. So box for a second, like, so Kubernetes was the idea of like, I'll take away your SSH keys away, right? That's Kubernetes. So get-ups is the idea of I'm taking your cube CTL away. Like I'm taking OC, cube CTL away. I'm installing Argo CD. And if you want any changes, submit a PR, right? And so, and then, you know, it's kind of like that, that, that, if you think of a pipe, the faucet, right? That has multiple points where you can control the water flow, right?  That's me, but I understand not everyone has the same ability to do that or even the same, you know, even wants to do that. But then anyways, Luis, I agree with you. I'm all about putting Argo in and then controlling everything in the get workflow. Yep. We had this discussion actually because we were doing some commentary on education. And one of them was, you know, configure your application securely, but then deploy your application securely and somebody commented, well, aren't those kind of the same? And I'm like, well, are you giving somebody the ability to cube CTL deploy? Because that also means they can cube CTL delete. And so that's not necessarily, right? And how is those permissions working? And by the way, like, are you giving cube config files out to everybody? Like, how are you ingesting that? How are you getting your developers up and running? There's so much more operationally that it's involved. So, you know, setting that up where, hey, you have this, just here's your developer environment, check everything in to get. And maybe you do an automated scan, check configuration in dev and we leave it alone because it's in a private cluster. But by the way, you need to fix these three things before the next release in six weeks or we're breaking it, right? Like it's just not gonna get done in six weeks. And also your security team can have a little bit of give and say, okay, well, there's six top things, but these are the three things we need to focus on because we all know that it's engineering cycles. You can't be too strict, right? Yeah, yeah, exactly. We'll get to it to the next sprint. Someone mentioned in the chat, is there a CIS benchmark which includes Argo? I'm not sure, but that does put a good segue about the vulnerability that was on in Argo like a couple of weeks ago. Yeah, we do have five minutes left, so I would love to talk about that. It was fun. Yeah. So this company actually was hired by Intuit. I believe it either Intuit or just like the Argo project as a community to do a, what do you call it? Security audit, like an assessment security audit. And they found the CVE. So actually, can I put this in the chat? Can I chat here? No, I can't chat here. Anyways. I can put it in the chat. I have it, I'll put it up. Oh, you have it? All right, cool. What I found funny most about everything is they used the Red Hat logo and just turned it black to indicate a hacker, but okay. That's funny. That is pretty funny. So anyways, a too long didn't read version of the CVE is that there was a way to traverse the directory structure within inside of Argo. So specifically when you were using a Helm chart, there was the ability to, so the way Argo works, there is a, I can like quickly say OCD get pods. I'm going too much into detail.  One of the controllers is, oh, I just noticed that you can't see it. There we go. Here, one of the controllers is called the repo server and it's job is to basically do the get clone and the like get, essentially does a get clone, get fetch, does everything with the repos. Even when you're using a Helm chart, it saves it in the repo server, right? It's just like a generic repo server. So this here, they found a way to hop from one application to another. So here, here Argo CD, so each one of these little cards here is an application. And they're supposed to be self-contained, right? So the vulnerability is that they found a way to hop from this application to another application and see that application's data. Regardless of- The metadata, the secrets, what's injecting. The metadata, secrets, yeah, all that stuff. So me in a multi-tenant system, that's a big problem, right? Me, my app, I can go into Priceless app and look at the secrets, right? Database access, right? Especially namespaces, namespaces being such a significant security containment unit, let's say. Anytime you can hop namespaces like that, it's scary. Yeah, so when you can hop from one thing to another, it's definitely scary. Read that article, it goes into detail how it worked. So what can you do is essentially update, right? So- Yeah, it was fixed extremely quickly, right? I believe a full five days. Yeah, like within the week, they pushed a patch update. The community pushed an update and then Red Hat then not only pushed that same update, we actually back ported an update- To all the previous versions? Of OpenShift, right? So OpenShift 4.6 has a version of OpenShift GitOps, which is kind of older version of Argo, but they back ported that fix for that. So it's one of the, you know, it's a Red Hat show. So I'm telling you, it's one of the many great things you get from Red Hat is the back porting fixes. Yeah, and we're honestly like a lot of companies will just say, hey, we're not supporting this version anymore, you have to upgrade or get off, right? Yeah, essentially it's upgrade would be here. But we went back and we back ported that fix. So if you're running OpenShift GitOps, just make sure you're running the latest version of whatever version that you're on. So actually, I had this up here, where was it? It was on, for OpenShift 4.6, you need to be on 1.3.3 or newer or 1.4.2 for any other version. So, awesome. So this is cool, this is kind of, you know, showing, and this is kind of cool, it actually goes into the code. But it shows that the Argo CD community has essentially security built into the culture, right? So they hire people to say, hey, find something wrong, they find something bad, all right, cool, we'll patch it. It's kind of like their mantra for is security first. And honestly, this is, we do have to go, this is a great wrap up actually, because I did want to touch on two things. Is one, the speed at which open source community comes around to fix security vulnerabilities when they're there, I think is impressive. I think people are used to having their security applications gated behind some sort of subscription service. But then, how secure is your security platform? You normally are giving your security platform admin permissions in a lot of your clusters and environments, right? So it's nice to be able to have eyes on these things, to trust them. And yeah, that's a great segue to basically just saying, everybody come back in a month, we're gonna have some Stack Rocks open source news for y'all that are watching. And you can join Christian, get ops guide to the galaxy, is it every two weeks? Yeah, every two weeks, every other Thursday. So not this Thursday, but next Thursday. Next Thursday, I'll be talking about Argo events.  If you wanna go check it out, I did a talk about Stack Rocks showcasing ACS when we first got acquired and brought over Red Hat. So you can check all that stuff out in the library. Christian, thanks for joining, thanks for your talk. Oh, thank you, I appreciate it. Appreciate the time and look to see you back here next month, third Tuesday of the month for some Stack Rocks news. Take care, everyone.


Good afternoon everyone and welcome. I think we're going to get started. I'm Ann Del Castiglio, commissioner of the mayor's office of media and entertainment. As some of you may know, we started out as the film office and our office has since expanded to now five divisions. So in addition to supporting film and television production in New York, we also run the office of nightlife, NYC media, the municipal broadcast and radio network, press credentials, and creative sector programs, film, television, theater, music, advertising, publishing, digital content, and video games. Together these sectors represent close to $150 billion in economic activity for the city and close to half a million jobs. Film, of course, continues to be the lion's share of that activity, representing $82 billion in economic activity and close to 185,000 jobs, representing about 6.5% of New York City's overall domestic product. We know that film fuels more than just production businesses. It also fuels small businesses such as yours, such as the ones probably represented on this call, cafes, florists, slumber yards, dry cleaners. And so, you know, knowing that we recognize that the strike, the writer's strike going into now going into its second month is likely having an effect on many of you. And so we wanted to be sure that we were able to connect you to resources and information that can support your businesses through during this time. I am delighted that we have our colleagues in government represented, Lily and Vale from Small Business Services, Matthew Kennedy from Empire State Development Corporation at the state level, and Rob and Daniels from the Small Business Administration at the federal level, who are all here to share with you some supports and information that will hopefully be helpful to you. I also want to thank our staff at MOME for pulling this together, in particular, Etabria Crouch, Noel Murray, and Marisa Rodanti, as well as our comms and marketing team for getting the word out about this. If there are folks that you know who weren't able to join today, we will be recording this session and it will be posting it on our social channels so that if folks weren't able to attend, they'll still be able to access the information presented to you today. So thank you again for being here, and now I'm going to turn it over to Tabri. Thank you, Commissioner. Just a reminder for everyone, please type in your questions in the chat box. There's a Q&A chat box, so not the chat to the group, but the Q&A box. Thank you. We will devote the last half of this hour to questions and answers. So we're going to start with our first panel speaker, Lily Vail. Lily is the Senior Program Manager with Capital Access Funds of the New York City Department of Small Business Services. Lily has seven years of experience working with small businesses in New York City and the Midwest. As a Senior Program Manager with the Capital Access Team at New York City's Department of Small Business Services, she supports with the operations grant and loan funds focused on supporting small business owners access the capital they need to grow and succeed. So I'm going to let Lily explain more about what she does at the Small Business Services Agency. Of course. Thank you so much for that introduction, also for the opportunity to share more, both about the Department of Small Business Services, as well as the specific ways and resources that we can support everyone here. So with I am a part of the Capital Access Team within the Department of Small Business Services and as a whole, the objective of the department is to really ensure that every single either business owner or aspiring business owner entrepreneur in New York City has access to the resources, whether it's capital resources or social resources, have access to the things they need in order to grow in order to create stronger businesses and really ensure that we have thriving neighborhoods. Currently, some of the services that we provide and we can connect you to, we have business courses, legal assistance that we can connect you to, financing assistance that we can connect you to, as well as different training programs and certifications. How can you get in contact with those or how can you get connected with those? Well, first with financing assistance, well, we don't have the city or the Department, SPS, we don't have a grant or loan program currently active specifically for this industry. We currently have the contract financing loan fund if you are working on any sort of city agency contract and this can be a loan up to $1 million with a 3% interest rate that you can apply for either while you're bidding on a contract or once you receive a contract. That said, what we can do and one of the best services in my opinion that we provide is we have a network of business solutions centers and industrial business service providers that can reach, that can meet with you one-on-one or through a webinar to share more about what resources they can connect you with. The industrial business service providers, it's a network of nine centers that are physically located in all five boroughs and they serve businesses in the industries of manufacturing, construction, transportation, wholesale, utility, film production industry, sorry, and the business solution centers, those are seven centers within the five boroughs who can work with all other industries. And with these centers, they can either connect you with a lender or an organization that is doing loans or grants that really fit what you need and they can walk with you through the loan application process and really help you pre-financing but also walk you through so you really understand what you are eligible for and what you need. I think with access to capital, making sure that you are looking for the right type of capital as well as answering the right questions is one of the biggest parts of ensuring that you are successful and we're helping you reach that. Another huge part of the financing assistance or the sorry the business solution centers are our financing managers and their ability to support you with credit resources, whether it is building your credit or finding out different ways to build your credit and then building a strong budget infrastructure for your business. So through these resources, we can ensure that you have an understanding of how you can grow and what are the different stakeholders or what are the different tools and resources that are available to you within the five boroughs. Thank you. Oh, and you can access all these resources by going to nyc.gov forward slash financing assistance and I can share that in the chat. Thank you so much for that Lily. Lots of great resources coming through and support services coming through the Small Business Services Agency. Our next speaker is Matthew Kennedy. Matthew Kennedy is an economic development program specialist with the Empire State Development Division of Small Business and Technology Development. He is alumnus of Potsdam College and Clarkson University with a degree in economics and a lifelong resident of Half Moon, New York. His primary focus at ESD is promoting the growth of small businesses through access to capital with a New York State contractor financing program, JDA agriculture fund and small business revolving loan fund lending programs. And prior to joining ESD, he was a successful small business owner and real estate developer in Saratoga County. He has also worked as a loan officer at several banking institutions, spent over 10 years as an internet sales and marketing manager and operated a successful retail computer store for a decade. So he knows a few things about running a small business. So Matthew, thank you for joining us today. I'm going to let you explain more about what you do and what services you provide through ESD. Terrific. Thank you. Well, thank you for having me. So my name is Matthew Kennedy. I am from Empire State Development. Of course, Empire State Development is New York State's economic development finance or economic development agency. I am specifically from the Division of Small Business and Technology Development, part of the access to capital team. So I do have some quick slides I can go over some of the programs, some of the things that the Small Business Division does offer. And what I'm going to touch upon today is just a fraction of what the Small Business Division does. Small businesses do make up 98% of New York State's businesses. And we focus specifically on those businesses with a hundred or fewer employees with a variety of different programs, including our technology related programs for small businesses, things like NYSTAR, our New York Ventures programs, our Life Sciences programs, our New York State procurement assistance programs, which would be things like the contract reporter for helping you do business with the state of New York, and our lending programs and our financial resources, which is the team that I'm on. And that's what I'm going to talk about a little bit today. So our lending programs are traditionally going to be programs that we would call facilitated lending programs. They use a third party to offer capital to the marketplace. The loans are not directly handled by Empire State Development. They're handled through agreements that we've made with different organizations. So we have quite a few different programs that are part of it. The New York State Contract to Financing program, which is one of the SSBCI initiative programs I'll talk about in a moment, our link deposit program. We have our alternative lending programs through our CDFIs, which is various revolving loan funds, actually about a dozen different programs that are all lumped together. Also, other programs just to be complete here that I'm not going to talk about today are surety bond assistance program for contractors that are doing construction work, our GDA Agriculture Fund, which is specifically for agriculture businesses. And we have something new coming later in the summer of the New York Forward Loan Fund too. I don't have too much to share about that, so I won't be talking about it yet. But that is a new program we're going to have later in the summer that follows on our original New York Forward Loan Fund from COVID, from the COVID times. Also, there are some direct lending programs. I'll just mention, just in case you've ever heard of them, the GDA Job Development Authority, the Metropolitan Economic Revitalization Fund, and our Capital Project Loan Fund. There are some programs that do exist that are direct lending programs. Those are primarily institutional and project-based, so I'm not going to talk about them too much today. But they are out there. I just want to give you an idea as to the depth of the sort of programs we offer. So you heard me mention the SSBCI programs. So to give you some background, the SSBCI is the State Small Business Credit Initiative. It was part of the American Rescue Plan. Actually, technically this is SSBCI 2.0. But all of the states, all 50 states, the federal government granted funds to the states to execute programs to help small businesses. New York State's allocation, one of the things that we're doing with it, or one of the things our team is doing with it, is setting up work programs in particular that are using this funding to help get the money to small businesses. Our Small Business Revolving Loan Fund, round two, is a new revolving loan fund through CDFIs that will help CDFIs get a capital that they can re-lend to you. That should be starting actually probably the first week in July. Our New York Forward Loan Fund 2, which I mentioned briefly, that's going to be an online platform that's going to offer small business loans through multiple lenders where you can go online to essentially a clearinghouse to make a contact with a lender. Our bonding program has expanded along with the contractor financing program, which I'm going to talk about now. Our contractor financing program is specifically for helping businesses that are working on government funded projects. So the contractor financing program provides loan loss reserves to lenders that are participating in the program. There's going to be seven of them. To help contractors, and when I say contractors, I don't just mean construction contractors, I mean contractors in general doing any sort of work on a government funded or government-based project. Typically, these are going to be smaller businesses, the fewer than a hundred employees, about $5 million in revenues. And these lenders can offer lines of credit or managed lines of credit to help you complete those particular projects to help smooth out payments. They're offered again through seven lenders, which are contact information is available on our site, which I will show you in a moment. But they are for eligible contracts, or you can be a prime contractor or a subcontractor on a prime contract working on any project that is receiving financial backing from a state agency, state municipality, city, county, town, school districts, a federal project if the work is being done in the state of New York will qualify. So this is a very wide array of things. If you were doing tourism work for a county, that can qualify or for a city or for the state or anything that is going to have that funding. And again, those loans can be used for deployment of that contract, advances against inventory, sometimes it's purchase orders, managing payables, you have to hire temporary staff, all sorts of different finance costs, which work in capital to help you complete the projects. And the duration of the period of the loan is going to be based upon the work that you're doing on your actual contract, the lender is going to establish that with the interest rates and so forth. Generally, they're going to be in the 18 month range, it's going to depend upon what your contract is. The lenders that are part of the program are experts in doing these types of loans. So they will be able to help guide you through the process and to help you understand exactly what the pitfalls are, what will you be waiting for, things like that. So now to change gears real quick, I mentioned our CDFIs, our CDFI programs. We do offer an alternative lending directory on our website. CDFI is a community development financial institutions. They are non-traditional lenders. They're lenders that are typically nonprofits that usually work in a smaller geographic area that are more directly linked to individual communities that can offer lending to businesses that might not conventionally be able to get lending through a traditional larger bank. So startup loans, small working capital loans, small lines of credit, small expansion loans, there are 60 lenders that are in our directory. Some of those small business evolving loan fund lenders will be in that directory that cover different areas of the state. You can go into the directory, choose your county. It's going to give you a breakdown of as to who you can contact in that area and the types of loans they offer. But these are organizations that can offer loans anywhere from $500 to $4 million, depending upon the organization. It is a great resource to look at. They all have some sort of credit evaluation system that steps beyond your traditional credit score. They're going to be able to look at your ability to pay. They're going to look at your cash flows.  And now finally, as one quick step, we do have a program that's through commercial banks that's called the Link Deposit Program. And this is actually a terrific program. It's been around for a very long time, but it can offer you an interest rate reduction on a loan, a fixed rate, a fixed rate fixed term loan through a traditional commercial bank. These loans can offer it's up to a 2% interest rate reduction based upon the lender's four-year CD rate. Since interest rates have popped up so much in the last year, this program activity is picked up dramatically.  The state of New York makes a deposit into that bank for the amount of that loan in a four-year CD and up to 2% interest that they would typically get paid on the four-year CD. Instead of New York getting paid the money in interest, that interest money comes off of what you're paying on your loan. So it reduces the cost of your loan by that interest rate on that four-year CD. Right now, again, interest rates have gone up. So four-year CD rates are high. A lot of times now they're starting to hit that 2% cap or a couple years ago, maybe they weren't. Typically, these are going to be businesses that are manufacturing firms. They're going to qualify for this program. Manufacturing service businesses that are business to business services, not necessarily service businesses that are going to individuals like a daycare or a funeral home or a salon, but a business to business toward a service business. Start-up businesses, unfortunately, are never eligible. Business does need to be headquartered inside of New York state and business headquartered outside of New York state can't qualify. Typically, in most situations, there are certain businesses that are not eligible. Retail businesses aren't eligible. Usually professional services aren't eligible. That would be something that if you required a license to operate in the state as far as like a healthcare legal accounting, so forth. Or a personal service business again doesn't qualify, but there is an important exception I want to point out that a business in a distressed census tract can still qualify even if they're in that category. So if your business happens to be located in a distressed census tract anywhere in the state that can potentially qualify for you that will bypass that requirement, you can be part of the link deposit program. So there are 64 current lenders that are part of the program, but if you're looking for more information, I would check out our website, or you can always contact that phone number and the link deposit website or link deposit email address directly, give you more information on link deposit. We are going to answer questions at the end, but I do want to give you my small business hub web address, and I want to show that QR code again, you scan QR code, go to our SSBCI website. So thanks for the time. I know I ran through as quickly as I could, and if you have further questions, please reach out. That was fantastic. Such a plethora of products and resources through the Empire State Development Corporation. Thank you so much, Matthew. Okay, all right. Last but not least, we have Robin Daniels, our lenders specialist from the New York District Office of the U.S. Small Business Administration. Robin has been working for the U.S. Small Business Administration for more than 30 years. She started her career in disaster assistance, dealing directly with barbers and loan collection. In 2004, she moved to the 8A Business Development Division in the agency's New York District Office. Following her work in business development with small businesses, she took on a new role as a lenders relations specialist. In this capacity, Robin provides training to lenders on loan processing, servicing, purchasing, and liquidation. She also represents the New York District Office finance division in matters relating to financial institutions and the operation procedures necessary for use of the SBA's loan programs, while creating outreach efforts, coordinating with lending institutions, and SBA's resource partners to develop access to capital roundtables, panel discussions, and networking events. Well, welcome, Robin. We look forward to hearing more about the services and products offered through SBA. Hello, and thank you. As she introduced, my name is Robin Daniels. I'm a lenders relations specialist with the Small Business Metro New York District Office, which means I am here to help you understand how the SBA gets you access to capital via our loan guarantee programs. Capital access is one of our three C's, which are counseling capital and contracting. The U.S. Small Business Administration makes the American dream of business ownership a reality as the go-to resource and voice for small businesses backed by the strength of the federal government. The SBA empowers entrepreneurs and small business owners with the resources and support they need to start and grow their businesses. It delivers services to people through an extensive network of SBA field offices and partnerships with public and private organizations. The SBA loans, they are not a direct loan from SBA itself. Rather, it is a loan that has been made by a commercial lending partner, which are our banks. The SBA has the guarantees for these partners and has been structured according to SBA's requirement. This helps to minimize the risk for both partners and borrowers. Only those reasonable access to other lending sources are eligible for such a loan because guarantee requirements and practices are dependent on the U.S. government, I'm sorry, changes to policies or economic conditions that may vary from change during lending terms. With that, the SBA's resource partner network is a network that helps to expand our outreach to service the nation's small business community outside of our district office. Whether you need to create a successful business plan, get expert advice on expanding your business or train your team. The SBA makes sure you're never far from the help that you need. You can find our resource partners closest to you by going to our website sba.gov slash local assistance. Next slide, please. Our SBA loan programs are our 7A loan programs. We have our 504. We have microloans. We have community advantages as well as international trade. Our flexible loan programs can provide financing to a variety of business needs. We offer fully amortized loan with no balloon payments, balloon payment surprises, as well as offer flexible repayment options. What's more, what's more is SBA loan provides the proper debt structures that's matching assets to liabilities. Next slide, please. Now, we have our 7A loan maximum guarantee is the 85% for loans up to 150,000 and 75 for loans above 150,000. Maximum loan amount is $5 million. The maximum guaranteed amount on that $5 million is $3.75 million. We have our SBA express program, and you can see that is a streamlined process. The maximum on that one as of September 2021 is $500,000. Next slide, please. Okay, so we have our community advantage loan program that is actually going to underserved markets. Businesses located in low to moderate income communities, empowerment zones, or enterprise communities, which is the EZ slash EC. We have our historical underutilized business zones, which is the hub zone, and that is in rural areas. New businesses, those in operations less than two years can actually qualify for the SBA advantage. We have our micro loans on loan amounts up to 50,000, and those are administered by our CDFIs, which are our non-profit lenders. The maximum interest rates going from some of those lenders are prime plus six percent, and the maximum guarantee on those ones also are 85% for loans, 150,000, which goes for the community advantage. Next slide, please. Now, we have our 504 loan program. The loan program is a long-term financing tool for economic development within the community. 504 loan program provides growing businesses with long-term fixed rate financing for major fixed assets such as land, building, or heavy machinery. Our CDFIs, which are not CDFIs, our CDCs, my mistake, which are certified development companies, is a non-profit corporation that is set up to contribute to the economic development of its communities. We have three CDCs that work for us, and they actually do all of our paperwork for SBA. They also help with 90% financing. You can get mortgage recording tax savings, as you can see, and can be up to 20 years to 25 years for term real estate. Next slide. So the 504 project includes a loan structure with a senior lien from a private sector lender, covering up to 50% of the project cost, a loan secured with a junior lien from the CDC backed by SBA 100% guarantee, and the last percent, and that will cover up to 40%, and the contributions of at least 10% equity is going to come from the small business that is actually being helped. Next slide. We have our 7A international trade loans. 95% of the world's customers are located outside of the United States, and SBA international trade loans help small businesses explain globally and increases revenues, and they have a guarantee from 75% to 90%. Now, there are three distinct loan programs that we do have. We have an export express. We have the export working capital program, which is the EWCP, as well as international trade loans. Next slide, please. Now, this is the disaster assistance loans. Sorry, I got kind of lost each year to SBA provides billions of dollars in low interest, long term disaster loans to help small business owners and renters recover from declared disasters, such as real estate, personal property, economic, machinery, inventory, and if you're active duty military. Now, this is our last one. As I stated before, we have our guarantees fees that are actually based on the total loan amount, 150,000 or less is 2% guarantee fees. Over 150 to 700 will be 3% and so on and so forth. Now, to offset the cost of SBA loan program to the tax pays, the agency does charge lender a guarantee fee and a service fee for each loan and that's it is dispersed. The amount of fee is actually based on the guarantee portion of the loan. So with that last one, my name is Robin Daniels. If you need any of the further information, all of my information is there and that is the end of my presentation. Turn it back. Thank you so much, Robin. And now we will open up the floor to audience questions and have our panelists answer your questions. It looks like we've got one question up so far. Can Mr. Kennedy please show the slide with the QR code and contact information and then we'll also have Robin share the project. Let's see. There you go. Great. And this webinar is being recorded and it will be uploaded to YouTube so everyone will have access to the webinar post this. Yes, and then we'll also be emailed to register and so do not fret. You will still have access to this information once this webinar ends. Can we also see Robin's contact info? The last slide on her deck. Fantastic. Okay. Well, I'm wondering, you know, these are precarious times for many small business owners affected by the WGA strike. I'm wondering anyone from our esteemed panel can help address how best are the recovery loans, how best can our businesses, our small businesses navigate this time with the services and resources that either you've mentioned or any other resources that you can suggest even outside of our agency's resources. So thoughts on that? Well, I would suggest to, you know, take a look at our website. Please take a look at our ESD website for the Small Business Hub. Small Business Hub, there's a great deal of information in there that goes beyond these particular programs. There is a great directory. There's a directory of small business programs that gets updated on a regular basis that goes across all agencies, goes beyond our agency as well. Also, you know, we do offer support services, things like our entrepreneurship assistance program, our entrepreneurship assistance centers around the state along with there are, you know, other technical resources that are available, even if you're looking for work or if you're looking for context to fulfill, you can always look at our New York City contract reporter as a place to find work. Not everything gets listed there, but state agencies and authorities that are required to post anything, it's over $25,000 in allocation are required to post it there. So there are different avenues that you can go down to try to maybe, the word I'm looking for would be diverge a little bit from what you might be doing if you're looking for other work that you could do to try to make up the ground for not being able to do the work that you normally would do because of the WGA strike. So ways to diversify, you know, it's the same thing that in some ways we all went through with COVID where a lot of businesses needed to pivot and this is just another opportunity. We seem to have to pivot so much these days, but, you know, there are resources out there, there really are. Don't lose hope. Thank you, Matthew. Robin, Lily, is there anything that you can add to that from your agencies in terms of resources and opportunities to pivot in these times? Okay. Let me answer that one. So SBA, can you hear me? Okay. So SBA, we actually do have, we do service the 14 downstate counties from New York all the way up to Elster County and we do have free resources. We have our resource partners and they are our small business development centers. We have our veterans outreach business centers. We have SCORE as well as women's business centers and all of them are our free resources that where you can go and have your business plan looked at, your financial services, any aspects of your business, they will give you free small business guidance or one-on-one counseling and I'm going to stop saying free because it's actually your tax payers dollars at work. So those are our SBA resources and if you check out our website, you will find a whole lot of information there. Thank you so much, Robin. Before we move on to our audience question, Lily, can you, is there anything that you'd like to add? For sure and not to reiterate, just presenting on what I did prior as well as what the state and the federal government, all the resources they have. I think there are so many just tools and resources out there. It's just understanding what is best for your business and I know that we are all three of us. We are working to ensure that we can connect you with the most appropriate tool or resource no matter if it's at one of the business solution centers or at the SBDC, the center that Robin just mentioned. We want to make sure that you are able to look for it. So to reach out, we can definitely find a way to put you in contact. Essentially, you're not alone that there is support out there whether that is, you know, like, you know, it's a check up for sure in terms of the health of the business that resources are available to help with that. Thank you, Lily. So we have a question from an audience member. What can you offer smaller soundstage facilities in assistance or grants while the industry is shut down in a shutdown mode and the impending SAG after strikes? So essentially what smaller soundstage facilities, we have our bigger soundstage facilities but we also have smaller soundstage facilities, what can you offer in terms of assistance or grants while essentially there's some business because of the work stoppage? Well, okay. I'll go ahead. Well, I was going to say just so that you know, the federal government does not have grants. Unfortunately at this time, during the pandemic, there were different things that was offered. And unfortunately, those programs have now closed. So right now, unfortunately, the only thing SBA has are our resource partners as well as our loan programs if your business is in need of them. Matthew, you can say. Yeah, we're really in the same boat. The grant programs, unfortunately, have all now ended. Of course, we did have two major ones that were part of the COVID recovery. One was COVID recovery and the other one was our seed grant that was last year. There's nothing really on the horizon than I am aware of as far as grant programs are concerned. But again, there are those access to capital resources. So there are some capital resources if you need a short-term loan. Those might want to look at the capital resources to contact the lender in your area, contact the CDFI in your area, someone that is familiar with your plight and the plight of your community and try to reach out to them and see what you can do. That's probably what I would suggest. So given that there are no grants, is it common to have our small businesses approach each of your agencies for assistance in loans and essentially just sort of keep them a prop while they wait for the work to essentially to return? Is that common? This is sort of a common scenario. How likely are they to be approved for access to these loans? Well, I can say for our group, it's fairly unusual for us to even offer grants. Really the first time the Small Business Division has offered any grants was due to COVID. Prior to COVID, there were no grant programs. We've always focused on lending programs because the limited amount of resources that we can put in them can recycle and can touch as many businesses as possible. Sometimes our funding sources don't even allow for us to use the money for grants because once a grant goes out, the grant is gone and then the money is used. Occasionally, we might encounter something where we got some special funding for a grant program, but unfortunately it's a very rare circumstance. Usually it's a matter of unfortunately hunkering down a little bit trying to control your costs and waiting out the storm. It's not that unusual for someone to have to go get a loan. Like I said, these lenders, the lenders that I mentioned are all very capable of looking at things, your cash flows, looking at other situations, realizing this isn't going to last forever. The work will be back and chances are the work that's not happening now will happen then. It's a matter of kind of holding out hope, but it's not unusual necessarily to go have to go get a loan. Thank you, Matthew. I think we're all needing a bit of hope. Robin, Valen, do you have any additional thoughts on how to mitigate these times? Our SBA funding programs have been a major part of the struggling businesses. SBAs and that work of resource partners have given guidance, like I stated before, throughout every aspect of business. So you can actually call or go to our website, find the local assistance in your area, set up an appointment, talk to them on the phone, ask them any questions. They might have different information than we do on the side. I'm not sure, but it's free. You can always go to the Small Business Development Centers, the SCORE. We call them WBCs. We have our VBOCs. So any one of our resource partners, you can reach out and see what type of information they may have or any assistance they may be able to help you with. So that's outside of our 7A and our 504 loan programs. I think to add on to that, developing relationships with the CDFIs, the Community Development Finance Institutions, at a time like this, they're all mission-based lenders, so they won't put you in a situation where it's going to be detrimental to your growth once it returns to quote-unquote normalcy. And really looking at more often than not, they'll look at your financials and see what will get you through X amount of time. And there are also opportunities using these business resources, whether it's a business centers with either of us, to just reorganize your budget or reorganize your finances to see what you need to do in order, like Matt said, just to hunker down to ensure you're able to move, just to move forwards. Thank you for that, Lily. We have a very specific question from an attendee. So the question is, we have variable interest loans for equipment and capital improvements that are very expensive right now because of high interest rates. What are specific no low flash fix interest loans that could be used to pay the variable loans off or transfer them? So what they're looking for something is called a debt consolidation. That's what they're trying to do. I'm going to ask, you know, to get a better understanding. They want to combine everything into one big major loan, right? It sounds about right to how do you navigate paying off these high interest loans into something it sounds about right to transfer it to a lower loan situation? Well, the only thing that popped into my head was to refinance them. Some of the SBA loans do offer refinancing. But do I actually have to go through the lender in order to get it all done? SBA does not actually do it. But if there are SBA loans, you can actually talk to the lender in regards to see if they can get those debts refinanced smaller or see if they can put them all together into one. But it's up to the lender. They need to go to the lender and actually talk to them and see what they can offer. Yeah, and I would completely agree with that that it's about making a relationship with your lender and talking to your lender. Unfortunately, a lot of the programs that we offer because our economic development programs are about creating new activity, not previous activity. A lot of times there are clauses, there are things that are written into the programs that don't allow for loans to be refinanced because they're not for paying taxes and arrears. They're not for refinancing things that were existing activity because the idea was to create new activity. Now, there are some programs that we have offered like the small business revolving loan fund that will allow a certain level of refinancing if there's a benefit to the borrower. But again, I would reach out to the lender because lenders have lots of different resources, especially CDFIs right now. There have been many different CDFI programs that go beyond just the couple that I've talked about. They may be able to help, but start the conversation with the lender. That's really the key. Don't be afraid to talk to a lender to say, can you help me get my payments down? I know I'm making my payments on time, but it's becoming a strain. Can you help? It just doesn't hurt to have that conversation. Thank you both. Can you remind us what a CDFI is? Again, I'll let you mention that they're mission-based lenders. What does that mean? CDFI stands for Community Development Financial Institution. They are typically going to be nonprofit lenders. They are smaller volume lenders that aren't traditional commercial banks. A lot of times they may actually get funding from the commercial market. They may go to a commercial bank to get a loan as a nonprofit to re-lend, but all the CDFIs that are part of our directory are mission-driven lenders that are designed to help the communities in which they're functioning. I'll give a good example of one that's up near me here. There's the Community Loan Fund of the Capital Region, which is a local community lender. It's a small CDFI that's here that can offer low-interest loans to small businesses around it that it's in its territory. In this case, just the capital district of New York, it wouldn't go down into the city. But how do they get their funding? They happen to get funding by people like us who go in and set up an account. Maybe I've set up an account and I've deposited some money into that bank. That CDFI re-lends that money makes it available to the people in my community. Each CDFI has got a little bit different structure. They do a little bit different thing, but that's really what a CDFI is all about. It's a non-commercial lender that can help various organizations with capital. There are various options out there beyond the typical commercial bank. In an industry, that's many times atypical in how it functions. It's helpful to know that there are other options out there. Do we have any other questions? I'm looking at the time. We're about five minutes out until the end of the hour. Are there any questions from our attendees? Are there any lasting thoughts for our panelists, Matthew, Lily, Robin? Any words of encouragement? Words of encouragement? I would say just keep doing what you're doing. Go to the banks, find out what they actually have to offer to you during this time, and see what's out there. Go to the lenders and have that relationship as we have stress throughout this whole panel. Go to your lenders, talk to them, communicate with them, see what they have to offer, and go to your resource partners. We have our website, sba.gov, slash local assistance, local hyphen assistance, so that you can actually find that resource in your area and use your partners. That's what they are there for. So good luck. Absolutely. I think that's great advice. And with that, we thank you, everyone, for attending today's webinar. We hope it was beneficial and provided you some direction as we hopefully navigate a not too long journey in this strike. So we're all aware of the situation. We want you to know as a city agency, federal, and state that there are resources available to you. So thank you for joining us, and please stay tuned. We do have another webinar coming up tomorrow, 1 p.m. Eastern Time. It is available on our MOMES website, a link for registration to attend. It is specifically targeting workers. It's part two of our series, Targeting Workers and Providing Resource for Workers during this time. Thank you again, and please stay tuned for more information, as well as an email, as a follow-up to the resources available and discussed in today's webinar. Thank you. Thank you. Bye, everyone. Have a great one.