Good afternoon everybody and welcome to this week in cloud native episode number six This week. We're gonna get a little bit more into API deprecation and removal We're gonna play with like, you know kind of work our way through sort of the blog posts that talks about it we're gonna evaluate some of the tools that are out there for For testing that we have we still have that problem and we're probably gonna build some kind clusters tear some kind clusters down that kind of thing so I'm glad you're here.  So welcome. Welcome to this week in cloud native number six Let's take a look at the news this week of which there is quite a lot So again, if you go here on the left hand side hack indeed.io at twicken if you go here You're gonna find a link to the latest notes and if you have anything you'd like to Share or link in there feel free to do so Hey, how's it going good to see you Glad you're here. So this week is number six We always start off these pot. We should be starting off these broadcasts with a roller-minder of the COC So this is a cloud native Foundation Video broadcast and so with any of those things it's important to remember the CNCF code of conduct So, please don't throw anything with a chat or questions that might be in a violation of that code Basically, please be respectful of your fellow participants and presenters registration for cube con cognitive con 2021 is open for in-person and virtual so definitely check it out. I'm gonna be there and I hope to see you there That'll be a lot of fun Cloud native TV has a bunch of different shows this week So this week some of the shows are coming up. We have cloud native Latin X and vivo on Tuesday We have cloud native live on Wednesday And we have fields tested capture the flag and Kubernetes and I'm actually really curious which one Casin is gonna take on there. I wonder if it's gonna be kctf or if there's some other thing that Should be playing with them. They'll be playing with on that on that episode. It'll be tremendous. I'm sure though So definitely check that one out So there's new content every day of the week Now the news of the week I actually was able to attend a cube a Meetup of the Atlanta, Georgia Kubernetes meetup and it was really great and I'm actually giving a shout out here to James Searcy Who is a good friend of mine works at T-Mobile and we work together for quite a bit I was really impressed with how Joe put the news together for that And so I've copied a lot of what he's done there and put them in here and figured I cover them here as well So as you already know API removals are happening in version 122 definitely check that out Hello, and welcome back Joe and Hello to you guy with a cube. Wait, you have a cube like this kind of cube Or some other kind of cube I'm really curious So some of the stuff that happened this week or is it getting ready to happen link or do you got it up? Oh, they have graduated from the sandbox to the CNC of sandbox. That's pretty exciting the contributor summit North America 2021 has finding has begun if you have actually been there definitely come check it out if you haven't This is what a great opportunity. I can't wait to see you there. It'll be really a lot of fun So if you're curious about the event, here's the information for it It'll be at the JW Marriott LA live Here's where you can register the location and the schedule all of that information is posted I mean as soon as it gets updated. I don't think the registration is there yet either but Yeah, no as soon as that registration link is available you will be able to find it here. Oh That's awesome I've actually on a personal note I've actually just recently decided to start exploring cubes because when I was a kid I tried it and I was not very good at it And uh, like the only way I could really solve these puzzle these twisty These twisty puzzles with a butter knife and lately I've been totally obsessed with like solving all of the twisty puzzles So now I have at my disposal like a two of four a couple of threes four a seven and this A 10 and it's it's super fun.  So Pretty cool Hello Russ. Good to see you Welcome welcome So another thing that I don't know if you've uh, if you've ever contributed to kubernetes as a project You've probably interacted with phasia bot and phasia Phasia bot has been pretty amazing and there, you know, it's got some pretty fun stories like this one Or basically basically somebody opened an issue to make sure that phasia bot knew admit that it was a bot But what the bot does is basically Mark issues as stale if there's no activity on them after a time and it will also take care of a lot of A lot of other kind of maintenance and housekeeping stuff But The awesome part is that it's being replaced With an official bot. So it's being retired All of the automated comments from really made by phasia bot are going to be made by the k The kubernetes triage bot as of this pr the k kubernetes triage bot account is fully under the project's own control Owned by sick contrabex github management sub project It means basically that like it's another one of those pieces of infrastructure That is now actually managed by the community rather than by an individual which is really pretty great It's a pretty important bot and it handles things like life cycle and a bunch of other stuff So it was pretty important that we that we get that pulled into the kind of under the auspices of the of the project itself So you'll still see those same warnings and stuff. It'll just happen a different way There is a proposal. I don't know if you've looked into the sidecar stuff But within kubernetes, there's a proposal to support the idea of sidecars now I know that this is kind of an overloaded term In that when you think about a kubernetes pod and you have Multiple pods inside of that pod or multiple containers running in the same pod You think of all of them as sidecars of one another effectively, right? Um, what this proposal does is it describes a model around which we might be able to describe that Individually that some of those containers might need to start before other containers right now It's sort of just a list of containers all which will start somewhat Uh in an uncontrolled rate, right and this is not taking into account in it containers That's not what i'm talking about. This is actually just within the construct of the the containers list Um, or I guess it's an array really within the within the pod itself Uh giving some giving some uh capability there to kind of control it Some of the problems that are trying to be solved there are like if you had a log forwarder running as a second process Within a pod you clearly want that log forwarder to become the last thing out Right because you want to make sure that it's able to forward all of the logs that it has Access to before actually shutting that log forwarder down So these these are just some of the uh relatively obvious use cases that People have and if you're interested in this they're looking for feedback And so if this proposal makes sense to you if you're happy with how this works Um, definitely give a thumbs up otherwise give some comments that To uh indicate like what you think might be an improvement If you're operating a helm repository This is it kind of a surprise one So if you're operating a helm repository repository credentials pass to alternate domain So there's like there's a hack that kind of lets people Let's people kind of get a hold of The credentials when in an unexpected way So while working on the helm source a helm core maintainer discovered a situation where a username and password credentials associated with a helm repository Could be passed on to another domain inference referenced by that helm repository In the index demo within the helm chart repository contains a reference where the chart archive for each project is So that means that if you're like say you took the dependency for engine x in your own chart When you do an a helm get or helm fetch or helm install or any of those things um of your own umbrella chart in your own private repository then What could happen is that your username and password could be passed To whatever repository is holding the helm the engine x chart as well Um, and this is you know, unexpected behavior. It's likely that they wouldn't do anything with it But it's definitely an important one to understand. So that was a security problem and that has been addressed Next up we have A pull request. This is a work in progress Introduce documentation around managing a separate mount namespace now. This is a fascinating idea And I had I had not heard of it until uh, joe had mentioned it in his in his announcement of this particular issue so um And I think I could see how there would be challenges. But anyway, so the proposal here is that You have the ability to define a mount namespace in which The ephemeral mounts that we create for pods would be associated right by default right now The mount namespace for all of the ephemeral stuff like, you know, if you're going to do like a amount of type of You're going to mount just a scratch space within a pod or what are any of or any of those sorts of things like empty deer Then empty deer is mounted on the underlying hosts in the hosts mountain namespace and then passed as a volume into Um, you're running container as part of the instantiation of that container when you kick off a pod or I should say when the cubit kicks off the pod In this model the idea is that anything that we would create ephemeraly We would actually associate with a different isolated mount namespace And maybe even share that mount namespace with like some other entity And or and then that mount namespace is actually where we're going to mount any volumes for your given pods in from What gives which is great because it improves the level of isolation between the pod and the underlying file system But I could see that it also might add a little bit of complexity. So it's an interesting idea I haven't actually played with it myself. Um But if any of you out there play with it, definitely i'm curious to get your opinion on it Definitely a cool read Improves the security isolation boundaries, which I think is good kubernetes release cadence change Is the blog as of july 20th um So starting with the 122 release a lightweight policy will drive the creation of each release schedule The policy includes the first release And the last release of the calendar year And the kubernetes release cycle has has a length of 15 weeks So the week of kubecon cloud native con is not considered a working week for the sig release mainly because like obviously everybody's at kubecon or cloud native con Should be the weeks of actually because there's a couple of them, right? There's eu and the us So might be interesting to see if that was taken into account as a little kubernetes will follow will follow a three releases per year cadence kubernetes 1.23 will find will be The final release of the 2021 calendar year and the new policy results in a very predictable release schedule allowing us to forecast upcoming release dates There you go. That's our new release cadence This means that there will be fewer releases per year Um, it doesn't really do anything with lcs or anything like that, but it does mean that there will be fewer releases And it means perhaps it'll be a little bit easier to adopt and and pick up the latest releases for kubernetes as things go Yeah, it is a great question. Russ. I'm not sure what what happens with uh with the old with the old phage about as it goes out to pastor Be kind of a fun fun idea The sysbox container runtime. I was playing with this actually this week. So Uh, if you're unfamiliar with this idea the idea is that you want to be able to run, um, docker containers that have system d or Or or running inside of them make them look like they're actually a little closer to vms now like, you know I could definitely hear that on some level. This is a weird thing to want right because you're you're operating a container Why would you want it to look like a vm? Uh, you should be able to like, you know, keep with it You should live with you've been able to live within your means within a container not necessarily Try to run like all of the linux operating system stack inside of it Well, one of the great use cases for this would be something like kind right wherein you could have um Or you could run your kubernetes nodes as containers Inside of a inside of a cluster and so you're able to do a lot of testing Um in that scenario lightweight testing you'll be able to spin these things up and tear them down Effectively the same rate as containers themselves, but not necessarily Um without the cost of virtualization So it also enables a lot more kind of like it's a great learning tool You know kubernetes and in docker are kind gives you a great ability to kind of like play with all the different knobs and dials of cuba dm and that sort of stuff Well sysbox is another one of these and there are a few that i've been playing with lately Footloose is another one of these sysbox is one And and this tooling basically sysbox is actually sysbox is pretty low level it had and happened it has a Um a run c driver for it. So you can actually plug it into your existing docker and i've done that Play with it and see what it would look like This is basically what you would add to provide another run time for docker by default docker uses container d as the run time But if you wanted to add another run time you could add one like this And what this does Is it gives you another command so you can do docker run actually history correct sysbox That gives you the ability to do something like this Or you can run a specific container Under that other run time In our case i'm going to use the docker run time. I'm going to use the sysbox run c run time I'm going to remove it when it's done I'm going to call it my container and I'm going to pull from the image from the nesty box registry And it's just the ubuntu bionic system d docker And we can see this thing kind of starts up like a container It looks and feels very much like a container not too dissimilar from the way that if you were to do this with um Kind it would work as well. Now. What's interesting is also the The mechanism this this particular container image has pre-installed bits of docker, right? So I can do docker ps Didn't find the image locally So now it's pulled it then i'm inside the container Inside of another container Kind of like docker and docker in some ways, right? That's that is basically how it's working. And so this gives you kind of a more generic way of handling system d pieces one of the other pieces that's nest that Sysbox does is it implements a user ns. So if you don't already have User ns it won't work for you But there are some there were some interesting challenges there like I tried to run this on An arch box that had like the latest kernel and it was not working for me at all I had to drop back tomorrow an lts release because apparently This works really only really well in kind of like the older versions because of the shift to fest requirement And so if you want to play with it, it's here. It's a fun one It seems to work pretty well. I've had I've had good luck with it One of the challenges I had previously was I was trying to use ansible to I was trying to use kubespray to install A kubernetes cluster and I wanted like docker containers to do that because I didn't want to Go about managing all the things now one thing I learned was that in ansible's kubespray project or in the kubespray project If a hostname isn't already set correctly then a kubespray tries to set it and the way that it does that is through this command I'm as root right now if I do hostname kettle hostname foo And that fails and it gives me the output could not set property failed to set static hostname device or resource busy and that's wacky because like even if I would assume that that would be possible within the container right because I can still do something like echo foo hostname And then log in again and boom is foo right so I can still change the hostname that way But something in the way that hostname kettle does it like makes use of something that I don't understand yet and and blocks it So at some point I might as trace that and see if I could figure it out, but kind of an interesting challenge So this in my I guess all that to say it didn't solve my problem Which was a very obscure problem to begin with and that was the problem that I was running into But it is neat so definitely check it out if you're interested in VM like containers Argo vulnerability leads to crypto mining if you're using Argo definitely check out the vulnerability The cncf white paper. I haven't had a take a chance to look at this But this is a a new tag for app delivery and they're talking about Um the white paper the the final version here. So this is actually Kind of I think a pretty solid write up that was funded by the cncf To talk about the operator pattern how it works and all of that stuff So if you've heard people talking about operators and you want to know more about it I think this is probably be a really good reference to begin with Um, so definitely check that one out. I like that. It's somewhat agnostic It talks about the different frameworks that are out there qbuilder cop cncf operator framework the metadata controller Um talks about life cycle management and use cases for an operator It talks about a bunch of different um cool stuff and does it does a pretty good job Yeah, so A great reference on operators There's a new admission control micro framework and then there's also the crosslet, which is a It's been moving along pretty well, but it's a cubelet rust and awesome to give you the ability to run like Web assembly as uh containers instead of containers as containers pretty neat stuff Nothing new in the cve ground and the next thing I wanted to start playing with was I wanted to kind of explore that Take us back to that blog post About api deprecation just in case anybody had not already seen it Here we are So this article was written by christina calari and tim banister And it talks about the api removals in version 1.22 So when we get to version 1.22, which has already been cut If you start migrating to it one of the heads ups here is that you're going to start seeing um Things get taken away. Uh, you're going to see the api not present for a particular group So let's talk through like which ones are going to be removed and I we covered a little bit of this in the last episode If you want to check that out. It's on youtube But the thing that I wanted to point out here is like, you know, for example validating webhook configuration mutating webhook configuration originally was V1 beta 1 and now it is just v1 So and and the removal means that if you still have your object defined that manifest defined as api group Admission registration dot k. So if you i o slash v1 beta 1 then it's it's going to fail and it's going to tell you There is no object of that URL right and so that's Going to be the experience that you have and you'll be surprised by it And there are a few other ones here custom resource definition That's a big one So if you haven't if you have if you're not updating those custom resource definitions that you've created Then you're going to get caught out by that It's a great example of custom resource definition in here somewhere In our docs and it makes me wonder if It's good. All right So here's an example of a of a crd that has been defined And I'm going to actually go I'm going to go ahead and apply this. I'm going to go up here to raw I'm going to grab that URL and we're going to go ahead and apply it and see what it looks like. So let's do Kind create cluster And while we're doing that. Oh, that's going to bring him up 122. That's not what I want. Uh Do I have a 122? Let's go and take a look Kindest It looks like to test this we would have to build 122 unless I've already done that which I might have done but All we're waiting for that. Let's go ahead and do this kind complete cluster So I'm going to go ahead and build kubernetes real quick. I know that sounds kind of weird, but we're going to do it service kubernetes And I've got it checked out locally and the way that I checked that out This is something I learned or kind of relearned recently was that You have the go 111 module command and if you set that to off Then you can do a go get k8s.io kubernetes And it will put it in go search k8s.io kubernetes for you If you have modules on then it actually just checks out whatever the release version is and props it into your modules So you can reference it But from my case, I kind of wanted to have it just checked out locally so I could play with it Make polar quests and that sort of stuff So here we are we're in go source k8s.io kubernetes. I've done the check out already and I'm going to go ahead and change to a release branch v1.22 We'll call it rc0. I'm going to do get check out dash b That's the first release candidate for it and then I'm going to go ahead and do kind build node image Name equals v1 Kindest to zero rc zero Unname its image And because my local checkout in my go in my go environment is set to v1.22 zero rc0 That's where I that's what I got checked out locally then kind will actually build that particular version And make it available to us and we'll see if this works. I might have to actually grab the the The current release or the top of tree release for kind to make it work But let's see what let's see what happens if we bundle it up this way first and maybe we won't have to do too much more But this will give us the ability to go ahead and test out those expiring apis and see what that looks like So while that's happening, let's come back over here So these are the things that are being affected. So any automation that you have that Does a token review you might want to check a look at anything with subject access review or local subject access review Self-subject access review anything you're doing that's actually checking the credentials Or any testing that you do for any of those things any of those objects have to be defined in that way The beta certificate signing requests is now no longer beta and it's not going to be available The lease api if you use it And the ingress object right ingress extensions v1 beta 1 And networking kh.io v1 beta 1 this one has been around for quite a long time And it will be removed from serving It means it will not be available And if you were to try and create an object with that old version, it will not be available Now there's a couple a couple things that I covered last time that i'll just read i'll reiterate here real quick And that is if you ever ever wondering what version is the right version right you can do kubectl explain For that particular object. Let's take an ingress for example Have a cluster up so kind create cluster Am equals 1.21 kubectl explain ingress And kubectl explain gives you lots of great information including Right up here at the top What the target version should be And what the kind should be Right and if you wanted to know more about the spec if you wanted to kind of explore that spec somewhat dynamically the entire spec is defined here right and so you could do things like kubectl explain ingress recursive And it will give you all of the fields that you could possibly define within that object And if you wanted to know what status load balancer ingress was going to be right you could do kubectl ingress spec And here are the fields that are viable for that particular object right and so you can dig into a particular object Lower and lower you can look at the spec And at any point you can do recursive and see like the entire Construct of the spec all right there very easy to kind of navigate and troubleshoot and see what's happening here So this is one way of understanding like if I wanted to do like kubectl explain self subject Access review Right So it is under authorization k it's i o v one That is the correct version and the kind would be self subject access review And again, here's all the information for the object That's one way of determining the group another great way to look at it is this like kubectl api resources so Actually Ingress oh in this particular output And this is a tricky one because this will actually show all of the api Resources that are being served currently like what things you can define And in this case you can see that you could define Uh ingress under networking case i o slash v one or you could also Define an ingress under extensions v one beta one and that's because we're running version one dot 21 We're running 122. We would not be able to do that. It's like we're almost through our build here Lots of good cpu time. Oh look Let's slow it a little bit Hope y'all can hear me. Okay Shouldn't be too long So that's another one and then the last one which is also useful is kubectl api versions So we can tell that like, you know for particular groups what versions are available So networking that case i o has networking v one and networking case to i o slash v one beta one Let's just build for a second and then Jump back into our docs so But one of the things I really want to make sure that we highlight is that like removal means removal It means it will no longer be served. It means that if you try to create that object, it will no longer be there, right? um And we're going to play with that just in a minute when I get 122 up We're going to start up a 122 cluster and like deploy some stuff that is that that don't work anymore So for ingress migrate to use the newer api. This is going to be true of pretty much everything um The related api ingress class is designed to complement the ingress concept allowing you to configure multiple kinds of ingress within one cluster If you're currently using the deprecated ingress.clask annotations plan to switch to ingress class name field instead And I believe that was actually handling being handled kind of somewhat automatically when migration would happen That's great information here for each of the different kinds of things that are being torn down There's a plugin to kubectl that provides the kubectl convert sub command It's an official plugin that you can download as part of kubernetes. So you download kubectl for details I was not aware of this plugin. This used to be a function that um, that's kind of a bad link I want to know more about convert So this used to be a functionality of kubectl But i'm really curious about the plugin. So let's let's actually go looking for that I mean Crew might have it I hope maybe it's already there I'm missing something super obvious So they reference Sting so they reference it, but it doesn't look like it's there No, I don't have it So in theory, I should be able to grab it from here if it exists So let's do that. Let's let's try that out kubectl Version I'm currently running 121.3 It would be held in someplace different than this. I don't think It's a file I'll be damned All right, that's cool So they basically moved the functionality of convert into a plugin and they made the plugin available, but they didn't necessarily It doesn't look like it's packaged it packaged together yet Nor have they registered that with crew That's really cool I mean it's cool that it exists the packaging could use a little work probably should update this to uh include the kubectl convert piece But the way plugins work If plugins work is anything that has the word kubectl and then a dash and then something after it That's actually how the plugin trick will work, right? So kubectl would discover that and make those plugins available to you. That's pretty neat Let's take a look at our build here Okay, that's done. So I'm going to do kind create cluster config actually image equals v one zero What did I call it? Oh, this may or may not work. Let's see if it works. It might work. It'd be awesome if it did Hey kubectl version 122 rc0 Woohoo. All right So now if I go back over here And I do kubectl create ingress kubectl create Oh, you know what? I'm on the wrong one. So I have to um deployment test Which equals and next Replicas equals one Suppose port equals 80 Context 21 I'll send you expose deployment dash test Equal 21 I'll create a service of type cluster ip for that guy and then I'm going to do kubectl Create ingress all back end equals default test Context equals 21 Now here's the fun part. You can actually do uh, this is what I wanted to show you You can actually pull the previous version of an object. So if I do kubectl get ingress test Context equals 1.21 Yemo So there is the the current Valid object and I could actually create the same object but by copying it to our new context But I'm not going to do that right away Instead what I want to do is I want to pull an older version of it. I think I can still do here Ingress This is kind of wild but check it out. So this time I'm actually going to use the old extensions the one that's actually being reviewed from the cluster so if I do that I can see that I'm doing kubectl get ingress's extension. So I'm telling it that I wanted to convert Whatever object it has an ecd into this particular version of object so that I can see the results of that object, right? And it's going ahead and it's gone ahead and done that. It's created extensions v1 beta one It's a type ingress and here's the content that I had Back end was service name test and the service port was 80 And if I get if I just go back to just get ingress You can see the difference here, right? So create timestamp is the same generation is the same name is the same resource version is the same The configuration looks a little bit different Before it was back end service name service port now it's default back end service name service port Okay So let's grab that old one and then we're going to just pipe that to our 122 cluster and see what What happens here? We had two really interesting outcomes The first really interesting outcome is that kubectl itself on evaluating The object that I was getting ready to present to our 1.22 cluster Told me hey warning This is deprecated. You probably want to move to networking k8s.io slash v1 That is awesome The second one and more interesting and more relevant to this particular testing Is that when I applied this object to the 1.22 cluster? I saw this error come back unable to recognize No matches for kind ingress in version extensions v1 beta one So this is the area you're you're going to want to watch for Yeah, the thing doing the conversion is the api server. Correct. Yep But this is the error that you're going to kind of want to watch for If you see if you're you'll know that you're hitting this problem when you see this error No matches for kind is the key is the thing right? That's actually where you're going to where it's going to catch you out The next thing I wanted to show you I wanted to do is just grab another example So hold on one second here It's response body That's everything All right So now I'm digging to the the bones here a little bit and show you some other interesting stuff That's happening behind the cover So this is like probably the easiest way I can think of to do a test on like self-subject access review So first understanding what self-subject access review is One way to conceptualize this Is to is to is to basically it's the idea that you can query the api server Uh with specific questions to determine what access you have Right, so if I were to do kubectl auth And I one of my favorite commands actually Uh, and then I do list for example But it will show me all of the permissions that I have With my current credential Against the api server According to the api server itself and the way that it does this is through this self-subject access review And actually in this particular case it has to do with a self-subject access the self-subject rules review Because I'm having it list all of my permissions For this given namespace As an authenticated user Now let's say I create a new uh a new user, right? So let's do kubectl Actually, let's do this. Let's do kubectl auth Can I list And we'll just use the default service account in the default namespace and take a look at that one, right? As I'm going to impersonate system service account default And that's the default service account in the default namespace And we've identified it as a system service account. Now what's going to happen Is it's going to run this command the self-subject access review? But it's going to impersonate This particular service account and get the result back So my question was what permissions does the default service account have We can see that they're very different than the permissions that we have as an admin, right? Like I basically have like cluster admin role right now So I have all of the permissions for all of the things But the default service account does not It can do a create of self-subject access reviews It can do a create of self-subject rules review And then it has the ability to understand a little bit about the configuration of the cluster So it can basically walk the api and see what resources are there, but that's about it Right interesting stuff Now what I was showing you before was if I wanted to see for example, what the api call Was that made this api that made this request I can actually pull that open and take a look right And so here is the request body And you'll notice that it looks a lot like the json stuff or the or the yaml stuff that we normally see So it's a self-subject access review api version Authorization k8.io slash v1 And then there's no the metadata is effectively empty The spec is defining the namespace in which I want to spot to see And and then a status object exists, but doesn't matter And so here is the curl request if I were going to use curl to do this And we can see now one of the biggest takeaways of this particular piece of it is that you can see kind of how The api removal Will affect you Right if we are looking at the api object the group Is defined right after The group and the version are defined right after this well-known path slash apis So inside of my yaml document, right? That's that's actually let's take a look at that ingress again that we were looking at before Inside of here. I'm saying api version networking k8.io slash v1 And I'm saying kind ingress right And if I just apply that kubekettle apply Oh bunch of data, but actually you know what would be easier There we go So here's the curl making the call and it's saying it's going to put that according to the document kubekettle has converted that Based on the yaml that I have provided It has picked up the group and it has applied it in that uri So it's going to send it to my api server On slash apis inside of the networking k8.io on version v1 It's a namespaced object. So it determines what namespace i've targeted the default namespace by default And then the ingress create the ingress object i've created is called tests right So that is effectively how that conversion happens and if that api gets removed you won't be able to see it if you send The 121 version of the output of the ingress to the kubekettle convert plugin it will convert it to 122 And it should deploy correctly in the 120. Yeah, that's right. Let's try that. It's a great example. So the question Is if you send the 121 version Of the object to kubekettle convert Well, it converted is interesting I expected that to go a different way honestly, I did Because what's fascinating about that is that it it came out As incorrect as it went in Which is really neat. I know that you can pass versions And so I wonder if I have to be explicit about the version or maybe Actually, what if I do output version No, I shouldn't have to do that You know what will do the conversion? This is kind of an interesting one. So before we get into it here, let's do kubekettle apply dash f dash If I apply it If I apply it to a cluster that has both versions Then the output will be converted I know and we Ah, it still took it. So what I want to see Look at this one more time it might be a way to get it to That's neat. I mean Oh, I see I see I got it now. Hold on I pulled the plug in That's cool. Okay, hold on see, um The history grip convert Yeah Although that's still pretty busted But let's grab the 122 version of this and try it that way Still fails Interesting Yeah, I mean I could do the output version thing But the problem is that you would have to know what the output version was, right? And the idea is that this would actually automatically convert this for you networking dot kxio slash V would be one out of curiosity If you know it it will actually convert it for you correctly But if you don't know it then that one still works Yeah, that's kind of a trip. That's a very good point Because otherwise you wouldn't you wouldn't really know. Yeah Anyway, that's fascinating. So thank you for pointing that out. That was a great question I do think that it would be good. I wonder if there's like some other piece of kubectl convert that I'm missing If valid use cumulative validates I want to look it for output version basically Should be like a kind of a latest or something. It does not appear to be I think the old kubectl convert did do that the old kubectl convert did actually Expect and it basically comes down to discovering what the preferred version is and then using that preferred version So I wonder if they've wired it up to a new api that doesn't exist yet because there is now There will soon be a uh an api that describes the like oh actually in my kubectl Yeah api resources No, it doesn't have it yet But there is a new api behind like a beta piece that I read about last week That gives you the ability to kind of define The preferred version in an api and so I wonder if if that existed whether kubectl convert would be able to consume it and make the right decision But it used to be that kubectl convert would determine what the preferred version was And then it would pick that And that's not what's happening here instead. What it's doing is just I don't get what the convert part is it's actually just taking the object and Dropping it in the same way that it went in they're dropping it out the same way it went in So it seems like a bug on the On the plugin part anyway So last few things I wanted to cover before I bounce out of here are there are a few other things That are worth calling out. Um, and these are different projects that I found That give you the ability to understand whether things are deprecated or falling out of uh, or or have expired Uh, thank you So cube no trouble is a great example of this. It basically looks for um objects that have been created and then Uh on a deprecated version and warns you about them Now this is a neat one because it actually looks at The thing it looks like home charts. It looks at things that metadata that's been left behind, right? So like a few for example were to do kubectl get ingress test Ah, it didn't work here, but um, sometimes it'll actually leave behind a metadata for um, actually Let's see what if I just do get deployment, but from time to time When you're deploying things it will actually keep a meta it'll keep a metadata record for what was what was the deployed configuration and things like q-bend can actually look at that Metadata and determine that When the object was applied to the cluster it was stored in an old version Some of the other tools out there are pluto by the wonderful folks at fairwind ops It does a very similar thing you can apply it You can actually point this at your cluster or you can point it at your source code And it will try to evaluate whether those things are expired or not cube pug The third one written by riccardo cats community member might even be on the call today Gives another way of actually just exploring this as a crew plugin And then the last one is deprecation written by a good friend steve wade And he is doing exactly the same thing just like evaluating Those manifests that you've provided and trying to determine whether those manifests are using expired apis And that is our session for today. I wanted to say thank you very very much for joining me It really means a lot. Um, I hope these sessions are useful and I look forward to the next one in two weeks And uh come right back here and I'll I'll meet you again in two weeks And we'll cover some other interesting fascinating part of all of this So I hope you all have a great week and I'll see you soon If you like what you see shout out on twitter follow me on my following Follow me at mowye line anywhere and also subscribe to the channel Talk to y'all later. Bye. Alrighty


So we've seen this dramatic rise in fragmentation of code and of complexity, which has driven complexity. And developers today are just literally overwhelmed. What we see, and part of this comes through the platform engineering movement, where people want to consolidate things down into a platform engineering scope. But on top of that, they have to have a set of APIs that are consistent across their application developers. So what we see is this increasing decision of how do I have APIs that my developers have in the same way that Kubernetes brought a set of APIs for hosting and everyone could agree on for the hosting level. What are the APIs that everyone could agree about their application level? So if we can reduce the fragmentation of large numbers of frameworks, but we can agree on a set of APIs around this all, all of a sudden developers from different languages and different frameworks can coordinate together. Hi, this is your host Apil Bhartia and we are here at KubeCon in Chicago and today we have with us once again Mark Fussell, CEO and founder of Diagrid. Mark, it's great to have you on the show again. Thank you, it's wonderful to be here today. Yeah, it's my pleasure to host you again. First of all, we are here at KubeCon. It's almost the end of the day. Everybody is in tired, exhausted. They are waiting for beer. How has been the event so far? It's been fabulous actually so far and talking to many people. For me, I think one of the highlights of the event was App DeveloperCon, which was the first time this co-located event was being held. And App DeveloperCon was something that we initiated with CNCF to get application developers talking about what gets built on top of all this infrastructure. So we had a huge turnover in the audience. The room was packed with 300 people. We talked about designing architectures, developer tools, serverless systems, this sort of thing. From the context of Kubernetes, how much room is there for developers? Well, exactly this. I mean, we come to an event like this. So if you look at the CNCF numbers, 3,000 developers turn up here asking to think about building their business applications. Now they have to understand Kubernetes. They have to understand all those things on top of all. But that's like the baseline for them. They're the ones who generate the money for the company. And so we need to have conversations about application architecture, event-driven applications, how we build workflows. What are all the tools and application designs that the business needs on top of all this? So that's what they have a conversation here for. They need to understand these ops technologies. But at the same time, they want to see things themselves about the applications that get delivered on top of them. For a while, we almost forgot about developers and app developers. Everybody was talking about DevOps, SRAs, platform engineers, DevSecOps. But as you rightly said, developers are the ones that are making the money. Every team in a waste, but that's where the real business value is coming from. With the whole shift-left movement, a lot of things are moving in developers' pipelines also. We talk about that. What does the developer experience look like today? Well, I think the developer experience today is very fragmented. Developers have to look across a whole industry of frameworks and libraries and languages. And the tech sprawler they have to deal with is very, very, very difficult. So you get a lot of this tech fragmentation. I think developers have to also deal with quite a bit of lock-in, where they're locked into particular technologies on a particular cloud, and they want the portability in the open source side of things. So I think you see developers wanting to break the challenges of a lock-in to a particular framework or a particular cloud provider. They want to prevent technology sprawl, and they want to be productive in what they build. So I think there's a lot of movement towards consolidation of frameworks, how they can easily build applications, and that's where DAPA comes in in many ways, which is an open source project. I work on as a maintainer that reduces the amount of code you have to write boilerplate code and makes developers productive building applications without them having to reinvent the wheel or, as I like to say, reinvent the pattern. DAPA and then Diagrid, you know, open source can go only so far. That's the nature of open source because you have to get it to a larger ecosystem, you know, not every, and that's where the commercialization comes at play because then you can fine-tune things for the customers to talk about DAPA and Diagrid. Yeah, so DAPA, which is open source projects, has had a tremendous growth over the last year. We're up to 3,000 contributors inside it all. It's the 10th largest project inside CNCF and we've had big organization from financial organizations, manufacturing, healthcare, all building on top of it all. In fact, we launched a brand new website for that project just a couple of days ago and, you know, it's seen huge growth in helping developers build long running, distributed applications. One of the most exciting things inside DAPA, the project is a workflow engine so you can combine workflow along with a vendor of an applications. But what we also see is that a lot of those developers who are deploying DAPA on top of Kubernetes need some help and they need help with how DAPA gets deployed and upgraded, best practices, they want better security. So that's where we Diagrid come in. We provide a service called Diagrid Conducted that helps to manage on Kubernetes. But we're doing more than that as well. Since we are here at KubeCon, did you folks make an announcement here? Anything, any launches? Yes, we did a launch here at KubeCon. We were excited to launch a new service called Diagrid Catalyst and think of it as hosted DAPA services in the cloud. It's DAPA paths in the cloud framework where you can build your application on any form of compute infrastructure and be able to build applications that communicate across compute, across cloud and really it's about democratizing, building these large scale distributed applications using open frameworks like DAPA but hosting them as an API. And we're pretty excited by this because it means now that any form of developer, whatever language you come from or whatever compute you want to run, it fits with where developers want to run things today. So let me give an example. We've been working with customers who have legacy applications that need to send event messages to new applications. They'll be able to do that through our Diagrid Catalyst by publishing and subscribing messages between two completely disparate apps running on two different platforms based around open APIs and open standards. Earlier we were talking about the whole developer experience. When we look at Kubernetes, we do talk about complexity a lot. How is Diagrid kind of working as a buffer to kind of, you know, up secure that complexity for developers. So they do not waste a lot of their time in all those things and continue to focus. As you said, they are the one who drive the business. Exactly. Yeah. I mean, this is what we've seen. So we've seen this dramatic rise in fragmentation of code and of complexity which has driven complexity. And developers today are just literally overwhelmed. What we see and part of this comes through the platform engineering movement where people want to consolidate things down into a platform engineering scope. But on top of that, they have to have a set of APIs that are consistent across their application developers. So what we see is this increasing decision of how do I have APIs that my developers have in the same way that Kubernetes brought a set of APIs for hosting and everyone could agree on for the hosting level. What are the APIs that everyone could agree about their application level? So if we can reduce the fragmentation of large numbers of frameworks but we can agree on a set of APIs around this all, all of a sudden developers from different languages and different frameworks can coordinate together. So that's what we're trying to do at Diagram. How is it we can reduce the boilerplate code, reduce the amount of patterns that you have to reinvent, constantly reinvent the wheel. And we do that because DAPRA's APIs that we host in our catalyst service in order for you to take away the grudgery of building code that spans across different teams and across different frameworks and different platforms. And we see a huge movement that was going to take, it's a democratization of building applications at scale. When we're talking about democratization, when we're talking about making life easier for developers, automation comes at play. And the next step of automation is generative AI, right? Everybody here, there are a lot of companies who are now offering products based on generative AI. So from Diagram's perspective, how do you look at gen AI? We don't have a lot of generative AI today but what we'll see emerging is that generative API will be used a lot in applications all the time now. We're seeing that people want to bring in different applications at genative AI. In fact, the general trend that we see is that people try to chat GPT first and then that's a choice that we use. Every app developer is looking at... Everyone is like that, yeah. And then what they're then going to do is they then try and move and test out on a few other models. So I think that you'll probably see us think about how we have a general purpose API that allows you to test and run lots of different models to optimize things against. And I think that's going to happen a lot where developers will be using generative AI in our applications but need to test on different models in some way. And I think we in Diagram will explore that space quite a bit of crossing the boundaries of developers building traditional applications but bringing generative AI in, whether it's in testing a model for some medical application or because they might want to generate some specific code inside their app. So I think there's a lot of exciting things that can happen here. Mark, thank you so much for sitting down with me and of course talk about the importance of app developer and also thanks for organizing this conference. Thanks for all those insights and I would love to chat with you again. Thank you. Thanks for having me. It's been wonderful to be here.


All right, welcome back after our short break. And we are going to continue with our review of house bills and we have H430, which we received, I don't know, a couple of days ago. So Representative Black is here as a reporter of the bill. Welcome. It's good to have you here. I think it's the first time that we've had you in the testify in our committee. I think so. Thank you, Madam Chair and senators. So for the record, Representative Alyssa Black, house health care. So H430, which is expansion of eligibility of Dr. Dinosaur to all pregnant women and income eligible children in the state of Vermont who reside in Vermont, regardless of their immigration status. So this proposal was brought to us by the office of health care advocate.  Not knowing what it was going to lead to. You know, just sort of researching the topic. I think it wasn't something that had ever been on my radar before. I'm not sure it's really, you know, it's sort of that out of sight, out of mind thing, but meeting with various stakeholders and just learning about this really, really small population that we're talking about who really have no viable access to any sort of health care, except for frankly the most expensive health care that we have, you know, talking to open door clinics and the free and referral clinics, which is sort of the clinics that serve this population, they don't serve children, you know, they're kind of at the mercy of the volunteer physicians. So if that physician's specialty doesn't allow pediatrics, then they have nowhere to go. They don't do obstetrics. So, you know, these children and parents are left with essentially bringing children to the emergency department. And then, I mean, they don't have any ability to pay this. One emergency room visit is going to be thousands of dollars. Some of the patient assistance programs through the hospitals do not even extend that coverage to someone with a status who's undocumented, which means they can't even make payments on it. Anyways, so this is a really, really simple bill. It just extends this coverage, you know, it solves the issue of simplifying it. There's this whole other kind of category. I'm sorry, there's really two categories of people we're talking about here. We're talking about our farm workers and we're talking about this sort of, these immigration statuses where you're, you know, you're applying for status, but you're sort of in this limbo, you've yet to apply, you have not been approved for it. So you're not covered during this limbo status. I'm sorry, where was I going with this? Oh, I think you were- I know clearing up confusion because there are people who are eligible who get denied. And it just clears up the confusion. Everybody's eligible regardless. All right, this is very helpful. Thank you for sharing your journey. In supporting the bill and we will listen to folks and see where the problems lie, but also where the benefits lie because I think there's probably both embedded in the bill and we appreciate you're taking the time to present it to us. I do have a question. One is the vote in your committee and in the house overall. So the vote in the committee was 10-0-1. And it was a voice vote in the house. Okay. On a voice vote, I think there were two nos. Wow, okay. So you did a good job reporting the bill. Yeah, I'm sure it was all because of me. The other question I have for you, Representative Black, is did the bill, then the bill must have gone to appropriations because there is a fairly hefty amount in there. It did go to appropriations. So the thought behind the bill was, I think that we really wanted to get this going for fiscal year 21, I mean, 22. And Diva has some really important upcoming challenges and we wanted to be able to give them some time to get this in place. So there is an appropriation. The appropriation is, and this is too for the agency of human services to stand this program up this year while Diva is able to take the time to get this in place. And the appropriation is based on the numbers that Department of Vermont Health Access anticipates that it might cost to serve this population. And additionally, some additional money for outreach, grants to the various groups that serve these communities in order to be able to do outreach to these communities that they are now eligible. Okay, thank you. That's very helpful. So you have a right-hand person here, Representative Houghton is here. Did you wanna add anything, Representative Houghton? Hi, thank you again, Representative Houghton House Health Care Committee. Nothing to add except I do believe the House appropriation vote was 11. Zero, is that correct? Yeah, 11, zero there. Okay, yes, thank you. That's helpful. So it sounds like it was a fairly unanimous vote all the way around with maybe two exceptions somewhere. Okay, why don't we, unless there are questions for Representative Black, okay, we'll move on to go through the bill and Jen is here and as well as Nolan, who would have the fiscal note for us and then we'll move on to hear from Nissa James, Corey Gustafson of DEVA, and Mike Fisher, a healthcare advocate. So that'll be the order of the testimony. So Jen, thank you for being here. Sure, I will put up the bill. This is H430. And as you heard, it's an act relating to expanding eligibility for Dr. Dinosaur to all income-eligible children and pregnant individuals, regardless of immigration status. It would add a new section in Title 33 in Chapter 19, which is the Medicaid and some other medical assistance provisions that I've titled Dr. Dinosaur Coverage for Undocumented Immigrants. It would direct the Agency of Human Services, which is the way we phrase a lot of our Medicaid-covered provisions, not specifically DEVA, but the Agency of Human Services shall provide coverage under the Dr. Dinosaur Program to children and pregnant individuals who are undocumented immigrants, but who would otherwise be eligible for medical assistance from the state under a particular provision of the Social Security Act. I will note, and I think you'll hear from DEVA that DEVA does have some concerns about this language and some potential unintended consequences of phrasing it this way, so they may have some proposals to make changes to the language, to achieve the same result, but without having it tied so specifically to the federal law. Section two would appropriate 1.4 million in one-time funds to the Agency of Human Services in fiscal year 2022 for the following purposes, grants or reimbursements or both, so prospective or retrospective payments to healthcare providers for delivering healthcare services during FY22 to children and pregnant individuals who are undocumented immigrants, grants to Vermont organizations that work with members of Vermont's undocumented immigrant community or with members of the healthcare provider community to provide outreach and information regarding opportunities for children and pregnant individuals in Vermont who are undocumented immigrants to access healthcare services at low or no cost in fiscal year 2022 and thereafter. So during this kind of transition year with the grants and reimbursements and then going forward under the Agency of Human Services program and also funds for implementing the technological and operational processes necessary for the Department of Vermont Health Access to administer the Doctor Dinosaur Expansion as set forth in section one, beginning on July 1st, 2022. Section three would require the Agency of Human Services to provide information on the estimated FY23 costs of expanding Doctor Dinosaur eligibility to undocumented immigrants under again that language in section one, beginning on July 1st, 2022. So that information on the estimated FY23 costs would be included as part of the Agency's FY23 budget presentation to the House Committee on Appropriations and House Committee on Healthcare to this committee and to the Senate Appropriations Committee. And then finally, we have the effective dates. So the funds, the 1.4 million in one time funds to AHS would take effect on July 1st of this year. The remaining sections would take effect on passage with the Agency of Human Services making coverage available to eligible undocumented immigrants under Doctor Dinosaur in accordance with that language in section one, beginning on July 1st, 2022, subject to FY23 appropriations for this purpose. And so some of that last piece is what was changed on the House side in the Appropriations Committee to reflect that this is not a commitment until the money is appropriated. And that is the language. Okay. So, and we're gonna hear from Diva's comments, specific comments related to their area of oversight. Questions for Jen. So go ahead, Senator Hardy. Unmuting can take time. Well, first of all, before Representative Black leaves, I just wanted to say thank you for jumping on this idea. I actually am a former Executive Director of the Open Door Clinic. That was the first job I held in Vermont. And I wish I had thought of this bill because there were so many times when we had to scramble to try to find healthcare coverage for pregnant women or their kids. And it was really frustrating. And sad. So I'm really happy that we're working on this. And I hope that we can find a way to make all the details work. So thank you for jumping on it. And Jen, my question for you is, and maybe this is also somewhat Nolan's question, but first of all, once, if a woman has an undocumented immigration status, she would qualify under this bill. But then once her baby is born, her baby's status is that the baby, if born in Vermont is a US citizen, and that baby would presumably be eligible for Dr. Dinosaur under sort of quote unquote regular provisions, is that correct? That's my understanding. And certainly the Diva folks can correct that if that's not accurate, but I think income eligible children who are US citizens are eligible for Dr. Dinosaur. And how long a woman whose income eligible becomes eligible for Dr. Dinosaur as soon as she becomes pregnant or is verified to be pregnant, I think. And then how long after the baby is born is the mom, does the mom continue to be eligible? Do you know? I think it's 60 days postpartum, although I think there may be some expansion of that under the new federal law, at least in the short term, or an option for states to extend that up to a year. Okay, right now I believe that's 60 days postpartum. 60 days, okay. And then the presumed, maybe this isn't a question for Nolan, but once we're through this first year of this transition with the $1.4 million appropriation, would the appropriation for this part of the program, if it moves forward, just be encompassed in the sort of larger Dr. Dinosaur budget, or would there continue to be a line item specifically for this purpose? I think for the record, Nolan and all the joint fiscal office, I think it says in the second year, they would add that to the budget. So I think that in 23, it would probably be an initiative specific in like, I'm thinking of envisioning a diva budget book, and it would be a line item where they show it. And then after the following year for that, it would be base. It would just be incorporated into the larger budget. Eventually, yes. Okay. Ray, I think it would be built into the base, but the important distinction here is that this would always be state-only dollars. We would not be getting federal match on this. So whether they would list that out separately because it's handled separately. I don't, Nolan knows the budget book better than I do, but I don't know if they would, in their minds, consider it Dr. Dinosaur, or if they would consider it state-only dollar coverage for this particular population. Yeah. The coverage would be the same. There wouldn't be a distinction between a woman who's qualified under this provision versus a woman who's qualified under the larger program, correct? In terms of what you get. From the patient side, right. From the patient side, my understanding, and that's why the language is drafted the way it is, although as I said, I think it may need to change, but my understanding of the intent is for the coverage for the patient to be the same. It's just the funding side of things that would be different. Okay. And Nolan, I'll let you finish, but I think these are the questions that we need to, I think we should move ahead, listen to Corey and Nessa, and then we can have the discussion, but Nolan, why don't you go ahead? Yeah, I think I'm just gonna pretty much just follow up on Jen and just be like, it would be somewhere in the Diva budget book, they might have broken out along with all the other Medicaid eligibility groups that we have, but I think that it would be really more a backend calculation. Got it. But from like Jen said, from the beneficiary standpoint, it would, no, just like beneficiaries don't know now. Yeah, okay. These are good questions that we're gonna have to answer as we go along. Anything else, Senator Hardy on that? Not at this time. I just wanna make sure that there's not some way that the services or the process or whatever from the patient perspective would be different, that these women and children would be treated like any other women and children in Vermont who qualify for Dr. Dinosaur. So you want it to be nationality or immigration status blind is what you're saying? Yeah, I mean, we have to do what we have to do from the sort of legal perspective, but from the patient care and all that perspective, it is the same program. That's what I would want from a patient perspective. I'll let Corey talk about that, but I think it'll once they get that card that says green mountain care. Nothing. They don't know, they just know they have coverage. It's all the backend stuff. So for instance, we have all these different Medicaid eligibility groups, but people don't really know which bucket they fall into from the administrative. They just know that they have the services and everybody has the same services. So again, I'll let the commissioner comment on it, but I would suspect that it's once they're in, and they get the same services as everyone else. Great, thank you. Okay, any other questions from the committee for Jen or Nolan? Okay, and Nolan, at some point, we'll look at a fiscal note from you. We don't need to do that right now. It's posted, if people want to look at it, no question, reach out to me. Okay, good. All right. So commissioner Gustafson is here and this is James, also who is the healthcare director for Diva is here. So thank you both for being here. Greatly appreciate you're taking the time. And you've heard some of the questions that very definitely relate to your role in this. And so why don't you provide your testimony on the bill and then I'm gonna also have some questions. So you figure out who's going first and so on. Good morning, Corey Gustafson, commissioner and department of Vermont Health Access. I feel like the questions that we've heard and some of the answers we should maybe can just knock right off, I mean, Nolan, first of all, Nolan's right. I think that once you're in the program, you get treated similarly to everyone else depending on provider, I suppose. I hear a budding commissioner in there Nolan. So I might need your resume soon. Oh, I think we're here to inform as best we can. I think that you're right. There was Jen is right about the language that if it's going to be state only, the connection directly to the way it's written might make it, I don't have language specifically for you Jen, but you're right that we are going probably need, if we would think we do want it to be state only and I'll explain that why right now. The reason we want it to be state only is because the process would be different if we had hoped to get Medicaid match for the expenditures and different in a way that we don't think is beneficial to the member. The process of enrollment that we're talking about, I'm not trying to leave the names outside but basically the applicant would have to be denied and then come back in and they'd be denied for normal Medicaid and then come in through emergency Medicaid and it's a, it would be different, it would be more complicated and we think it would be cleaner. The state only should this become a program that the legislature wants to have as part of its. Let me clarify that. So you're saying that not only for the first year, it's state only, but it would continue as such? That it would be most beneficial as far as process goes. It's basically to the question that Senator Hardy was asking, how do we make this so that it is not the feeling of a different process and that making it state only allows it to be a more streamlined process. It's actually a more streamlined process on the operational side for the execution and of eligibility determination. So anything that is less complicated provides for less opportunity for mistakes or misunderstandings or system failures, system by system I mean the technology. Okay, then let's suppose after the first year, obviously the first year is an appropriation that is state dollars. And then the next year when it goes into effect as a ongoing program, but it still continues to be state dollars. How, if at all does the state get match for, no. Yeah, that's what the state only part actually completely means, Senator Lyons. I wanted to clarify that. Yes, thank you. So there will be no match. Yep, no. So it would become an ongoing expenditure of the state for initially it's X number of dollars, but that would change over time depending on demand, how many folks would be eligible and in the program. Yep, that's all correct. Okay, so, and I guess when you presented this to the House Appropriations Committee, was there clarity on that or not? Well, we didn't present to House Appropes. We were in to House Healthcare to basically share similar to what we have here for you today, essentially the cost estimates, which you, I believe you probably have the fiscal note. So as Nolan said, when that NISA worked with our teams on that, so that's one main reason she's here is if you want it to go into that cost calculation. I think the other thing we shared with House Healthcare is just operationally, I believe they lit, I know they listened to us about the system work that would be necessary to create a process for determining eligibility of the population. So we shared essentially the competing efforts that are underway or will be underway very soon, just to make that every program or project has to have a implementation and it goes into the world of the other implementations that we have either underway or coming at us. And when I say that, I mean all the IE&E projects that CMS and the legislature and the rest of the administration itself really has been pushing itself to move forward on. CMS is very, very closely watching our progress on that, those efforts, you have a budget approvals for those IE&E efforts also, you don't. I'm gonna ask, I think that there are members of the committee who are maybe unfamiliar with the integrated eligibility, although I know we went through it both in institutions and in health and welfare and other committees last year, but just one sentence on the- Oh, that's, Medicaid doesn't do things in one sentence. All right, two sentences. Yeah, the overarching goal for the agency of human services is that our eligibility system is able to process not just the healthcare-ass element of eligibility for our programs, as Nolan said, we have many different Medicaid eligible groups, but also that it's a sort of single point of entry for monitors to come in and access all many different kinds of programs. So SNAP, housing, et cetera, that's the big, what do they call that, the big audacious goal, right? So we have been on a pathway to this. It was and has been quite severely hindered by our system itself and where we've found ourselves due to the implementation of Vermont Health Connect. And so, and it isn't just Vermont Health Connect that is what we need to fix, but we need to fix, we need to bring the eligibility's together while fixing a system that is very much process-based. And so it's a pretty complicated program and CMS is very interested in our continued progress. So that's how that kind of fits into what we have going right now. The other thing is on our, in front of us, we have the end of the public health emergency. I think that's coming at us faster than any of us really will imagine, but it's coming quickly. And part of the changes that were made during the public health emergency was to not execute on redeterminations and we will have to get going on those ASAP. That is another CMS. I wanna say non-negotiable, it's non-negotiable that we get going, how quickly we have to get those done. We are trying to, through national organizations, negotiate a little bit, give us a reasonable amount of time to get those done. Every month that goes by is another, time of group of people that need to be renegotiated. Although we are past a year, so we're kind of, yes, that is, we will have a year of redeterminations as well as doing redeterminations. So it's kind of a doubling of the work that would need to be done. So the reason that's important in this context is the people that are doing that work are also the people that need to test on systems. And so we're already constricted. The thing I just described about IE and E projects with the people working on the redeterminations, there isn't just another group of highly qualified and trained Medicaid personnel who can test on this. We need to sort of use them in both contexts. So that makes it a challenge. There are two others. There is our environments on the systems that are difficult to do more things. We have a certain number of environments to test in. And so as we add things, it just sort of pushes back that our ability. So the last piece I wanted to mention is the ARPA, the American Rescue Plan. Contemplates or doesn't contemplate. It has now will increase subsidies and that falls right into the same system. So we have to make system changes to address the increased subsidies. And so sorry that it's a long list. I usually like to hit a couple of points, but it is just, it gives you a context of what we have in front of us. And I see Senator Cummings hand and I just saw Senator Hardy raise her hand. Before I turn to them, I have two questions and because you brought up two things that I had questions about. One, and then we'll go to Senator Hardy and Senator Cummings. The ARPA funding and if and how any of the ARPA funding might support the work that we're talking about in this bill, that's one. And I don't know that you can answer the question. I know there's some guidance out about that. Certainly the guidance does support specific racial and ethnic groups. And I don't know about, you know, so citizenship there may not be a part of that discussion. So that's one question. Then the other question that I'll ask off the bat and I think Nissa might be able to respond. And that is in terms of the costs associated with this, the ongoing costs, I mean, the initial, I haven't looked at Nolan's joint fiscal note yet, but what are the differences between the initial costs and then the ongoing costs for the implementation and then the ongoing operation for the program? Okay, so on the subsidies question, so I was kind of referring to the qualified health plan subsidies and the work to do there. Like there is other monies. I mean, I think there are direct to provider. There's an appropriation in the ARP for direct to provider. I don't know that there is a direct line to the conversation we're having now and the population we're having now to those dollars being dedicated towards that kind of, those kinds of services. But, you know, that's, I think that's the best answer I can give on is there something in ARP that connects directly to this? I don't have that answer right now. We'll look at it again. Well, I think it might be helpful because knowing that funds are available, might be able to, you know, clear the pathway at least initially. The second question was the cost and the ongoing. There is obviously the build, the build on technology. We have an early estimate on pricing for the implement the DDI work of 100,000 dollars. And then cost ongoing, we don't have an estimate on that. But I would say obviously less than $100,000. So, you know, there's the work that needs to be done by creating a new program. You know, I don't think it's a huge population. So I wouldn't afford a huge amount of staff time and resources to it. Of course, depends on what the process-based is, process for the eligibility determination and enrollment. But, you know, I don't think we have a great estimate, but I wouldn't, I just would say to the community, I wouldn't stress about that. I think that- Okay. I mean, actually this is very helpful. The initial work I know, which would pull you away from the ongoing IEE work is problematic. But if there are federal dollars that support any of the work initially, that would be good to know about. And then it sounds like the operational expenses, because, you know, talking about a small population of people is something that at least our committee might be able to sell to appropriations. I mean, that is, yeah. Because we won't, because what I'm hearing you say is that we would not in the future years be eligible for a match. Yeah, never say never, right? Yeah. If federal law changes and something happens, then I guess match could be a possibility. I don't think we have the, I mean, we, yeah, I think that this is the proposal as it is now as a state-only program is what we would say primarily best for the member or for the eligible group. There is just, you know, it is secondary, but it is true. It would make a more streamlined process for determinations and for system. And so that is, you know, one of our priorities to not make things more complicated. Okay, thank you. There is one, the other piece I just say is we, we, well, I think I'll pause, I'll stop. Okay, so I'd like to turn to Senator Hardy and then Senator Cummings for their questions. And then we'll- Senator Cummings is not, is saying, she doesn't. She doesn't. Her hand is up and that's- You don't have a question, Senator Cummings. Okay, I was going to yield to Senator Cummings because I've been asking a lot of questions, but since she doesn't have a question, I'll ask my question. First of all, commissioner, thank you for your testimony and what, when you were going through the sort of list of systems changes that you have to make over the next year or two, those weren't specific to this program. Those were sort of broader systems changes that you need to make for the whole universe of the work you're doing, correct? Well, they each have a specific thing that they're related to, but they are not, they aren't prereqs to this. They are just the outline of operationally, we have a lot in the hopper. And so consideration for that, I was sort of outlining that we shared this with the house healthcare committee and they understood that and gave us that timeline of next July. So July, 2022 as a way to understand all those things. And if I did not recite all of them, you couldn't have known, I apologize for how long you took that, you're correct. It's not prereq for this. All right, that's helpful to know. So the timeline with the one year sort of buffer zone kind of thing, you're good with that. You can get your team to work on that. We think we can accomplish it. I mean, we thought we'd have IE&E done a long time ago too, but. Okay. That's a little joke. Should I hold you to this or not? Yeah, no, I'm just, you know, I just say it as like. Wait, wasn't there was something that got in the way of that? I'm trying to remember, it's called pandemic. Yes. Yeah. There always is. My other, I appreciate your comments about wanting to make this state-only funding so that the process and the care and the whole thing would be the same for people eligible under this program as it would be for people eligible under the larger program. And I think that is exactly what I was wanting and talking about before you testified. So thank you for sort of underscoring that and saying that that's what you're also looking for. And my impression just we, last session I was on the agriculture committee and we did a lot of work on trying to find, to get the payments for undocumented Vermonters, the stimulus payments that most of us got to provide equity payments for them. And Nolan actually worked on this with the Ag Committee and we ended up not using any federal money for that because of the complications that that caused. So I guess even potentially using ARPA money for this I think could complicate it in a way that is undesirable for moving this along. So it just wanted to put that out there. If you're looking at ARPA money, fine. But I'm gonna guess it's gonna make it more complicated. And the, now I can't remember my other question slash comment that I was gonna make for. You want a second to think about it? Cause I just. Yeah, go ahead. You hit on the piece that I decided not to say which is on the idea of ARPA money. I don't think there's anything in there that we would brag into this process. It was my comment was more about there might be ARPA money that it goes into this healthcare, into the healthcare that comes into Vermont's healthcare system that could be used for it. But that would be very indirect and not necessarily. So I think we're all aware of what you're referencing. When you involve the federal government, the money's great, the requirements and the execution becomes more complicated and sometimes not the best for the staff or the user of the system. Right. Especially with the federal morass right now with immigration policy. It just becomes extremely complicated. I remember the last thing I was gonna say in terms of the sort of fiscal impact of the program. I think one of the reasons they're not here anymore but that the house was so in favor of this despite the $1.4 million price tag is that over the long run and even not such long run over even a short period of time, the health benefits and savings and healthcare costs to the system as a whole are very real. Because women have better outcomes if they have healthcare when they're pregnant and newborn babies and children have better outcomes when they have healthcare. So I think that that is not part of the, of no one's fiscal note, but it's sort of should be embedded in our thinking about the fiscal impact of this. Well, we'll also hear from Mike Fisher who I'm hoping will reference that benefit to the bill. I think a couple of points on the money pieces or the federal piece as well. Yes, please. So I just wanted to make sure it's clear cause I'm not sure from some of the earlier questions that it was that a big part of the reason it's state only dollars is because a lot of this is specifically not matchable under federal law. The federal law says we can't do this under the Medicaid program. And so that's why we would be doing it with state only dollars. But I think another piece to know, then Corey mentioned that it's administratively burdensome to go through the emergency Medicaid process where somebody has to apply and get denied. And then, but that's also, I think only for emergency medical situation. So it's not preventive care for children and things like that that would be eligible under this kind of Dr. Dinosaur like program. It's just emergency and emergency based medical care. So it's not as comprehensive as what we're talking about either. Right. No, that was clear. But thank you. That was helpful. It's important to underscore that. And Nolan, I did look briefly at your fiscal note and you seem to indicate that the ongoing expenditures would be closer to 260,000. I think Corey mentioned 100,000 after the initial implementation. I'm wondering, Nolan might be talking about, I'm not sure what the 260 is. I was talking about the design development implementation on for the system work, not anything related to the expenditures on the program. Yeah. Okay. And the 260 was specific to the... PMPM 100 kids. Just for the kids. Yeah. Okay. Just a part of them. Yeah. Okay. Got it. All right. And then in terms of the outreach to different organizations to get the information out to the population that we're talking about, maybe Mike Fisher will be talking about that. So I think we should probably move to Mike unless... Sure, I'll just say on that, that we don't have necessarily a plan but we do have our assistor groups, our assistors across the state that assist with qualified health plans and other Medicaid enrollments that we partner with. They're not, we have Victoria Jarvis is one of our people that works all the time with them. And this would probably fall into that category. And then I think we would rely on other organizations that specifically I think if you went down the path of appropriating dollars to be granted out in the first year, then where the grants would go, we would be relying on those groups to really be communicating out. And of course, Mike Fisher organization. Oh, that's right. So you can ask back for comments on that. I think this program is one place to understand how that might not popularize but let people know that it's available. Okay. Okay. Thank you. Very helpful. Nissa, thank you for being here. You're the backup. It's always important to have a good backup. We appreciate the time, your time as well. And I just ask you if you have anything that you'd like to add at this point. So legislative council for the record, Nissa James, healthcare director for the department of Vermont health access, legislative council had indicated that the department had provided some recommendations related to the language as passed by the house. And that is indeed correct. If the intent of the legislature is to create a state-only program that is administratively, I'll use simple, but we're really talking about as minimally burdensome to ensure that these individuals are covered. We would have recommendations for the language as passed by the house. It really focuses on removing citations to the Social Security Act and specific references to the Medicaid or Dr. Dinosaur program, because we want to ensure that we're able to implement the most efficient and flexible way for this expansion of coverage without having applicants have to go through the full Medicaid screening process, be denied, and then be granted coverage. That's good. I think that that clarifies and Jen also clarified, so now I think it's even clearer. But if you don't mind communicating the language that you have, and I don't know if you've already done that, but it would be helpful to send that along to Jen, and then we can look at it and as we're reviewing the bill. Okay, good. All right. So anything else, Nissa, at this point? I should say Dr. James. No, but thank you for asking. Okay. Mike Fisher, I need to grab another pen. My pen just ran out of ink, but Mike, go right ahead. I'm listening. Thank you, Senator Lyons, and thank you to the Health and Welfare Committee, Mike Fisher here, healthcare advocate. I want to attempt to sort of recognize the importance of this moment. This is a pretty phenomenal bill, and pretty important bill. At this moment, for us to be able to consider this sort of really important address, way of addressing a recognized racial disparity in our healthcare system, and to actually do something about it, is really phenomenal. And so I just want to, before sort of launching it to any details about the bill and how it came to be, I just want to pause and recognize that with full appreciation for the healthcare committee in the House and for the House, and for various legislative leaders. And now you, for taking some time to address this longstanding recognized real problem in our healthcare system. So to that end, we at the healthcare advocate's office brought this concern to the House Healthcare Committee last year at the worst possible moment. COVID really interfered with any real consideration and really stopped me from bringing it to your attention last year. And so we took another run at it again this year, and we're just again very, very pleased. And I often, you've probably heard me talk about the different sides of my shop, the individual advocacy side, and the sort of more theoretical policy side that tends to come to the legislature with concerns. This issue really grew out of the individual advocacy side. We had real people living in our communities coming to us saying, I don't know how I'm going to afford to have my baby. And bumping into one problem after another, after another, after another, sometimes without real answers to help these Vermonters be able to maneuver through what should be. What we all would hope would be a joyous moment, give me the opportunity to say, nobody should have bad debt on their birthday. Now I know it's the parents who have bad debt on their birthday, but nobody should have bad debt as a result of a birth. That's maybe more aspirational. So because no one's talked about the numbers, I'll spend just a second. We talked with the people who support the farm workers, primarily the Bridges to Health program, and also talked with the Immigration Assistance Project, people who are supporting people who are seeking humanitarian status to try and get a sense of the numbers. Now that's not an exhaustive list, but it is a substantial list of the people who are here with that documentation. And I'll just say, overall, we came up with an estimation of about 100 kids and about 20 pregnancies. So to a representative Black's point earlier, this is a relatively small population of Vermonters, but it has a substantial impact on their lives. You didn't hear anything about the testimony that the house took, and there were a couple of, should you feel the need to take more in-depth testimony to hear from people affected by this, both migrant justice and Bridges to Health and the Immigration Assistance Project assisted in bringing people forward, real people's lives to really describe the dynamics and sort of the pressures on them as they go about their lives here. That thank you, that's helpful. And we may call on them to bring some real stories to us where they were in our committee last year as we were talking about some other issues and actually related issues. So that would be great. So I just messed up my screen. So one of the things that we have learned some about in the process of advocating for this bill is some more details about a program called Emergency Medicaid. There is a program called Emergency Medicaid and it is for people who are in true emergency who would otherwise be eligible for Medicaid but for their immigration status. And my office supports people and supports providers in applying for those funds. And so one of the things that became very clear to everybody and I appreciate Diva's recognition of this and verbal commitments, public commitments to work on it. It is tremendously underutilized. You know, again, we estimate that we think there's about 20 pregnancies a year in this population and I think we heard that they covered one pregnancy in the last four years. So that means that the costs of those pregnancies are turning into both unpaid bills for people. I don't want to minimize that and bad debt for the hospital that gets cost shifted. To the point, so there needs to be some work there, both on the provider community to know about this program and in communication, I believe, to people who would be affected by inappropriate language. So I think maybe I'll just list off the organizations who have expressed support for this bill. Can I just ask a quick question before you go on please? So the underutilization of the emergency Medicaid is because of the problem in applying for it. So cumbersome, what would be the reason for not utilizing that? It is likely one part what you just mentioned, Senator Hooker, how cumbersome it is. And I also believe it is one part lack of knowledge about the program. And while the likes of me is going to say, hey, Diva, make it easier to do, I understand there's federal rules that make it burdensome. So you're agreeing then with Commissioner Gustafson to do it through the state to fund this with state dollars would make it a whole lot easier. Yes, yes. I'm sorry, the emergency Medicaid is a little bit of a side comment. This proposal in front of you considers a state funded program for children and pregnant people. Emergency Medicaid is eligible for all Vermonters who are here who are not eligible because of their immigration status. Yes, Senator Hardy. Sorry if I can call on. Madam Chair. Okay, nice try, Mike Fisher. But well, I just want a comment on emergency Medicaid. I think the commissioner alluded to this or said this that it is really an emergency program. So it wouldn't cover the sort of general care for kids that we're talking about going to the doctor and getting their annual physicals or because they have an ear infection or whatever. Those aren't considered emergencies. And I think a lot of the prenatal care even is not covered. It's really just the birth itself is considered quote unquote an emergency, which is problematic in and of itself. And then though there's not the 60 day other period afterwards. So it's really just this single event that's an emergency. And so it's very limited in what it will cover and the process for applying is complicated and not user friendly. Yeah, absolutely. And that also gives me the reminder of making the point that I'm sure everybody here knows, but it just needs to be said. Clearly, everything we're trying to do in healthcare reform is getting the right care to the right people at the right time to avoid more expensive costs later. If there's any place where that is tried and true, it's in prenatal care. You know, if the funding of this program over the last five years prevents one premature birth, we've paid for it. So, and that's just on the money side. I'm also here to say the human side. Well, the human side I think is really important. Really important. Yeah. So let me just, you know, we did, we did not do a really exhaustive search to try and find supporters of this bill. These are largely people who came to us when they heard us discussing it. Of course, my office, ACLU of Vermont, Vermont Academy of Pediatrics, Bi-State Primary Care, Migrant Justice, Planned Parenthood, Vermont Academy of Family Physicians, Vermont Association of Hospitals and Health Systems. Free and refer. Oh, you just, whoop, you froze a bit. Just when I was going fast. You were. Keep going. I'm going to just keep going. Vermont Interfaith Action, Vermont Medical Society, Vermont Worker Center, and the Open Door Clinic. And then because of their association with the university, they can't express a real support for a piece of legislation, but we're integral to helping us understand the population also bridges to health. So that's the list of organizations that have stepped forward in one way or another to support this effort. And I really thank you members. All right. Thank you. And now this is, this has been a very helpful, helpful discussion. And I think we're, I think we're seeing the issue around, you know, state dollars versus federal match. You know, if we don't, if we don't match that 1.3 million, then we don't get another 1.3, you know, where we end up with 2.6 million less to spend on current and existing concerns. However, it's probably more than balanced by the benefit that people are receiving through the state-only program. Wouldn't it be nice if the federal folks understood these needs? But I guess, you know, it does raise the question. I don't want to prolong the discussion because I think we've had a good, healthy introduction to the bill, but it does absolutely raise the question about how healthcare is being managed along the Mexican border right now. And what, what resources are being invested there to take care of people? Just, I don't know the, I, we've heard some of the answers to that, but as it goes through FEMA, does it go through other emergency funding processes? And then what are the long-term, what are the long-term policies that are going to be put in place? And, and can Vermont benefit from that? You know, that doesn't, not going to hold this bill up necessarily, but certainly something to keep an eye on. We'll have to ask our congressional delegation to keep an eye on that for us. I'll talk with Congressman Welch, perhaps. If we can be of any assistance in helping you get people to the table. Thank you, I greatly appreciate that. And you certainly will be. Be careful what you, be careful what you ask for. No. Thank you. Any other questions for Mike Fisher, healthcare advocate? Okay. We're good. All right. So I'm going to suggest a two-minute stretch and then we're going to move on to H-153.