Good to know I Posted a link to the agenda to which I personally have added no items, but I figure if you want to put your name on the attendees list I Know Tim had something on here about walking through the backlog. So I pasted a link to the project board and I Added all of the issues or PRs that have area conformance since I last looked at the board which brings us to a grand total of 49 issues to be triaged and Then I wasn't sure if Sreeni you had mentioned something on the mailing list about maybe updating us on globet Yeah, actually I've been attending the global meetings on Thursdays for the last few weeks and We They are working Pretty good team. I just have to say that first There they have done pretty in-depth analysis on some of the problems like pod life cycles free stop hook They found out there are some behavioral things and they did a few PRs in there Apparently one of the problems we are facing is essentially the PRs are There's a lot of time. It's taking for the reviews to happen. So Some of the issues like, you know, you have to keep rebasing this and then Hoping that somebody will review is taking a bit of time Other than that the work is going great out there. So they they are working on Also the DNS test one of the things that are happening there is that a Lot of these tests that we are running into they could be the next only kind of test. So Or we Should they be validation test? So that there's a point I want to discuss a little bit because we're not sure How to approach those I'm talking about the existing test and we are also writing you this We in the sense that So one approach I am going with is like first storage we are I'm working on some tests and I'm tagging it as storage validation test So that you know, there's a validation speed that goes and then we can eventually tag them as Linux only if we need to or Is that the right approach or? So usually we just write the test and Then promotion comes later right the conformance Yeah, I'm talking about the existing test So during promotion to conformance, maybe we can tag it as Linux only That's what it feels like to me because that's when the windows guys are going to be affected, right? All right, I make sense so but Okay, so something I haven't had a chance to look at to in depth But I've had a couple PR slow by that start to introduce the Linux only tag Right, it's unclear to me whether or not that tag is getting introduced because it's a behavior that windows will never ever ever do Or if they're just trying to get it over the line and get this particular Linux variant of the test through versus a windows variant of the test so the one I most recently ran into is a test that was related to DNS Was split up so that Previously the tests looked for both DNS and Etsy hosts And then it was split up into a test that looks for a cluster DNS And then also looks for entries in Etsy hosts and the Etsy hosts test was tagged as Linux only Because windows can't mount individual files for pods I see Brian raising his hand in response to that take it away Brian Yeah, so I reviewed a lot of these test changes to tag tests with Linux only including that one The I pushed the windows folks pretty hard to get a very Detailed description of what is possible today and what will ever be possible and why What the reasons are for the limitations for windows? So please refer to the kept and if something is not clear from the windows support kept we should get it added to the kept They've been very good about adding additional levels of detail like the latest one is ICMP support Which actually isn't even well documented anywhere for Linux or otherwise The on the Etsy hosts issue the The yeah, it is limited by the single File mounting problem and there are quite a number of features that are in that bucket Quite a number of changes need to be made and a number of components to make that happen is As it's been explained to me And that is kind of summarized in the kept so it falls into a big bucket You know, there were claims made that maybe OS changes even Were needed. That's not a hundred percent clear in terms of what OS change means in that context exactly Like how deep of a change that would be But it's definitely not a Kubernetes only change that would be required So I would suggest that we just tackle all of those at the same time If somehow that becomes feasible in the future. They should all be labeled with the reason that You know, this is marked Linux only due to the single mount file thing So it should be possible to find them. Maybe not fully automatically Because there's not like a tag for that specific issue, but they'll have English text that explains that In in the comment section. Yeah. Yes Yeah, so any for anything I Any test that got Linux only I asked them to enumerate all the reasons why it was marked that because for some things There were multiple reasons like this requires privilege and it requires single mount files and it requires, you know Some third thing in some cases So all of them should be tagged to the best of our ability With all of the reasons why they're Linux only and Is does that show up in the description field that is parsed out into the conformance stocks? Excellent question. Okay. I mean was your my I would assume We would want that information to show up in the conformance stocks. Was that your intent? The conformance stocks are the thing that scrape the tests and lay down an ID the name of the Ginkgo test and then the description stuff in the release that the test was added in that right Serene Let's go and I don't know the answer off the top of my head I agree there needs to be an automated way if we were to you know Try to ascertain Conformance for Windows there need to be an automated way to sort that out, but given that it's not Part of conformance yet. I didn't worry about that step yet Like we haven't figured out if it's going to be additive or orthogonal or whatever even right, okay Stringy, do you maybe want to take an action item to have the global folks confirmed that that has been done? I will yeah Okay, if you can make an issue and then tag it as area conformance and add it to the backlog that way we'll know It's an outstanding piece of work And then it's getting worked on Also hi, I really didn't want to run this meeting because I'm not super actively involved I'd love to hand stewardship of this over to somebody else But I see Tim's name is on the other two agenda items, and he's not necessarily here I Did just want to point out the code freeze is coming up in the next week and a half So if we do want to push to get any more conformance tests added now would be the time to Help with the review problem that Stringy mentioned I to encountered Review bandwidth issues and just general shepherding and ping-ponging PRs across a bunch of different six to help out the global contractors Looking at the existing conformance test dashboards on test grid It appears as though the 113 release of Kubernetes contains 214 conformance tests and the 114 release of Kubernetes as it is currently standing Contains 216 for a grand total of two additional conformance tests. I feel like we could be doing better Well, yeah, as soon as the reviews happened problem have a lot more What active steps are you taking to ensure reviews happen? I?  That's I Haven't done that yet, but That's probably the best way because on the issue itself responses could be leaders so One So sorry, I hadn't interrupt when you were mentioning the review problem Yeah, so there the process is are these new tests that are being written In which case they're not even conformance yet There are there are also. Yeah, there are existing tests that table tests that are converted to individual tests and there are tests that are that are written recently for the for the pod lifecycle and Yes, there are tests that requires to be reviewed and So can I maybe Take a step back and ask hands up who here has been reviewing kubernetes tests or conformance PRs in the past month Serene, I can't see your hand. So you're not on video. I'm not Okay, when I get tagged I review otherwise, I'm not actively So of the 12 people attending this meeting there are only two people who are even bothering to drop LGTMS Is what I'm hearing And I feel like perhaps we could do a better job of addressing that Yeah, so process-wise what I generally do is You know, if it's a new test or it's a significant change even a refactoring to an existing test I make sure that a domain expert in the SIG looks at it Right because just as one example that I like to pick on Was there were some changes being made to a networking test And they said well all these things don't need to be privileged and the test still passes but actually What the test was doing was trying to connect to the cube proxy port on the same host and It was totally unclear that the test was all doing the same thing anymore With the changes that were made to the test, right? So we need to get the attention of the folks in that SIG and I strongly recommend not relying on GitHub notifications I will not look at anything if you just tag me through a get out notification. I am signed up for like Tins of thousands of issues in PRs. So it is all just white noise It helps to apply structure. So if there are multiple Tests that need to be reviewed by a SIG I suggest framing it in an email and sending it to that SIG's mailing list saying look we need to To make the cover these features or whatever That covers functionality of your SIG can you please make sure that we have the right people Looking at those changes. Okay, and then once it is passed the SIG level of review by the appropriate Reviewers and approvers in that SIG if it's just a Test change for test not labeled Conformance then it's done and it can just get merged if the test is already labeled conformance or It is being added to conformance then one of the conformance approvers Needs to be added to that and there are only four of those right now because we need to you know on board more folks So if you are interested in that Definitely volunteer and we can start doing some sort of shadowing much like we're trying to do with the API review process in kubernetes But right now those people are Aaron dims Tim Sinclair and me and well, I guess there's a fifth one Clayton who sometimes will Review things that when he's tagged, but he he's not as active on that You know and and even among those people like if you're not comfortable Approving because it you don't have the domain expertise Definitely loop in those domain experts But it's really important that we understand the intent of what the test is supposed to cover and some Parts of the system are more subtle Particularly I'm thinking networking, but there are probably others as well So don't feel bad about poking those folks, but definitely do not rely solely on get hub notifications. They are worse than useless So I want to Am I trying to say so we've tried a couple different like let's use a smaller more focused channel to make sure that people are aware There was a project board in the architecture tracking project Where I would move things around in the column and then I would try to Paint Brian or Clayton directly in slack and point them at that column when things had stacked up there That works well for me actually Okay, so we could go back to doing that model another thing I feel like you suggested on the mailing list when Patrick had a question that he wanted to address was Just use the mailing list. We don't often send a whole bunch of traffic on the mailing list. And so we could Send something about a PR being ready for conformance approval on the mailing list Yeah, so there's the conformance mailing list which is super duper low traffic, so I don't have to filter it And probably other people don't either if they're really interested in this area. They're also the SIG specific Mailing list so again get have notifications are super hard The teams don't have the right people in them a lot of people are subscribed to issues in PRs And they're no longer the best people or the right people at all And so on so I really recommend You know using other forms of contact the get have notifications are completely unmanageable for multiple reasons And adding a little bit of structure and context will help people understand Like importance and urgency and the bigger picture around that as well Right, so I understand shepherding around other things I'm just trying to like so what I've what I want this group to try and get to you to help out newcomers such as Steve Who have volunteered to help us with this problem is to make sure that we're all looking at one list That is prioritized according to this group's agreement That is not something that I personally individually can do. I feel like we owe it to ourselves to prioritize that list and get agreement that that is correct Prioritization, so that's why the project board I have linked in the meeting notes has a two triage column and then a prioritized inbox column is that what I called it Sort of backlog column and then we could be working off of that sort of backlog, but I also want us to use one channel for that High signal. Hey, we think this is ready for conformance approval So to me, I feel like using the mailing list, which is currently low traffic would be the path forward Whether that's a link to the PR in question or if that's a link to a column on our project board That way we are all looking at the same thing and we're all using the same channel so beyond that like I've got to stop talking because I'm really focused on 114 and haven't had any time to dedicate to this. Yeah, one thing I can do is basically I'm maintaining based on Some of the areas that team asked us to work on I've created a spreadsheet. I can actually on a weekly basis update the status on the spreadsheet and the link that spreadsheet to Send it to the work group, a conformance work group mailing list on a weekly basis or To start with and if that's too much of emails and I can Do it by Shini, I, I, I like that idea just because once we get into some kind of rhythm, then people will say, Oh, okay, there is this email. There is this four things that that I can help with. So go do that. Then go do something else. Right. So just getting into that rhythm seems to be like the right thing to do here. I agree. Yeah, that's a good idea. I hit hippie hacker. Did you have something to say we can't hear you still can I hear you. Okay, here you while we're waiting. I'll just do my usual spiel where I have found that spreadsheets are a source of pain. Because you ultimately have to reconcile that spreadsheet with the state of GitHub, which is where all of the work actually happens. Which is why I would suggest that we try using a project board and consistent use of priority labels, if nothing else, to give us some form of rough ordering in the buckets. I have a question is what works for you, then please just any anything I absolutely agree. But there are items that Globe and had questions about whether to start working on those and there are things that may not show up on the dashboard. So it would be My response to that would be why aren't those GitHub issues Just just to better help us visualize this the scope and volume of work and who is blind of what I wanted to echo Aaron's thoughts around not using the spreadsheet and see about having someone go through that on a regular basis like on a weekly And make sure that we have this email that's after we've gone through and contact the SIG. So going through the SIGs in that week and then going through the The prioritization and looking at those tickets with an email. And I know we've talked about doing the spreadsheet. We get to maybe shadow with where that spreadsheet is now and see if we can combine those techniques. I'd be willing to step into that. Yeah, and just for my own workflow. I also prefer the A GitHub based solution, not notifications, but the project boards when they are up to date have been working well for me. They're easy to discover. They're easy to search. They're easy to manipulate. If you're in the right permission sets. So it's really low friction Yeah, that's my preference as well from we're using cube docs and it does work well. So hopefully we can move as much into that. Yeah, so the distinction between the architecture board and the main board is the main board has Lots of conformance related tasks in it and the architecture board was specifically focused on the final approval For changes to the set of conformance tests that that small group of approvers needs to sign off on So that created a really super high signal place that I could pull when notified through some other high signal channel to go work through the set of things that were ready to be worked through Doshini, can you paste a link in the doc the current spreadsheet itself. So maybe that'll help People go look and see how things should which which of those columns should go where in GitHub like Well, this is very rudimentary right now but But eventually the idea here is to add a couple more columns to to to see if these are black at issues and there will be sub issues that we'll be adding here and what the status of the sub issues like for example Pre-stop hook we added a new test that is still needs to be go through the sake to get added and then we'll push it to conformance. So that level of tracking I cannot do it through project dashboard but in the spreadsheet I'm planning to do it. We use the umbrella issues also for things like this. So right in the first box we have we have the list of items and we check through each one of them when they are done right. That might be helpful as well in addition to the project board project board plus umbrella issues should be able to do what you are doing in the spreadsheet right now. Wait, sorry, I'm looking at the spreadsheet and it just has links to issues and that's totally something a project board can do so I don't understand the problem by the way in the Kubernetes or we're also working on a bot that will automatically populate a board from query. So that hopefully will make it easier to Slurp content into project boards automatically. But for this number of things like 12 things that they could just be copy pasted into a project board in like two minutes. Right. I agree. We caught it early. Sounds good. I can work on that. But still we do a weekly email. Do we want to assign someone to that? I'd like to step up and be involved in the board and SIG reaching out to the SIGs. If someone wants to co-do that with me, I can just take it. I would be happy to help with that and we can coordinate. Thanks, Sri. Let's do that. Yeah, I think the weekly or the email will really help because I'm sure we're all getting pulled in different directions and I'm going to follow my sword and say, yeah, I haven't done any reviews because I got distracted on 20 other thousand things. But seeing the email and seeing what's going on, well, I'm sure helped me and also help a lot of you folks who are even busier than I am. It was a good point, Aaron, to pull that out. That has been happening. It's been a crazy start of the year. I mean, I just raise it because to champion the individual who's not here, Tim claimed he had like rallied the troops and had was going to bring some review bandwidth to this group. I understand there are people who need to like talk about this strategy and the path forward here, but we also actually have to have people who do the work. And so I'm just trying to make sure that we have the right structure in place to encourage that level of growth and then we have the right people showing up. Perfect. Yeah, in particular we had, you know, I'd sent out the email about the priorities to the list and there was an action item to convert that to issues or something has happened done. I am not aware of a one to one mapping of that having been done. I believe that's why you're seeing the global contractors work on some things related to life cycle hooks. And yes, determination and things of that nature. All the items that are listed in that email. They are now issues in the guitar. I fear that those that there are any others please send it to me. I fear the issues and follow up on them. May I bring up another issue? So for storage test, storage test, we need set up with strapping to run those tests. So we're creating storage validation speed kind of like no performance. At some point in Seattle, we discussed that instead of calling node components, we should call it as node validation. And similarly, I'm calling the approach to for the storage adding a tag called storage valuation. Is that the right thing to do or is there any other. Yeah, so it's not going to be part of the conformance suite, especially if it doesn't interact with a as an end to end test with the whole cluster, then we should call a validation suite. This came up, I think just earlier this week as well where that signode is planning to create a test suite to validate CRI implementations.  And we're going to need that for CSI. We're going to need that for other things. The big challenge we have with storage. I haven't had time to get back to it is you don't have sufficient abstraction around all the different volume sources with consistent functionality that we can really test it in a general purpose way. I think it is possible, but it needs more thought. I don't know what the current status of that the storage was working on a proposal at some point, but I put working with them on the back burner to get some of the more basic urgent things covered. Yeah, I'm more interested in the general approach so that we can we can have a uniformity in in how we tagged the test. But which which API is with the storage validation covered the CSI level or the API level or. It's some of them are API level. Some of them are CSI level. There is a list that we're working off of right now, which involves some amount of things like volume sharing between different pods and Wait, that's volume sharing between pods can be done with empty during should be portable. So that could totally be just conformance and should not be storage validation. Right, but yeah, could be right. Then there there are CSI driver level for spot if are any with the default storage plus then those tests may not be portable. Maybe in those case we need to have it right so right so if it's CSI level I would call it CSI validation if it's Testing optional functionality that or functionality that might not be entirely portable. Those are just end to end tests. Assuming that they are end to end. So I don't know that they need a specific term to describe them we have lots of end to end test coming covering such parts of the system. Is the six storage driving this shiny or you you're doing it. I'm helping out the six storage. They are driving this and they are identifying the test but again we need to bucketize those tests like Brian said some of them could be We can blanket call them as storage validation or we can Subdivide them as CSI validation versus Flex volume or validation or whatnot right Because not everything is probably available by default without some bootstrapping On the on the cluster we are running so I kind of get it and for CSI we can start using CSI validation there. I cannot think of test that Are not portable But can fall under the blanket of storage validation so Well, so example of portable tests would be would they have to do a storage would be any Most things in teacher related. There are some exceptions to that like It's depending on the media type But you know Local storage emptied or Should be portable. It's all the other network attached volume sources that are all non portable and all optional And that's where you know I'd ask them to think about how we could abstract that into some kind of basic functionality that could be Just represented through the default storage class that a PVC Could take advantage of and those could be candidates for Incorporating into conformance some way in the future But still I would just categorize those as they're just into in tests It's really the extension point Validation tests where they want to determine does this Flex volume work at all. Does this CSI Driver work at all where maybe we need a different set of tests that are more like the original node conformance tests Where they actually launched cubelets as part of the test and just exercise specific cubit functionality So those sorts of lower level tests, which don't even have the mechanically aren't even Possible to incorporate into conformance when written that way Those are tests that clearly should have just a different kind of Different kind of name where someone who's building a cloud provider or building CSI driver or something like that might want to run those more targeted Tests that just exercise that interface Yeah, so this is Good Yeah, that's what I'm targeting basically Which I'm trying to name them as validations rates So to subcategorize the test. So is that the right approach on one of the things that I wanted to do is also take the node performance tests and call them as no validation tests But then there will be a lot more deeper impact if I if you do that. I don't know Because right now I don't know if those there are scripts that are already referring them as node components tests and things like that. Should we should we provide some direction and how this test needs to be changed Essentially Sorry Naming is hard. So Let's get the tests in first and then we'll leave and tag them I guess So to respond to Aaron's last comment. I don't think it's a new way of punting profiles Because just again like the examples that have been raised to me In other forums like CRI Tests If it's actually literally just exercising the CRI Extension API then it will never ever ever be a conformance test which requires a whole cluster. So For that for those kinds of tests I just recommend I just pleaded with them not to call them Conformance weeks because it in the past like with node conformance has engendered confusion They should just from the get go call them CRI validation or something like that because it has a totally different purpose So this is Go ahead Sorry, so this is the what continuing one of the items I have at the end of the agenda. So the cloud provider wants to have a test suite That validates behavior in for entry and out of two providers So like providers can use that test suite to to make sure that when they do the switch over that, you know, things work as expected So yeah, I talked to Tim about profiles and he said that was kind of stalled right now And so I personally don't care what it's called But I would like to know if Or let I guess like best practices for how we label categorize these tests like are we good to just add a bunch of Test and tree And then worry about what to call them later or do we want a solid plan for what is going to be called now before we go to the test And add a bunch of tests in there Would these cloud provider tests Use the existing end to end framework or some new sort of integration testing mechanism So that's something that I would want guidance from this group But it would be much easier for us to because there's already like a provider framework in the end to end test so leveraging that would be nice but With the whole like out-of-tree direction going I'm not sure how much it makes sense to be adding more provider specific tests into KK These would be provider specific or these would be generic for all providers Sorry one second I had a phone call So they're not so the tests wouldn't be provider specific but the implementations like we would still need underlying implementations that do provider specific operations That makes sense. So like one example is like the delete node case right so the test case or what we're testing for is generic but We will need we need to call like an underlying implementation that can call the AWS or GC API and delete the node to validate that behavior Well as far as how you mechanically implement this I think this is a good discussion for the testing common sub project which I literally have to make a zoom for right after this meeting And their next meeting will be Friday at 730 a.m. Pacific time This is the sub project of sick testing where we talk about the best practices for how to write an architect tests just for what it's worth And I feel like there's a concerted effort to try and extract the provider specific stuff out of the e2e framework and so that would be the right audience to ask those sorts of questions to Okay, but I guess okay, but we know for sure that we don't want to be labeling these tests as conformance or like we may have profiles in the future but that's different discussion tabs Correct so what I would do for tests that exercise cloud provider functionality is I'm good with labeling them cloud provider or something like that so if a cloud provider just wants to exercise The set of tests that's going to test whether their cloud provider implementation works there's an easy focus that they can set to do that and if we develop a profile that incorporates common Cloud provider functionality in the future it will be easy to incorporate them but also easy to exclude them right now so I so I wouldn't use conformance in the name at all I would just call them Yeah, like feature cloud provider or something that Aaron is typing in the chat seems like a good way to go assuming that they're going to use the end to end framework. Okay, sounds good. Thank you. I feel like you're going to run into the same sort of issue that like CSI stuff would run into where there's a certain set of behavior that you expect to work across all cloud providers, but that there is also some cloud provider specific stuff. Like, you know, maybe GCE offers some things that Azure doesn't offer that AWS doesn't offer that sort of stuff and you're going to want ways to exercise that but it sounds like what you're talking about is making sure that there's a minimum bar that all cloud provider implementations meet to do things like node management, storage management, load balancing, yada yada yada. Yeah, and pretty much given the profile stuff is still they just wanted to know next steps for what we can do now and it seems like we can just add tests entry labeling them with feature cloud provider and then figure out how to put it into profile later. Brad. So Aaron made a great comment about hey, we talked about this but nobody ever wrote this up and it's a little squishy. So, so myself and Trini will be happy to take a first draft at writing up what we talked about and why we're calling things validation suites. If that's okay with everybody will be happy to grab that right it up. Because I mean think you all verbally are talking about what we agreed to it's just anybody who doesn't have that tribal knowledge is going to get frustrated and I think that's a great point Aaron so we'll take that. Thank you. That be part of the existing best practices document somewhere or it would not be part of the conformance documentation would be part of our testing documentation. Fantastic. I think you had a couple items on the agenda that we haven't gotten to yet. So, if you remember, we were talking about the adding a end to end test for the conformance image. I ended up filing a PR against testing for for that. Please take a look when you get a chance. Right now it'll run only one test. This one just to kick the tires and it's an optional test which will not be triggered, but you have to trigger, you know, by hand if you want to try it out. So it's so that it doesn't stop anyone. So once we get that working then I can remove the restriction of just one test and run all the end to end tests so please take a look at that. Sure. Yeah, it's easy to try it out in your local environment also because it uses kind and then it has some scripts to build the conformance image and then run the conformance image and get the e2e log and the g-unit log out. So please take a look at that. If anybody else is interested, Steve, maybe you might be interested also because you're doing stuff with Sonoboy. This is the Sonoboy image which got imported into the KK repository last cycle and then the cycle trying to do a little bit more with it. I also had a question about whether or not this supersedes an existing PR that Srini opened back in October to try and get conformance tests running with this image. Yeah, I think it supersedes that one. I don't think we made much progress there. We did not make much progress. We need to add the job into the... Right. So I lined up all the ducks in a row now. So there's some changes in the KK I had to make that merge this morning. Tim did that. Okay. Yeah, I think there are issues with the image being built that blew up timing of releases being cut. So I'll keep an eye on that. Thanks. Okay. No, we haven't changed how the image itself is built. I'm not adding it to the any of the make cross any of the things that we use for the release itself make release. It doesn't show up there. Okay. So then the next question I had was we I think we talked about publishing an exact list of tests for each release because you know, we and using that list to check if somebody even the conformance folks get a get a text file saying these are the tests that ran then compare it against them against that list to make sure that they didn't they didn't skip any of the tests. Right. Do we have such a list? Are we publishing it with any of the releases yet? Like I mentioned last week, I mentioned about a cap that is is still out there. Not a book. And I do have a mechanism. I'm testing out to build on this document as part of the release bill. So it will be part of the tar under docs. So if that kept goes through. So that way, the conformance document that will be generated as part of each of my minor releases at least so that they can be a reference document. Right. So that was the first part of the equation and the second part of the equation was I remember Ben the elder had a PR out which would take two sets of lists and compare and see if anything got skipped. I think that was the other. So give people a tool so that they can check if they accidentally skipped any of the tests. So I have to go chase that I think. So I'll do that. Those are the two things I had Aaron. Okay, yeah, as far as I'm aware we don't currently produce a list of conformance tests as a release artifact and they kept for that didn't make it to implementable status by the enhancements freeze. So we have to proceed with what we have been doing thus far, which is a manually produced list of tests being sent to the Kate's conformance repository. I feel like we have a number of PRs that Serene has opened that are a little backlogged on chef tacos approval, which I can try to poke on some more. I feel like I've been poking the past couple of weeks, but all I can try again. Thanks. Thanks. Wait, so this is different than that conformance dot txt file. The list of subset of tests that are included. Actually, I'm kind of unclear that list is is is hard to deal with sometimes because it strips away all extraneous tags. So you just get like some of the text. What I'm talking about is a document that is generated a markdown document that's generated that contains the blocks of comments parsed out a little bit. So what release does this test added in what's the description, the human readable description of what this test does stuff like that. Okay. About a machine readable thing dims. Um, yeah, machine readable was more like it because then we could have a tool which will spit out what this is. This is one of the things we've been looking at with API snoop was actually to have a basically like a web site to go to where you have each of the API's and their history of when they were added when they were permitted to conformance and what applications within the Kubernetes. Test to hit them since we know their user agents and eventually what applications in the community you're hitting them. That's of interest. We could also generate the user generated portion if we want to do that. Okay. Um, yeah, and that's more information than we have right now and as I would think we need to consider who's going to consume the information like who the audience is. If there's if there is anything that is actually checked into the repo, I think it should be relatively straightforward to get it added to the release bundle because we copy a bunch of other files from the release bundle. At some point we used to include the whole source tree. I don't know if we're still doing that now. But one thing that I think would be straightforward to do at least which I requested recently and I just put a link to that but the notes and in the chat is for anybody who is reviewing changes to the conformance tests, just ask the author of the PR to include a release note. That should be a relatively straightforward thing that we can do. Um, that was probably of interest to some set of people, but the release notes in my mind are the places where, you know, cluster administrators and people building clusters and Kubernetes providers as well as, you know, end users should look for a while what what has changed that's relevant to me in this release and we might want to think about breaking down the release notes by audience that's a lot more work. But at least for now there aren't too many of these changes like Aaron said they're like to the coming release or something like that so just ask for release notes on those changes for now until we have a better solution. So that only compares the delta, but if you're running like 216 and if one or two tests are skipped, there should be a mechanism to compare what's been skipped. That's what I think that is asking about. So on the skipping. I actually commented on this somewhere as well or maybe I found an issue. We have a relatively small number of skip. Directives in the test framework. I think when a test is labeled conformance we should just set a global variable and if that thing is set when and skip is invoked it should just crash. Like we should not allow skip to be invoked during a conformance test. I, are you talking about the ginkgo skip itself, Brian, because ginkgo skip will make sure that it never gets to the point where the test is invoked. I'm talking about the skips that are in the code. Yes, skip if that sort of stuff. If provider is blah blah blah. I don't even think it needs to be a linter in this case could just be an assert. Yeah, those kips we already removed they were. No, I just found one. I just found you did. Okay. Okay. That's why I brought this up. Okay. I believe there's also still some confusion in the appropriate regular expression to be used in executing. Maybe we can hammer this out as we hammer out the test image dims, because I felt like at one point in time, we discovered that son of boy was using a skip regular expression that included where it's like. Alpha cube CTL feature, whatever, and you and I have been trying to reduce the delta so that all you have to do is focus on conformance. That's it. That's it. Full stop. And it should be really easy to generate the list of tests from that. Right. So just related to that Steve, are you working on son of boy too. Yes, yes, I am. So I'm, I'm going to help out with Tim Tim Tim's just here to help me help everyone here. I was more so I'm happy to help with reviews shadow of things. Sounds good. So I guess for the neck by next release opens, let's see if we can get to a point where son of boy can run the conformance image that is there and the main KK repository. So then we won't have to worry about two images. Yeah, that's one of our action items is actually get that working. Yeah, absolutely. Thank you. Okay, we reached the end of our agenda. Four minutes to spare. Four minutes. We did not talk about the coupon. Is there anything that Who's organizing or running the coupon stuff. I guess that'd be me. So my goal with this one I've requested a 50 minute session. As a combined session, I think it's more valuable. The goal is that we spend kind of a short amount of time giving an intro to anyone new, but then really actually just focus on making progress tasks that we have. Because I think that's kind of the value of us actually being there. So with that in mind, I am collecting topics if anyone has something they want to raise that that'd be helpful to kind of hash out in person. We know, for example, last time in Seattle. How much more progress. Do we think we can make then during this meeting. I don't know one of the main benefits of the last coupon meeting was it included people for whom the these video meetings are hard to make like people from Australia, for example, but that's like that was like one person. So I'm not going to be there in Europe. And that probably a lot of other people from the US are not going to be there. So I don't want to have another meeting where we end up just like rehashing all of the content of that meeting during one of these meetings after it. Yeah. Well, maybe I'll maybe I'll put out a call and see actually who will be there. So we can gauge on how useful it will be. And if there's critical mass. Does that sound like a good next step. Yes. Sure. The only other thing I had for this group was, I noticed that the K3S folks stripped out a CD and they're using SQLite three. And we don't have a conformance test and they pass the conformance test to so I don't know what we want to do for things like that. So please everybody take a look at that and we'll put on the agenda for next time and follow up on the mailing list if you find anything interesting with respect to that this with respect to the SQLite thing. That is one of the reasons why I said we need to cover all SED dependent behaviors in the conformance tests. We also have the Cosmos DB implementation by Microsoft. People are starting to swap this out. So we really need to make sure that the semantics we expect to be respected are actually implemented and covered by the conformance test because workload portability is one of the goals but ecosystem tool portability. Like, you know, there are literally hundreds of operators now. We need to make sure that those things just work. So with respect to the other things they removed, I haven't looked at that in detail. That bothers me less than, you know, coverage of pods and API server and networking. Like those three things really need to be very well covered. Being certified for Kubernetes 113 does not mean you are certified for all future versions of Kubernetes. That is correct. Yeah, so please follow up on the mailing list if you discover anything interesting and let's have a more informed discussion about that at a future meeting. Thanks. Sounds good. One minute. Okay, see you all later. Bye everyone. Bye.


Hello, I'm Russell Lai. Welcome to my talk on subtractive sets over psychotomic rings, limits of schnoll-like arguments over lattices. This is a joint work with Martin Albrecht. This work is about the relationship between the mathematical objects called subtractive sets and lattice-based schnoll-like arguments. Since you are watching this video, you are probably interested in lattice-based schnoll-like arguments in general. Therefore, I recommend checking out these concurrent works which are also appearing in this year's crypto. Let me begin with this slide that every other lattice top needs. That is the slide about the short integer solution problem, or the SIS problem. The SIS problem is parametrized by a modulus q and a norm bound beta, and a problem instance is given by a matrix A and a vector y, and it asks us to find a short vector x so that A times x equals to y, modulus q, and the norm of x is bounded by beta. Usually, the SIS problem is considered over the integer ring, that is, the matrix A, the vector y, and x are all vectors and matrices over the ring of rational integers. But in this work, we are considering the SIS problem over general ring R, that is, a mathematical object that allows you to perform addition, subtraction, multiplication, but not always division. The motivating problem of this work is that of proving the knowledge of an SIS witness x. To be more concrete, let me introduce this generalized SIS relation, which is parametrized by a ring element s, a norm bound beta, and also the modulus q, which is omitted. The relation takes as input a statement, which is specified by a matrix A and a vector y, and also another vector, a witness vector x, and checks if A times x equals to S times y, modulus q, and the norm of x is bounded by beta. Notice that the vector x doesn't necessarily satisfy A x equals y exactly, but instead there is an element s called a slack here. If the slack s equals 1, then we say that the witness x satisfies the statement ay with no slack, otherwise we say that the vector x satisfies the statement ay with slack s. Generally speaking, an argument system for proving the SIS relation involves two entities, the prover and the verifier. The prover inputs the SIS relation ay as well as the witness x, while the verifier only inputs the statement ay. The prover and the verifier may interact for several rounds, and eventually the verifier will output a bit b, which is either 0 or 1, deciding whether or not it believes that the prover p knows of a witness x that satisfies ay. An argument system could satisfy several properties, and in this work we are going to focus on the completeness property and the soundness property. We say that the argument system is complete for the relation r1 beta. If whenever the witness x satisfies the statement ay with no slack, then the prover is able to convince the verifier to output 1. On the other hand, we say that the argument system has kappa knowledge soundness for the relation rS beta prime, where kappa here is known as the knowledge error. If there exists an efficient knowledge extractor e, such that if the prover convinces v to accept the statement ay with some probability rho which is strictly greater than kappa, then the extractor, when given oracle access to the prover, should extract some kind of witness x tilde that satisfies the relation rS beta prime, with probability rho minus kappa. A recurring challenge when designing argument systems for the sIS relation is to design the prover and the verifier such that the knowledge error kappa and the slack s are both minimized. Of course, one would also like to minimize the difference between beta prime and beta, which we call the stretch in this work, but I will not go into detail in this talk. With this background, let me overview the existing argument systems for the sIS relation. Prior to the year 2019, there were basically three types of argument systems for the sIS relation. The first kind of systems are based on PCP or probabilistically checkable proofs, which is a kind of information theoretic proof systems. These constructions usually start from a PCP for proving statements from a certain MP complete language, such as R1CS, and compile it with a commitment scheme into an argument system. These constructions typically achieve logarithmic proof size and can prove the sIS relation without any slack and stretch. However, to achieve negligible knowledge error, a super polynomial size module skew is usually needed, which impacts efficiency. The second type of argument systems for the sIS relation are called Stern-like systems. These are combinatorial systems based on the classic cut-and-choose technique. Like the PCP-based proof systems, these Stern-like systems can prove the sIS relation without any slack and stretch. However, due to the combinatorial nature, they often require linear size proofs and can only achieve constant knowledge error, which means that the argument needs to be repeated lambda times where lambda is the security parameter to achieve negligible knowledge error. Finally, the third type of arguments are schnoll-like arguments. These are algebraic arguments which can achieve inverse polynomial knowledge error, and therefore only lambda over log lambda repetition is needed to achieve negligible knowledge error. A distinctive feature of the schnoll-like systems is their linearity, which could be exploited for recursive composition using a technique called bulletproof folding to achieve logarithmic size proofs. On the negative side, schnoll-like systems could only prove the sIS relation with both slack and stretch, which is significant because these get amplified when performing recursive composition. After the year 2019, there was an effort in combining the best of both the Stern and Schnoll worlds by introducing extra nonlinear constraints to the schnoll-like systems. Like the schnoll-like systems, these new systems can achieve inverse polynomial knowledge error, and like the Stern-like systems, they can prove the sIS relation without slack and stretch. However, due to the extra nonlinear constraints, these new systems are no longer compatible with the bulletproof folding technique, and therefore it is unclear how to produce logarithmic size proofs from them. Given this state of affair, a natural question is therefore, can we keep the linearity and inverse polynomial knowledge error of schnoll-like systems, but at the same time reduce the slack and stretch? To answer this question, let us take a closer look at some existing schnoll-like systems. Our first example is the lattice analog of the textbook schnoll protocol for proving discrete logarithm. In this protocol, the prover first sample a short vector u and compute a vector v, which is equal to a times u, and send v over to the verifier. In turn, the verifier would sample a challenge c from the challenge set big c, and send c over to the prover. Finally, the prover computes the vector x hat, which is equal to u plus c times x, and the verifier checks that x hat is indeed computed correctly. A point to note from this simple protocol is that the knowledge error copper is inversely proportional to the size of the challenge set. Therefore, to achieve inverse polynomial knowledge error, we would need to instantiate this protocol with a polynomial-sized challenge set. Next, let us examine how a knowledge extractor could be constructed for this schnoll-like argument. First, we recall the verification equation, which looks like this. We construct an extractor such that it runs the prover twice on two different challenges, c0 and c1, to get two different communication transcripts, both of which satisfy the verification equation. And in matrix form, they can be written as shown on the screen right now. Next, the extractor would try to solve the following dual-venomon system for a vector z over the ring r. I call this a dual-venomon system because this system of linear equations is defined by the transpose of a venomon system defined by c0 and c1. Suppose the extractor is successful in solving the dual-venomon system for the vector z, then it can compute the extractor witness like this. Let us look at the second example of schnoll-like arguments, which is the lattice bulletproof system proposed in crypto 20. For this system, we make additional structural assumptions about the statement ay and the vector x, so that the matrix a can be split into two parts a0 and a1 of equal dimensions. And similarly, the witness x can be split into two parts x0 and x1 of equal dimensions. In this way, if the vector x indeed satisfies the statement ay with no slack, then y can be written as ax, which is equal to a0 times x0 plus a1 times x1 modulo q. Given these assumptions, the lattice bulletproof system goes as follows. First, the prover computes the cross terms between the matrix a and the vector x. That is, it computes the vector y01, which is equal to a0 times x1, and the vector y10, which is equal to a1 times x0. After receiving the vectors y01 and y10, the verifier would again sample a challenge c from the challenge set big c, and send this challenge to the prover. Finally, the prover would respond with x hat, which is computed as a linear combination of x0 and x1. Notice that the set of verification equations checked by the verifier is nothing but another SIS relation. But here, the witness x hat is of dimension half that of the original witness x. Therefore, by recursively composing this protocol a logarithmic number of times, one would obtain an argument system for the SIS relation with only logarithmic communication costs. Like in the first example, we noticed that the knowledge error of this protocol is again inversely proportional to the size of the challenge set. Therefore, if we want to achieve inverse polynomial knowledge error, we again want to use a polynomial size challenge set. As before, let us look at how a knowledge extractor can be constructed for the lattice bullet proof. First, recall the verification equation shown on the screen right now. This time, our knowledge extractor would run the prover p three times instead of two times on three different challenges c0, c1, and c2 to obtain three different accepting communication transcripts. In matrix form, these communication transcripts would satisfy the verification equation shown on the screen right now. Then the knowledge extractor would try to solve the following three-dimensional dual-vendaman system, again for a vector z over the ring r. If it is successful for finding such a vector z, it can compute the witness extruder as shown on the screen now. You might have already observed a pattern that the knowledge extractor is always going to solve a dual-vendaman system of a certain dimension. Therefore, a natural question to ask is, for what challenges c0 up to ct minus 1 and slack s is the following dual-vendaman system solvable over the ring r? Our observation is that, if for all i from 0 to t minus 1, the slack s is divisible by the product of ci minus cj for all j not equal to i, then the dual-vendaman system is solvable over r. This motivates us to define the notion of st-subjective set. We say that a set c, which is a subset of the ring, is st subtractive. If for any t-subset big t, which is equal to c0 up to ct minus 1, it holds that, for all i from 0 to t minus 1, the slack s is divisible by the product of ci minus cj, with j running through 0 to t minus 1 except for i. Furthermore, if the slack s equals to 1, we simply say that the set c is subtractive. Before I move on, let me say a few things about the relationship between subtractive sets and secret sharing. We notice that if c is an st subtractive set, then for any t-subset big t of c, not only is the dual-vendaman system defined by t solvable over r, but also the vendaman system, which is exactly the system of linear equations that one would solve when performing polynomial interpolation. Therefore, it is natural to see that if the set c is st-subjective, then one can use c as a set of evaluation points when constructing t out of n secret sharing over the ring arm. To summarize, here are some sample implications of st-subjective sets. First, if we have an s-free subtractive set of size n, then we can construct a lattice bullet proof with slack s and knowledge error 2 over n. On the other hand, if we have an st subtractive set of size n, then we could construct lattice-based t out of n threshold primitives. At this point, hopefully you are convinced that st-subjective sets are some useful objects. Therefore, the remaining challenge is to construct large st-subjective sets, and by large I mean polynomial size, with small slack s over some interesting ring arm. For example, in lattice-based cryptography, a popular choice of r would be a cyclotomic ring, which is the set of rational integers c adjoined with a primitive m-fruit of unity, c to m, where m is polynomial size. To sum up, here are our results of subtractive sets over cyclotomic rings. First, for power of two cyclotomic rings, that is when m is a power of two, we can construct a family of st-subjective sets of size n for a wide range of s, t, and n. For example, we can construct two free subtractive sets of size m over two plus one, which use a bullet proof with slack two. On the negative side, we show that it is impossible to construct a family of two t subtractive sets, where the family is parametrized by m, such that the mth set, cm, is of size strictly greater than m plus one. Moving over to prime power cyclotomic rings, that is when m is equal to a power of a prime p, we can construct a family of subtractive sets without slack of size p, which yields bullet proof with no slack. On the negative side, we prove a matching impossibility result, which says that there is no subtractive set c of size strictly greater than p. If we instantiate the lattice bullet proof system with the st-subjective sets constructed above, we obtain lattice bullet proofs with better parameters. The improved parameters are partly due to the use of better subtractive sets and partly due to a more careful analysis. On the negative side, we show that for schnollike protocols, it is impossible to have both small knowledge error and small slack. We see this as a technical barrier for constructing better lattice-based argument systems. Finally, we also demonstrate how st-subjective sets can be used to construct threshold primitives, using distributed pseudo-random functions as an example. Before going to the technical detail, let me lay down some mathematical background and intuition. In essence, the results in this paper critically rely on the presence and absence of ideals in the ring R, and therefore I should briefly talk about ideals. For a ring element c, the ideal generated by c, written as bracket c, is the set of all ring elements which can be written as a product between c and another ring element. In other words, the ideal generated by c is the set of all ring elements divisible by c. Using the language of ideals, the definition of st-subjective sets can be rewritten as follows. We say that a set c is st-subjective. If for any t subset, big t of c, and any element, c in big t, we have that the slack s belongs to the ideal generated by the product between the differences between c and c prime, for all c prime in t which is different from c. Let us think about intuitively how hard is it to construct large st-subjective sets for a small slack s. Notice that to construct such a set, we want lots of elements to divide the small slack s. Suppose our ring R is the ring of rational integers, z, it is going to be difficult because of the following observations. For example, if s is equal to 1, we notice that the only invertible elements in z are negative 1 and 1. These are the only elements that divides s, which is 1. Suppose that s is slightly bigger, namely that s is equal to 2, still the only factors of 2 are negative 2, negative 1, 1 and 2. Over a cyclotomic ring, however, the situation is a little bit different. For example, in a power of 2 cyclotomic ring of order 2 to the l, the element 1 minus c to the power k is the factor of 2 whenever 2 to the l does not divide k. Similarly, over a prime power cyclotomic ring of order p to the l, the element 1 minus c to the k over 1 minus c is invertible whenever p and k are co-prime. With these observations, we see that it is quite a bit easier to construct st-subjective sets over a cyclotomic ring than in the ring of rational integers. With these intuitions, let me present to you a flavor of our technical results. We first focus on the simpler setting of prime power cyclotomic rings, that is when the order of the ring is p to the l. Consider the ring element mu k, which is defined by 1 minus c to the power k over 1 minus c, as introduced in the previous slide. We show that the set c, which consists of mu 0, mu 1 up to mu p minus 1 is subtractive. To see this, let us recall that the element mu k is invertible over the ring r whenever p and k are co-prime. Next, we realize that for i smaller than j, the difference between mu j and mu i can be written as c to the power i times mu j minus i. Since both ceta and mu j minus i are invertible over r, the product of them is also invertible over r. On the negative side, we show that there is no subtractive set c of size strictly greater than p. To see this, we first realize that the ideal generated by 1 minus ceta has p cosets and the element 1 does not belong to this ideal. Suppose there exists a subtractive set c of size strictly greater than p, then by the pigeonhole principle, there must exist distinct c0 and c1 in the set c, such that the difference between c0 and c1 belongs to the ideal generated by 1 minus ceta. However, by assumption, we know that c is subtractive, which means that c0 minus c1 is invertible, which implies the element 1 belongs to the ideal generated by 1 minus ceta, which is a contradiction. Next, we move to the setting of power of two psychotomic rings, where I would like to show you the following theorem. But since there are too many variables to worry about, let us consider the following special case, which says that the following set c is 2-3 subtractive, and the set c consists of the element 0, 1, ceta, c2 up to ceta to the power Euler phi of m minus 1, where since m is a power of 2, Euler phi of m is simply m over 2. To prove this theorem, we first notice that the element 0 is given to us for 3 because all the other elements in the set c are invertible, and therefore the differences between them and 0 are invertible as well. Next, without loss of generality, let us consider a set t of 3 elements, which can be written as ceta to the A, ceta to the B, and ceta to the C, where A is smaller than B and B smaller than C. Next, we want to show that the element 2 belongs to the ideal generated by the product, ceta to the A minus ceta to the B, times ceta to the A minus ceta to the C. And for convenience, we call this ideal I. First, we notice that ceta to the A is invertible, and therefore the ideal I is equal to the ideal generated by the product, 1 minus ceta to the B minus A, times 1 minus ceta to the C minus A. Next, by some routine calculation, we show that the exponents B minus A and C minus A can be taken out so that the ideal I is equal to the ideal generated by 1 minus ceta to the power of the even part of B minus A plus the even part of C minus A, where the even part of B minus A is the largest power of 2 which divides B minus A. Finally, we make two observations. The first is that the element 2 belongs to the ideal generated by 1 minus ceta to the power phi of M. And second, the sum of the even parts of B minus A and C minus A is at most phi of M. And therefore the ideal that the element 2 belongs to is just a subset of the ideal I. Therefore, we conclude that 2 is also in the ideal I. As the final result in this talk, let me show you that there is no family of 2T subtractive sets parametrized by M such that the M set is of size strictly greater than M plus 1. To show that there is no such family, it suffices to find one single M such that there is no 2T subtractive set of size strictly greater than M plus 1. For this purpose, let M to be a power of 2 greater than 4 such that M plus 1 is a prime. These primes are called fermat primes and currently known examples include 517, 257 and 65537. For any of these M, we notice that any factor ideal I of the ideal generated by M plus 1 has exactly M plus 1 cosets and the element 2 does not belong to the ideal I. Suppose that C is a 2T subtractive set and the size of C is strictly greater than M plus 1. Then by the pigeonhole principle again, there must exist two distinct elements C0 and C1 of big C so that the difference between C0 and C1 belongs to the ideal I. However, by assumption, we have that C is 2T subtractive and therefore the element 2 is divisible by the difference between C0 and C1, which means that 2 belongs to the ideal I, which is a contradiction. To conclude, in this work, we formalized the notion of ST subtractive sets. We applied them to schnolleg arguments and threshold C per sharing. We constructed polynomial-sized ST subtractive sets over cyclotomic rings with almost matching impossibility results. We obtained improved lattice bulletproofs by instantiating the construction with better ST subtractive sets and finally we showed that there is a trade-off between knowledge error and slack for schnolleg arguments. For the full technical details of our results, I encourage you to check out our paper which is available on e-print or if you find the paper a bit too overwhelming, you can also check out the blog post.


It's the last day of the DHIS2 Digital Annual Conference. First, we've got our final round of expert lounge and meet-and-greet sessions on GABA, where we'll be looking at DHIS2 communications and the community of practice. Then we kick off our first plenary of the day, what's next with DHIS2, where we'll be looking at the DHIS2 software roadmap and some of the features that are planned in the next version of DHIS2. We've also got our web and Android app competition, where we'll see presentations from our competition finalists, and you can vote for your favourite custom DHIS2 app. Closing the conference will look at a snapshot of the conference, where we'll see who's joined and where they're from, and we'll have closing remarks from Professor Kristen Braugh. Remember, you can find all the info on the sessions on our schedule page and read about the presentations on the community of practice. We look forward to seeing you at the last day of the DHIS2 Digital Annual Conference.


Hello, everyone. I'm here to talk about one agile transformation and two beers. So has agile transformation just become a meaningless buzzword? So a lot of these transformations that we see are really getting further and further away from people. And companies, organizations are just copying what others are doing, and maybe also just because people are not being powered to share what they really think about the transformations. And some of the things are just top down. Actually, this is something maybe it's not so good in reading. I found this last night. And five ways to screw up an agile transformation. First, call it agile transformation. That is a nice one. Write an agile law book. Call it agile playbook. Create a department of agile enforcement. Call it an agile center of excellence. Run an agile conversion scheme. Call it agile training and coaching. And keep outing the latest agile buzzwords. Call it OKRs. So I was looking at LinkedIn last night. I found this actually interesting. So why not put it here? Because I think it resonated with me. And the reason it resonated, and I think this makes some sense, is because McKinsey and company has a study that shows that 70% of all transformations actually fail. And if it shows that 70% of all transformations would fail, why we continue to do that? Why companies want to embark in this transformation training and do this? I would like to ask you some participation if you have the mobile phones. Scan the QR code or go to slido.com and use the number. So put your phones up and share your thoughts about why do you think transformations in general would fail. And hopefully, if you are able to go there, and things will change there. And I think you can input more than one. So go and we'll see things changing. And these are not comprehensive or total list of the study from McKinsey showed. So this probably, there are more. But I wanted to go with this. So OK, so things are changing. And lack of shared vision is winning. Then lack of alignment, lack of engagement, lack of aspiration. So when we think about why this can fail, right? So lack of aspiration, just maybe the company doesn't even know where it's going. So if you don't know, why would you embark in a transformation to a place where you don't know where you're going? Lack of a shared vision, which seems to be winning, but still votes coming in, is maybe the company knows where the company is going. But not everyone in the company has that same understanding of the vision where it's going. Lack of alignment, maybe the company is not aligned where it wants to go. And then if you do a transformation, probably you will fail. And lack of engagement, the company knows where it's going. But the people don't believe in it and they are not engaged. So most likely the company will fail. And we still have some votes. So great, lack of shared vision won here. So the lack of shared vision would be that the company has a vision, right? But not everyone understand the vision of going forward.  And the lack of alignment is just, you might have a vision and people don't really agree with that, disagree. So it could be something in between those lines, right? So for myself, I don't like the word transformation just to start with, right? So that's my personal view. Ideotransformation, to me, I always think of, I don't know, transformers, right? So someone coming, yeah, it's going to transform. So I don't like personally, right? So the word transformation and ideal transformation. When we look at what transformation can mean, a transformation is a dramatic change in form or appearance. An important event, like if you're getting married, getting a driver's license, that can cause a transformation in your life. A transformation is an extreme radical change. So when we are talking about transformation, to me, I have this view of before and after situation, right? So before your bed, and then after, oh, you are great because you were transformed, the company was transformed. And I don't see it that way, right? So a lot of things that happen is like you are talking about implement these, go with that framework, right? Maybe you heard of the Spotify model, right? So, oh, we should use the Spotify model. Let's put the Spotify belt on and we're gonna be agile in three months. And that's really not how it's supposed to work, right? I am a drummer. Anyone here play drums? Anyone? We have one drummer, two, three drummers. So you are excluded from this. Anyone else playing other instruments other than drums? Okay, so you have some musicians here. So I'm a drummer. And maybe I will transform you in the best drummer in the world, right? So a negative, right? So if I would say to other people here that would, I would say I will transform you in the best drummer in the world. Probably people would be, maybe not, right? Maybe I don't like drums that much. Maybe it's gonna be difficult, right? Some people here even might not like music. I don't know. So to come in here and say I'm gonna transform you in the best drummer in the world, it's not gonna make sense. My transformation will definitely fail, right? So I have to, I need to have other ways to make sure whatever I want to go after, I'm able to do. And I always think of this as a journey, right? So when we talk about agile, agile is not the destination, right? So agile is a pathway forward that we use. We can call it agile journey. We can call it continuous improvement journey. We can call it whatever, right? But the problem is that a lot of times agile is set as the destination, but that's not where we should be focusing on, right? So it's really the way that we go forward. These are some of the reasons, and this comes from the 15th state of agile report. These are some of the reasons that companies initiate agile transformations. And I think they are all valid reasons. The problem with this is when executive leadership, CEO, CTOs, they decide what needs to happen and it becomes top down. So in my previous company, that was actually the scenario, right? So I remember one day receiving an email from CEO or someone really high on the leadership, we are embarking on an agile transformation. There were reasons why we should embark on the agile transformation and the bottom line was now you employee, you have to do all these agile courses and you have to finish by this date and we will be tracking you, right? So yeah, and I remember we had the pressure to do all these online courses and after finishing the agile transformation was done. Of course, I think I left before they finalized, but I heard it was not that successful, right? So if it's top down, you are not engaged, you will do and then the company or the higher leadership will ask, oh, why did we fail? We don't know, right? So these are some of the reasons that I think are reasonable to start a journey but not from top down, right? So everyone needs to be really involved. And I found this very nice article talking about the 10 main challenges of this transformation. So I'm not gonna go through all of them, but I'm gonna focus on the four on the top there. Just because I see that these some connections to what we actually have been trying to do at Red Hat, right? So the first one that is a challenge is the lack of agile understanding. If we look at the agile manifesto, agile manifesto deals with software development. It doesn't necessarily deal with business outcomes. And a lot of times the transformation is really focused on engineering, product delivery. We need to provide products faster to the market. That's all good, but if we just solely focus on that, we are increasing our chances that any effort in transformation will fail because maybe we are forgetting that it's not only the engineering side. Now we have sales, we have marketing, we have a whole lot of more people that should be involved and focusing on only one specific part might cause it to fail. Another part of lack of understanding is when the higher leadership just says, yeah, we're gonna go agile, but they don't understand what really that means, right? They, because maybe they saw it elsewhere and then you will see buzzwords coming back and forth. No, and no, this framework and, you know, oh, the velocity, oh, the sprint, all that. And that just shows a lack of understanding and that's a lot on agile coaches to actually coach and mentor leadership on actually the correct understanding of what agile really means. If that doesn't happen, then, you know, the so-called transformation, it could be set to fail in the long run. Another challenge is related to the organizational culture, right? So the mindset of the organization, the perspective, the vision, everything is in there. The penny on the culture going this agile can be very, very difficult, a radical change and people will push back, right? And going in this agile road, we will definitely break that silos mentality, right? So, yeah, my team does this, we are great at this. I don't need to talk to anyone else because we are great at this or don't talk to me and we need to actually expand when we are going agile, right? So we are trying to make sure everyone improves and by breaking that silos, it's a one step further to actually being successful and when we don't think about how the organization is, how the culture is, it's something that could definitely make the journey or the transformation fail. Another point or challenge that I want to highlight is just copying other agile transformation projects and that's a good one because how usually agile transformation project starts. So the company, the CEO or whoever, a higher addition, oh, we need to go and start this agile transformation project because we need to go agile and then what they do, they go to the market and they hire agile consultants, right?  Most likely using a Gantt chart, very project management like with the deliverables and the milestones and all that and just, this is already a sign that whoa, that's not going to the right direction but this is how usually it starts, right? So there is a consultant coming in and the consultant knows everything, the consultant will share the roadmap, will share the best practices that the teams have to follow and there will be specific frameworks, specific training that only that consultant is certified to deliver. So and the thing is just copying what others are doing, just copying whatever is the model like Spotify, right? For example, it just simply doesn't work, right? Companies will have different requirements, different priorities, different ways of working and if we don't acknowledge that, if we don't understand that, we are really going to fail in the long run. And the last one I want to share is restricting agile to pilots, not actual pilots but in this case, pilots, we are starting an agile transformation but first let's focus on a few teams or just in this area. Let's do everything in this area and let's get the results, right? So because the leadership wants to see results, leadership wants to see metrics just for the pilot teams and see if it works. No, agile is really about thinking, agile is really about trying experiments, right? Trying improving quickly and hopefully easily, easily. So restricting this just to a few teams would be not so good way to go. What it could be differently here? Maybe all teams are shared and understand the principles behind what we are doing and they go and take their road with this and then we gather and constantly we see how they are doing. So the idea of just restricting agile to pilots and we do that for six months and we gather results for six months and then we start to think about how we're gonna do elsewhere. It wouldn't be a good way. I think if we start looking at it from kind of a innovation adoption curve, right? So understanding who are the innovators, early adopters of this, it would be nice because then they would come and then they would help put this effort forward of the agile journey, continuous improvement journey and then if it's successful from them, the rest will follow, right? So the early adopters, late adopters. So this is really kind of a way of working with the innovation curve and at Red Hat last year, we started this effort of going in the continuous improvement journey and a few people got together, right? So executives, senior managers and a few subject matter experts to kind of discuss the idea, how would this journey look like at Red Hat? And we used the open decision framework for that and the reason for that is that, well, it wouldn't be something top down for sure, right? That definitely would not work at Red Hat. So we used the open decision framework and what happened is after some time, right? So the managers, the executives, subject matter experts, they came up with a vision for this journey. They came up with roles, responsibilities, training and all that, everything documented, but the point that was really great about it is that it was an engagement. So everyone was engaged since the beginning, right? So everyone knew what's going to happen. Everyone was really empowered to provide feedback and we received actually a lot of feedback. I'm gonna make some assumptions here that what I remember is something like 600, so all in the document, 600 comments from 600 specific people commenting back. No, this doesn't make sense. Maybe you should think about this, right? So from everyone in the company and what happens is that we receive that feedback and we are able to adjust that journey based on that feedback from people, right? And this is something that in my previous company, as I mentioned before, I didn't see. It was like, you're embarking on a Nigel transformation journey, watch the videos, right? So in this case, it's very different. We are embarking on a transformation journey together. This is what we came up with initially. Now, what are your thoughts? And we received a lot of feedback. It doesn't mean that every feedback we're gonna do something but a lot of the feedbacks, like we incorporated and we changed the way we wanted to move forward. So to me, that was really good, right? And in the end, we get the buy-in from people. No, not everyone for sure, right? So they're still, I'm sure there are still people that don't like what is happening but we definitely increased our chances of people buying into the idea and thinking more positively of our way forward. So this is, I think to me, this was really, really good. So this is just something that we have been doing at Renhead and is it perfect? I would say no. It's not perfect. And if I wouldn't have a bigger no, I would put it here. Oh, I have a bigger no. So no, no, it's not perfect. There are a lot of things that could have been done differently that can be improved. But we are in the beginning of the journey and we will see how things will turn up. At least from my perspective, having people engaged in the conversations and them providing feedback, I think it's already a win. Maybe next year I'll be here talking about the success or failure of what we are trying to accomplish with the teams. So the takeaways from everything that I wanted to share here and with regards to ideal transformation or improvement journey, whatever you wanna call it, like moving one step ahead on just improving. When we talk about these changes, right? So engagement is really key. We want people to be engaged and participate in what we are trying to go forward, what we are deciding. So that's why feedback is really gold, right? So we want feedback from people. It doesn't mean that we are going to incorporate all the feedbacks, but it means that, yeah, we are acknowledging everyone. With that, we have the buy-in, right? So the buy-in is really essential. Not 100% of the people will agree with everything, but at least if they know that they are heard, know that they see that, okay, they heard me, they are not going with my idea, but they are listening to me, they pay attention. So this is really good. And if you have a framework to engage people, right? So with the transparency, like open decision framework, that really helps give the people and the teams ability to just see how things are going. That's what we are trying to do. As I said, we started last year, and we are, I think we're heading in the correct direction, but we'll see. And that's it. And if you want to talk more about agile transformation, or if you have ideas like, is agile dead or not?  Come tomorrow, I have a meetup that I wanted to hear from people, is agile dead? Is it not?  We're not gonna fight, hopefully, but we're gonna have some nice conversation around it, so it's gonna be tomorrow at 12.30. And of course, you have my LinkedIn if you want. And that's pretty much it for this talk. Any, I don't know, questions, comments? Let's go ahead. Yes. And what? Yeah, right ahead, yes. So you say, yeah, do this happen? Mm-hmm. So the question is if we saw, yeah, so the question to summarize, if we saw from the feedback that we received, what changed? That's kind of a, yeah. Okay, so go into more detail how we did it at Red Hat. So there was this small team of managers. We actually created four specific areas for this so-called transformation or journey, right? So we had a specific vision document, what we wanted to achieve, and actually we had values for that part, right? So what were the values for us to go in this journey? We had roles and responsibilities document highlighting all the roles that would be important in the journey. We had another part of training. So this was the initial idea. And the part of the feedback is actually directly on this, right? So there were people saying, and in my case I was more working on the training and enablement part. They were saying this training that you're highlighting here wouldn't make much sense because of this. Maybe we should consider this training instead. Or I have an idea that maybe we could do a training like this that would help in this or that scenario. And these were actually Google document comments. People were incorporating. And what we were doing is that we were reviewing everything. And once we would review and we would say, well, that actually really makes sense, right? So, and because we were so into it that when someone came and shared something with us, we said, oh, this really makes sense. So that's where we were talking to people that provided feedback and kind of getting together. So maybe help us align this in the document. And then what happened is that we ended up with, I don't know even how many versions of the document until we would say that's the final one and share like brother. Does it help answering the question? So in our case, so how long does it take? No, no, it doesn't. So how long did it take and did it satisfy everybody? No, it didn't satisfy everybody. Not everybody in the organization actually put the time to provide feedback, all right? So that's one thing. But hopefully people knew because everyone in the company received an email saying, please provide feedback. And I don't remember if it was a month or something like that where we let it open for people to provide feedback. We were reading in the meantime, adjusting what is needed. So that's where every feedback that we wanted incorporated would change and we would make a new version of the document. And after we received all the feedback and we decided, yeah, these are the good feedback that we're going to incorporate. Then we had, all right, so this is the final document from inputs of everyone. This is the final thing that we are going. And of course, when we did this, the people that were not paying attention before, they came with other, ah, but I have a feedback now. And I remember that we managed a couple of feedback after as well just to make sure everything was fine. So we had that document. That's where everything related to our journey is, right? So if everyone at Red Hat, oh, where is, where we are going as a global engineering in this journey? Well, look at the vision, right? So look at how you are defining the roles and responsibilities. And one good thing that we are doing right now is kind of a baseline assessment with the teams based on the vision, based on the values that we put there to understand where teams are. And it's one assessment and for all. So it would give us a very good perspective where teams think that where they are and would provide us ways to see where, you know, might be possible areas for improvement at the organization level, right? So this would be helpful for us. Does it help? That's great, all right, cool. Yeah, look, let's see. Yeah? So the way I see it, so with my knowledge, how would I, if I would have a company, how would I embark on an agile transformation or a journey? I would see it from the perspective of what I mentioned before, the innovation curve, right? So if I know who are the innovators or the early adopters in my company, I know that they will want to try or they are open to try something different, right? So it would be easier to try everything with them first. So trying, you know, what we want to change and getting ideas from these innovators, early adopters, how they see this changing, that would be a good way forward. And it's not like, we're not talking about, you know, implementing this, it's like ideas from them. And getting the ideas from them, I would see is that they would experiment, right? So that's what I do is about trying to experiment. We would have some success from that, some failures, of course, we would learn from it. And after this is starting to kind of ramp up with these early adopters, we would go, then the early majority of the company would start to look at, oh, what is happening with the company? Why these teams are working so well? We need to kind of see it. And then it's kind of ramping up from there. And it's not like we are calling, yeah, the agile transformation started. As I said, it would be a journey. It wouldn't have any sort of time frame, right? So as sometimes we see, oh, this agile transformation will take, or you need to complete it by nine months. You need to complete it by 12 months. That doesn't work, right? So transformation work or this journey would never, in my view, would never finish because, I mean, we can always improve, right? So I would think about as the innovation curve focusing on the innovator's early adopters, work with them on experimenting new ways of working, try to improve. And the theory of this innovation adoption curve is that then people would follow. People that would go in the end, they would never do change anything. But maybe they are not our focus, right? So if we focus on the middle and they change, then it would be, I think it would be successful. There's a question there. So the question is what is the measure of success and if we are gathering data? So great question. Hopefully the measure of success of any agile transformation, it's not velocity or story points or anything related to that, so that is clear. It's really more on, is the customer getting the value that it needs? And from my case, so this is the last question I'm out of time, from our case, with this baseline assessments that we are doing, we are gathering the same responses for 100 teams. So we are getting there. And we will understand from these assessments that we are doing, what areas from these teams we are having problem as our organization, right? So as an example, is it training that it's needed, right? So they don't understand what value is, right? So, and then this is kind of the way that we are looking at metrics. Nothing like, I know, or velocity or anything like that. So we are gonna have these assessments understand where in the organization we have maybe possibilities for improvement. Let's say a lot of teams don't understand if they deliver value or not. That would be an opportunity for us to, okay, so let's investigate further. Let's see what we can do to improve so that teams understand how to deliver value or if they deliver value or something like that. Does it help answering? All right, cool. We are out of time, so thank you very much for your time. And if you want, talk tomorrow. Thank you.