How you doing? So I'm Big Easy. I'm part of DC 217 and we're putting together the Ethics Village and when we were doing the CFP it was my privilege to read the CFP of this man, Enore, who happens to be, he has a conference in Germany and I did my first public talk at his conference in Germany over a decade ago. So it's my privilege to introduce Enore from troopers in Germany and the circles now complete Enno because Enno is a first-time DEF CON attendee and this is his first DEF CON talk and we chose him to do the keynote to kick off the village so thank you for coming Enno very much. Thank you for the warm words. Actually it's not only my first DEF CON talk, it's my first talk on ethics at all so I'm a bit so to say nervous but happy to see you so many of you. Just a quick intro who I am. I have been in infosec in different roles since 97. I have a technical background which means I'm from large-scale networking from carrier space. I've given a number of technical talks so that's my domain usually and one thing that could be of relevance for my talk today is that I run a company since many years and within that company we have an ethics committee which I installed and I will later lay out why I did that and this is one part or this part of one of the case studies. The purpose of today's talk is say free-fold that is to make clear that once say ethical questions ethical I use the European plural of dilemma which is Dilemata so once a Dilemata occur how to handle those so to understand there's different ways on a maybe structural level to tackle them then to like provide a I wouldn't say guideline but maybe some questions to us once you face a dilemma which say with steps of reflection to go through and then to make clear all this is not easy it's not just this is not about as Richard already laid out 15 minutes ago taking easy decisions that is so just that is the main the main intent here to educate you a bit but also to make you think in a certain in a critical way about dilemma dilemma you might face so where is ethics relevant for infosec practitioners pretty much everywhere there is spaces which are going to be discussed later on in more detail vulnerability disclosure exploit sales there is some debate already going on but other than that whenever there is an intersection between infosec and humans ethical questions might come into play again I will have a number of case studies later on where this becomes more clear some disclaimers in advance I don't have a formal education and ethics actually I have a formal education and literature which you will see later on in my slides somewhat but and during my studies in the 90s I worked with computers and I worked with networks this is how I got into this but from a formal perspective French and German literature as my background second disclaimer ethics is not something which you should discuss on Twitter I tried that's not the right format actually for the type of questions we are going to discuss today and the last one there is a there is a guy who I'd like to mention here Ben Zevenbergen I owe him a lot of the things I talk about today he said Princeton at the moment so if you want to look up his walk and that would be say going further than what I do here in this talk from a say definition perspective what is ethics or what is actually the part of ethics that I'm going to tackle that is practical ethics this is a formal definition which I took from from the work of Ben which he started at the Oxford Internet Institute the task of practical ethics is to identify more problems so it's about problems about dilemma to clarify and then to clarify say what is what are the values affected what to reflect on how possible actions could look like this closer vulnerability refrain from the clothes disclosing disclosed it via certain channels identify the the causes of action and then choose one which ideally reflects best say the the world use and the decisions you took earlier on so in short it's about doing the right thing based on a certain type of reasoning and again it's about dilemma time if it was about easy situations if the things that we are going to discuss today are which are being discussed in in ethics in general could be solved easily then it we wouldn't all wouldn't need this ethics domain it would be sufficient to have laws or to have stuff like 10 Commandments if that would be sufficient to steer human life and decision-taking then I could stop here but it's about dilemma time dilemma time means it's not easy this is I'm very keen on stressing this many times and in the talk this is not about the easy path and taking easy decisions as you will see later on the case studies from a formal perspective just to give you an idea of how to tackle say certain types of questions I will lay very quickly lay out three principle three approaches of the of the within the ethics world which is a which is a huge one there was many schools of thought there was many approaches there's different frameworks there was different types of terminology I'm going just to tackle very quickly three of those as those might be important for the for the discussion later on at the first one is so-called consequentialism there was one flavor of that which is UT literalism in short it's about say the end justify the means so the approach of a consequentialist would be okay here's some possible actions that I could take parts of action let me identify what has the highest benefit whatever its benefit might be or the spending but how this benefit could be identified and then choose the path that provides the highest benefit that could be one possible approach the problem with conflict consequentialism is in the end of the day you can justify all types of actions with that you can come up with oh I have to like torture people or I have to shoot down the plane because in the end of the day this is better for everybody or taking things into account doing this has overall for society higher benefit than another course of action that is actually the main problem of this one let me already state that in the technical domain which has a high highly represented at DEF CON there is always a temptation for say consequentialist line of reasoning we are used to solve problems to tackle problems and to to write problems by the body by the result and a consequentialist a UT literalist tackles things from a result perspective to illustrate a program of consequentialism nutshell say if in 15th century South America it hasn't rained for a number of weeks a possible cause of action was well we should come up with a sacrifice because that's good for everybody when it rains okay maybe one or two young persons might not agree with that statement but well this is a very consequentialist line of reasoning then there is a completely different line of thought school of thought that is called a deontology deontology usually works on the basis of very strict rules like do not torture period that would be a deautonomist approach to a specific question the problem here is this might actually lead to situations which can can have very bad consequences say you followed had a deautonomist approach of do not lie and in World War Germany you hit some people in your in your in your house and Gestapo turns up and ask you okay is there anybody hidden in your house and you follow that do not lie approach that could have very bad consequences so there is some the problems with this one as well but those two are in say from a certain perspective the main antagonists and then there is one which has gained quite some ground in the in the last say decade especially in the information technology domain which is called principlism which tries to identify okay what are the common values to for society or for groups of people and then walk along those values and say there is a well-known flavor of principlism works around okay there was a well your autonomy then we should take into account when discussing ethical questions what's the actual benefit for of causes for actions let's talk about do not cause harm do not be evil would would be a typical of a principlist approach and justice like when be fair in weighing options there is a say in the information I have to use a thing that I'm not used to use some but okay I'm working with this in the information in the IT domain there is a well-known say document the so-called mental report guiding principles for information communication research in the information and communication domain authored by Dave Dietrich and another guy and looking at the table of content you already get an idea what this say ethical framework is about so respect persons to get their informed consent try to identify who benefits from a from an action balance this with possible risks and in the end of today try to be fair when looking at the options this is an IT flavor of principlism so now once we want to do this a bit more in real life in general it's a good approach and this is one that I want to to lay out to you to once you face a dilemma very simple said okay try to understand the dilemma try to write it down try to identify what what actual once you have a feeling well this doesn't feel right and I should maybe shouldn't do this or your sweatshirt had okay we are in general I well maybe I should perform a well ability disclosure but in this case well maybe it might not be the right cause of auction try to write it down get effects again this will be I have some case studies this will be very important later on try to identify who's involved and what are the stakeholders of a decision and what are the values affected and then evaluate and this is often overlooked evaluate alternative options evaluate okay what if I didn't do this or whether any other approaches to get the same to gain the same insight say in the once you perform research projects involving stuff like pot scanning or so is there any other ways to find out a specific thing you're interested in many people don't do that actually and the last one I will skip this for a second oh no I will not skip the last one this one of those from my perspective my experience very good approach to when you go through options like I could act like this so I could do another thing is to ask yourself if I did the following would it reflect me as as a person I would like to be I mean you could say can I still look into the mirror but this is a bit a bit more formal approach my personal value system is an action which I perform aligned with my personal value system obviously you have a feeling for that already when you come up with well this doesn't feel right but still in the end of the day ask yourself is this consistent to what degree is this consistent with how I want to be and I want to act and maybe even act as a sample as a leader and of course evaluate in hindsight again this very often doesn't happen so this is a bit the very generic thing how to tackle an ethical dilemma facts stakeholders the values the causes of action choose one and reiterate at some point of time there were some additional things before we get into the actual case studies I would like to to provide as an advice be aware there in that in many questions and many say situations there might be a power imbalance or a knowledge imbalance especially when people like from a technical domain decide on well I should do the following because that's good for everybody how do you know it's good for everybody how do you know that everybody else is sharing your perspective that you maybe as a person from the Silicon Valley has so you might know more or you might be in a more privileged position than others you perceivingly speak for try to keep this in mind try to ask yourself in a specific situation is there a knowledge of power in the lens the sex one over this the next one keep in mind that the internet is a say social technical system which was designed by a specific or many parts of the internet very signed by a specific group of people and so things happening in the internet might reflect certain understandings and certain well your systems try to keep this one in mind maybe try to avoid always keep in mind your cause of action might act as a sample for the better or for the worse for others there was a common there was a very well-known ethical say the lemma thought of the so-called Kana botnet some of you might notice some where I think it's in the terms like eight years ago a botnet was created to map the internet and the guys who did this were actually arguing like oh this is good for everybody you have an in a map of the internet and they well they compromised systems for that which maybe you can come up with in an ethical reasoning perspective is the proper cause of action maybe not the thing just is this might set a precedent and keep this in mind I have been just to give a very quick example here it's as I mentioned I've given a number of talks in many conferences so I attend conferences as well not just until today because my daughter's birthday is in early August this is one of the reasons but at CCC I was once in in in a talk where a guy discussed how to compromise like the airport the airline codes like and he wasn't making fun of people well those guys are so dumb to put their tickets on Instagram and once you get the you get a code you could lock in or you could use their minds whatever I'm not a big fan of say this type of making fun of people anyway but there was different styles of speakers and you might find mine dry for example but the thing is in that talk my my at the time 13 year old son was sitting next to me and the first thing he did when getting back to the hotel room was like going to Instagram looking at airport tickets and hey the guy on stage did this so I'm I can do this too well you get the idea think about if a cause of action that you jump into it could be a precedent in one way or another for somebody else and this was mentioned during the ask your friend session as well I suggest to be very careful with analogies between the physical and the digital world like well if we do this this is like breaking into somebody's house I did my experience is very often analogies as do not really help or apply be careful with that one last thing be honest with your agenda obviously we are all humans and it's perfectly valid to have an agenda but well put it into the equation once it's about gaining you say famous speaker or getting money that's all these might be legitimate reasons and objectives of human action but be honest and put it on the table very often say in the in the academic internet research world for many years has been very popular to write papers based on numbers like we scan the internet forum IPv6 or open ports on that specific protocol and then we perform some some action to find out to find out which version of a protocol is used and this is a good contribution to scientific research if you look closer it turns out I mean the academic world they have their own incentives and their own ways like publications and being cited and quotations and all this that's a specific ecosystem when you look closer many of the things that have been have been done there might not have been necessary there might have been other ways of gaining the same type of information it was just well it's in many years it was a bit on walk to write papers of that specific type so everybody did it oh I scan the internet and I was entitled to do so as I'm a scientific researcher and obviously it helps to once you face a dilemma to discuss it with somebody who's not of you domain it's not of the technical domain actually in my case I'm not an overly say religious passion but I live in a small town and there was a that's a preacher I think a guy from the church who has an official position in the church he's a he's 80 I very much respect him and it already happened several times when I had a dilemma I just went to visit him and discusses with him as he has a very different and experienced perspective on the wall so that's it from the theoretical part now let's get a bit into the meat the case studies with one last warning before ethical dilemma to get through those is not an easy task if you have an answer like you you face a problem and you have an answer after five minutes you might have been doing it wrong you might have not have all the facts you might have not considered say the values in the stakeholders affected or you might not have say being self aware of your agenda case studies all the case studies I'm going to discuss have mostly they have happened in real life in our organization so I've been facing those and at some point in life let's start with the first one the organization which I which I founded and which I well then to some that we obviously represent here we do a lot of vulnerability research both in customer engagements and like as part of our company DNA and there was a situation where we have identified vulnerabilities in an alarm system in a type of alarm system which was sold in Germany in electronic shops like in the US that would be a best buy or a radio shake doesn't exist any longer isn't it but in these types of stores you can get an alarm system for your house or for your property and we found out for a commonly sold model it wasn't it wasn't exactly rocket science with software defined radio to actually correct the codes of communication which at the first glance doesn't seem so problematic from an ethical perspective but if you think closer about this would you disclose this we have at E&W in general we follow a mostly very conservative and since 10 years what at the time was called responsible disclosure I'm happy to see the time the time back there was a whole debate if the term responsible is appropriate or not but I'm not going to stick into this the idea is like have a defined time frame in form a window and after a specific time frame you disclose the things and you hope for that the window has some fix the problem in the interim the patches available in this patch can can be rolled out the thing is this disclosure approach might not work in this specific case very well getting the facts laid out at the first step that's not too difficult in this case we know the model we could identify okay with who are the stakeholders affected well anybody using this who else is involved well that's okay there's a users perspective there was a vendors perspective it wasn't really possible to identify the window as this was kind of OEM stuff sold again in popular electronic stores but the facts that were easy the values like okay is the harm versus benefit equation and who is affected this one becomes very interesting very quickly as looking from a traditional disclosure perspective there are some assumptions in this disclosure process which are vulnerabilities as close to the window when the producers a patch the patch can be rolled out and actually the people affected by the patch you know that the patches available and they can actually apply the patch that's the basic idea behind this like since 15 years when rainforest puppy wrote down the the first policy laying out your responsible disclosure idea there were some assumptions in that when that can be identified when the producer will produce a patch affected users get to hold of that and can apply it and many of those assumptions if not all which I mentioned do not apply here we couldn't really identify the window some well in Southeast Asia and in some country stuff was produced and it was sold with different labels on it throughout Europe so how to identify the vendor we didn't even couldn't really find out if it still existed well the stuff was sold so it was manufactured somewhere but that was already difficult but even if there was a patch how would the users know the patch was available and even if they knew like there was a public broadcasting all the following type of systems has a problem please show up in your electronics store if you are affected they will provide you an upgrade how would that upgrade be beyond the system so was traditional disclosure would have not worked here and what made the thing very interesting was the like harm and benefit equation as this affects well people's property so even if we had say perform and we happen as you will see in a second but if we had performed traditional disclosure and after like 90 or maybe prolonged that a bit make it 180 days publish the thing what if say some people would have known that not that have applied a patch and into their property was was broken into there would be harm caused real harm in a real world and that is one thing and maybe they would have showed up in the before in front of our company building with the fox like since you guys published this you have helped the bad guys to break into my house so the stakeholders and the well use affected this makes this a very interesting and kind of complicated case I will skip this so you might ask what did we do the thing is we try to identify by some channels in some ways the window this didn't easily work out we refrain from publishing the thing we had a free series blog post on like okay how to how to analyze wireless protocols with SDR and then here is say a case study but it was planned to have a third one on like okay the case study revealed the following and we never published that one it remained the draft in the in the block for for a while and then it was removed in a nutshell we did nothing which from a hindsight perspective that's highly unsatisfactory what we should have done but this is like five years ago we should have gone through a through a start nowadays that would be that the best possible action from from my perspective but at the time and at some point we like lost interest and we didn't follow up on this it was like it's still in our I think that there's the thing is still sold but we if we went public with this we would have to re-evaluate as the problem still exists and we were not willing to put time into this but there was an outcome which from today's perspective I'm not happy with that one second case study let's imagine you and this is a very typical type of engagements we have that large organizations bring us in like okay we plan to procure the following devices whatever those might be in this case a network security device can you have a closer look on this before we deploy this so there was a device and we stumbled across something which well you could consider this a backdoor a backdoor which looking closer you might get an idea like okay there is a specific actors behind this backdoor so how to handle this one this closest one funny enough the question was brought up in the similar question was brought up in the in the earlier session let's take a structured approach which I propose at a fast glance okay this might look like a vulnerability disclosure which looking closer it is not or maybe it is that's already an interesting question is a vulnerability is a backdoor vulnerability from the it depends on your perspective actually from the one who puts it in there it's it's not a vulnerability as it's a plant feature kind of from the ones not being aware of that are from the non-five-ice countries well it might be considered a vulnerability probably yes the friend of mine who leads a threat intelligence unit in a German highly specialized shop he uses to say one country's white list is another country's blacklist and there is a much truth in that what does this mean for handling this case I mean Germany is not a five-ice country so we are it's from the customer engagement the organization who made us look at the thing it's probably not what they want so this brings up some interesting questions and there is what made that the case or this type of cases much more complicated than they might already be then there is a thing a whole different alien in that moment type of value system is brought in as say confronting the entities responsible for the backdoor that would be immediately well we need this for national security this is needed to be to get hold of the bad guys in Syria or wherever which might be what might be a proof or not it might also just be used for industrial espionage whatever the thing is there is a whole different value system brought into the discussion which by the way applies as well once there is there's a lot of especially in the UK a lot of reasoning of measures surveillance measures which from probably a perspective many people in the room we would consider unethical but as always like oh this is this is meant to prevent a specific type of crimes against children and with that argument you kill everything which I mean I have three children myself I'm not against fighting crimes against children I just want to make you aware that this brings a whole universe of say questions to a technical or technology ethics debate and one has to be very very clear and aware of what is in the mingling of these when your systems actually produces again let's take the structured approach who are the effective stakeholders then again this is an interesting question as from a very simple equation we have a customer brought us in there is say probably a stakeholders here the entity who put the backdoor in it there might be thinking about it there might be other countries which are not five eyes which is again another group of stakeholders this raises some and you might come up with the line of reasoning well this is about internet security if those devices are deployed in the internet and the bad guys can whoever the bad guys are could compromise those this is bad for everybody's security in the internet but and this is why I asked that the question earlier when there was this oh would you report them if say a nation state actor compromised and an NGO the immediate answer of most of us would probably yes assuming and I'm not judging anything here I just want to make you aware of things assuming that the majority of people here in the room are US citizens what if well this is about a matter of national security for the US would you answer be the same so we have a conflict and this is a kind of classical conflict and I call this a conflict of scope where scope means okay what is good for the or bad for the internet maybe might be different once you look at it from a more narrow perspective something that could be good for my country and be bad for the internet as a whole whoever that is and these types of scoping and conflicts based on scoping we will face we will see there's this of course often there is no easy solution to this as to many questions the thing just is you have to be aware of this you must sit down and as part of your reasoning and your decision-taking process it's just to make this clear I'm not judging any type of outcome of your decision-taking process I just want to help you to have a structure decision-taking process if at that if the end of that one comes out well I'm a US citizen or I'm a citizen might be a German citizen I'm a family father whoever and that's that's why I decide or I think the proper cause of action is the following one that's perfectly legitimate you just have to be aware well maybe when I take this decision it's different it's another broader scope of stakeholders is affected in a detrimental way maybe so and then there is this thing what I mentioned values talk about from the principalism approach talk about autonomy talk about being beneficial talk about justice the the autonomy angle that's an interesting one for the back door how does say having a back door or not disclosing it affect the autonomy of all the organizations which might be affected if if like by keeping your mouth closed you you foster the practice that systems can be compromised by a nation-state actor which is maybe not in the interest of I don't know how many exactly how many countries are there in the world that say 212 or so minus five 207 countries might be affected in a way that is different from those from from five countries and 207 countries might be affected in a way that violates their auto autonomic decision-taking like informed consent think about a main law report there is the state there's this thing informed consent what about informed consent for the for the people who use this device which has a back door built into it so this was a was a quite interesting one and it serves as a nice example not so nice example as a very telling example why this principalism thing do not harm be good to everybody treat everybody fairly and in an equal way and respect the autonomy of human beings all this is nice I'm pretty much everybody in the room is probably willing to well check yes that's good but well the world is more complex you will see conflicts and you will see the amateur which principalism alone can can can solve and this is that the main weak point of principalism it's abroad usually it's good to say reflect on things but for actual decision-taking in actual situations might not be too helpful this is my experience so you might ask obviously what we did well this was a speculative case study it didn't really happen and I can neither confirm nor deny that this thing ever happened so I kind of skip that one I'm in I'm in in favor of disclosing the thing but if you do well think about countries affected and well get a good lawyer that's what I would in that case suggest but again this was wholly speculative next one domain controller case study customer shows up ask us like okay we have we have a team which is specialized in AD and Windows security and they got a request from customer can you help us with analyzing the logs of a domain controller for specific say behavior and we are like sure we could do this there's this well we have the expertise to do so could you please describe it in a bit more detail what what you need from our side well you know we have that guy Frank and Frank he's the leader of the local activist movement and Frank we think that Frank is leaking information well okay and so what and you need to you need to find out that Frank is leading leaking information and we were like wait a second this is not only a technical question can you tell us a bit more yeah hey you guys must help us getting Frank sued I mean it was not that drastic but thinking about it the more questions that we asked them what was like it became dubious what you want what is your what is your objective of the activity you want us to bring us in for this was brought to the ethics committee and again as a line of the structure the approach was kind of followed get effects well that's not easy in that case as well it's okay that's that's lots lots to look at the customer couldn't really specify which type of activity to look for but this could have been solved in a technical way or another but it there was some some elements remained unclear which didn't really help like well informed or ethical decision-taking the next one is some well look at a well use affected that's out of the first one is always autonomy you could ask a well what about Frank's autonomy when we perform this activity but the thing is Frank is has a has a contract with the organization which by the nature of the contract and this is fully legitimate there's laws and there's laws of contracts and there's laws of laws of walk restrict his autonomy and in some way and so there is a there's a frame which restricts that and you can easily like surpass this based on ethical reasoning well you could there is there might be situations where law and ethics conflict but when those situations occur be very clear about your own agenda and you should accept that laws are there for a reason and in most countries laws are based on common reasoning in some sense I'm not against like civil obedience or anything of that it's just life some at times more complex than your personal perspective or Frank's personal perspective in that case beneficials that again is an interesting one as say there's types of actions you could do the job or we can refrain from doing the job who's affected by this I say if the activity is performed oh I only have three minutes left then I will speed up a bit that is that is unfortunate can I get an extra five then I can manage thank you thank thank you oh I was I apologize then given I am giving this the first time apparently I spent too much time with the with the first part the thing is there is like human person was this organization which again is when it comes to who benefits was us who gets harmed it's a very common conflict like things that are good for an individual for one human might not be good for the broader scope or vice versa and in general humans tend to favor humans which is the whole theme some of you know might notice Casa de Papel there's a group of humans which you have some favor for and they in some way in a specific a kind of fight against or exploit a broader system and this serves as a perfect example of make clear who's affected like individual humans as opposed to an organization and in general humans tend to favor humans which is human but one has to be aware of this it might not lead to the right outcomes in or to the to the best possible from an ethical line of reasoning outcomes in specific situations not in this one I can I can already tell you what we did the ethics committee declined suggested declining the job and the ethics committee it's called recommendations but those are expected to be followed by everyone in the company including management so we didn't do the job but very quickly on this say things that might be good for smaller group might not be good for larger group there is a whole thing on on on internet scanning and I can tell you on this one it's a kind of the same dilemma there was people who are you like well we scan the internet forum IOT devices and vulnerabilities and that's good for everybody so there is a group of people technically skilled from a specific user often very often make okay the invite what and and they decide on what's good for everybody out there you can probably already get the inherent problems I'm two people show me five minutes I got that I see you I'm losing I'm losing time when I react on this I won't father yeah I know I will quickly go through this thank you this internet scanning usually violates the the principle of autonomy have those people whose devices you scan and you might turn out light bulbs or whatever with that giving you informed consent from the manual report probably have not you can already spot I'm not a fan of this at all at E&W you have to you have to cross very high say boundaries to get a project approved which does internet scanning in some way and when I was in the in the technically very sound subcontract yesterday a black hat and they were like oh we did the scanning and we found devices which at the time of the scanning there are 30 a thousand feet high vx walks shells I was scratching my head away wait a second did you have informed consent of the people in the plane at the time that you may be crushed a 20-year-old wash no vx walks by scanning it from the ground probably not these guys were these slides will be published but just to give you a quick idea the scoping thing is important the scoping might be problematic very quickly I already mentioned this we declined the project very quickly two more case studies this one was interesting as well that actually it's the first one on the time scale as this led to the creation of the ethics committee we got a request can you perform a training on telco technologies and we have expertise on this we were like yes and then exit and while we is doing the setup phase it was like well can you do it in a way with a Russian how do you call it translation in real time Russian translation and we were like well that's maybe people will learn a bit less but well if you pay us for this we can do this and then it turned out all we only want to look a whole week at interception capabilities and and surveillance some interfaces and that was why okay why why do they want this again this brings up many interesting questions who are the effective stakeholders what is the scope what and what a value what is about autonomy and form consent again we have this country was this other countries or was this the own population thing and we don't even know it might be perfectly legitimate but it sounded that's also a legitimate the thing is we did a job as we had committed but this led to the to the creation of the ethics committee to relieve individuals the guy who did it he wasn't happy about this at all we told him maybe I've all committed so do it but in the future we will have a committee which decides on this I have another one I can't tackle this one it's an interesting one as well you can probably get the main points from the slides which really published conclusions there's a thing I would like to to make clear here this one conclusions understand that ethics affect a lot of things you do and understand there is like these formal approaches and go through those get effects get a values get a stakeholders identify what could be done one way or another this is not an easy task and with this I will conclude I already told you I'm I have a literature background there is a whole genre of medieval literature with the court of artists and so and they're usually the put that the hero which in that case is pretty much always male which is I'm going to use he the hero sets out to perform a quest at some point the hero has to decide right or left and the first stage usually pretty much always a hero chooses right as this is the easier way the hero arise at court he might even win the turn at the first tournament he might even get the the the King promises you can you can marry my daughter concept of autonomy is violated but didn't so much exist at the time but then chaos kicks in and he loses everything he starts again at the same point right or left takes a left path that's more tedious that's more takes more time more reflection more walk but that's the one that leads you to to the to the Grail and this is the message I want to give you when it comes to ethics it takes time takes effort but it's worth it thank you


I can't hear you. You're muted, Daleks. I'm always that guy. Sorry. Good morning. Hi, Alex. Hi, Kiran. I can hear you fine. We'll just wait a couple more minutes and have a join before we start the presentation. We'll just wait until five past to give a couple more people time to join. I'll be right back, Alex. Okay. Welcome, everybody. Good morning. Good afternoon. Good evening, depending where you are. So the main agenda item for today is we are going to have an open EBS presentation. Kiran and his team are here to present this for background. Open EBS is a distributed block storage, cloud native storage system. It's currently a sandbox project of the CNCF, and it was introduced as a sandbox project, I guess, just before Barcelona. That's right. Perfect. All right. So the team have raised a proposal to move this into incubation. And we are looking to prepare the review. Kiran, how would you prefer this to run? Do you want the questions to be interactive or should we hold still the end? I think let's keep it interactive. What I have done is just put together a few slides just to set the context. I may or may not be able to answer all the questions today. I at least like, you know, will try to follow up on them in the next calls with the right people. Let's do it interactive. That sounds good. Thank you very much. I'll share my screen. And I also have some of the team members. I've shared this link with them just in case my internet goes off or things like that. All right. So this is a quick summary of the project since we became Missy and see of last May after the around Barcelona, we have around 35 new companies contributing in some capacity.  This is a part of the annual review as well as the incubation PR. And we seem to be attracting at least like five contributors every month. We have gone to a monthly release cadence where we're on voting new reviewers, new contributors. So far, we have not added new maintenance compared to like what we had in our own sandbox, but we are working on that at this moment. So just a quick recap for those of you who are just hearing about OpenEBS. And then we'll get into like quick updates on what we have done since OpenEBS 0.9 was released when we introduced it as sandbox right now it is at OpenEBS 2.0. Provide the summary of the changes that we have done. So we are, it's a hyperconvert storage, and we call this category of storage engines as container attached storage, because the storage services itself are delivered as containers. These containers are orchestrated by Kubernetes managed via Kubernetes native resources and custom resources and native resources as well. One thing about OpenEBS, with all of its data engines, we try to run them in user space, make them portable ability to run on any kind of Kubernetes platform or whatever it is. That's been one of the unique design constraints that we have set to ourselves. And at the same time, most of the adopters that we will see have mentioned that OpenEBS is easy to use. And even just briefly introduces what container attached storage is, and we try to. Kieran, if I may remember. Are you going to cover that there was some concern around relationships with other existing projects, I think Longhorn was one of them in the past. Are you going to cover the projects fit in here? Yes, I kind of missed all the storage engines and also we have covered that information in the annual PR as well, how OpenEBS compares with Longhorn, how OpenEBS compares with RookSafe. We can continue that discussion. And sorry, sorry, not to not just to be clear with my question was not how it compares with competitors but how it incorporates some existing technology. Yep. We'll do that. All right. So I think most of the orchestration work is offloaded to Kubernetes. I think that's one of the primary differentiation between container attached storage versus implementations of storage services that are done. So one of the cast engines is about data services that we kind of take care of this kind of storage management on the Kubernetes nodes, making sure that data is highly available for the applications and enabling the data protection on them. Those are the services that are implemented within the cast. Some examples, yes, we'll talk a little bit more about Longhorn today. I think all this in my opinion kind of fit into this example and Rook is a great example that actually orchestrates, that could orchestrate all of these engines as well as like the way it does with other HFS kind of storages. For somebody to get started with OpenEBS, these are like some basic examples, basic commands. You just use the single element store command to install OpenEBS. It ships with some default storage classes as well. And then you launch the application. The picture here kind of shows how it works. So the application PVC is backed by a target or the PV comprises of a target and a set of replicas and target is responsible for distributing the data across the different replicas. So slightly a high level architecture diagram of where the, what are the different components are how we categorize them. So just we have cluster level components and not level components. So most of the operators, including the CSI driver control, controlling components as well as the OpenEBS operators itself that manage the engines across the different nodes, they are all called under cluster components. These expose all these cluster operators work via Kubernetes resources. So you can kind of integrate this with other third party or open source or commercial products. So some of the things that we have integrated with our like weather of Prometheus and Kubera, Prometheus and Cortex in fact. And in terms of node components, again, like we kind of divide them into three categories. One is the management of the storage and volumes, which is the CSI node components or agents and the node disk manager. That's one of the components of OpenEBS that helps manage the block devices attached in terms of discovering, allocating them to the storage engines cleaning them up. Those kind of operations are performed by NDM. And then you have data engines that which are basically in the IO path and these are always running as part of node components.  I'll just get into the question that Quintan asked just before that I think this is like slightly more detailed interaction between the various components. And we kind of have collection of data engines and we can kind of segregate them as replicated versus local PVs. In case of replicated PVs, these are like used by applications that need high availability feature from the underlying storage. So here we with all the three engines that we support, Jiva, which is actually taken from a fork of a longhorn, even before Longhorn was part of CNCF. And there have been like slight differences in the way the Longhorn and Jiva have progressed in terms of how they maintain the data availability scenarios. That's the primary difference. And while the core engine parts are the same, the high availability aspects are where they differ and they continue to be in fork projects. But all of CSTOR, MYSTOR and Jiva have the similar architecture where a PV is backed by an ISCSI target. MYSTOR is starting to support different types of access targets, but ISCSI is the most common one used as of today with our end users. This is a Kubernetes service associated with a deployment object and that takes in the IOS and writes to various replicas. The replicas are what differentiates between whether you're using a MYSTOR, CSTOR or a Jiva based storage engines. And that again depends on the kind of storage capabilities that are there on the node. You typically pick up one of these engines. So we'll get into the differences of these three engines. Any questions till here before we get into the specifics of each engine? A couple of questions actually. If we look at the previous slides, the different components of OpenEBS. So I believe when we did the sandbox, we talked about the OpenEBS core and the data engines. Just to clarify, when we're looking at the project at this stage in terms of the incubation submission, does that include things like the operator and the disk manager function as well? Or are those external components? It includes definitely the node disk manager components. These are all written by the OpenEBS authors as part of the OpenEBS project. During the sandbox, we, for example, CSTOR data engine was split between the changes that OpenEBS authors wrote versus what comes from the ZFS. So that's the modifications done to ZFS that kept outside. But the lib ZFS, that's the one that adds the replication layer and all the part of the CNCF. So we can get in. If I understand the question correctly. Alex, so we should get into the details of like what are all the reposts that we get to as part of this incubation. Yeah, so, and I only mentioned this because we had this discussion with another project where we skipped out some of the dependent repos. And I just want to make sure that we're covering all the dependencies for OpenEBS as part of the review. Sounds good. Alex, I just was observing that on the TIKV project as well. Yeah, exactly. We'll do that. There are around 70 repos right now, but most of them are for dependent repos. Just for build compatibility, we maintain that the core repositories are spread across 14 to 15 repos and share that list. Okay, thank you. On this, on this function. So, and mostly just out of curiosity, so does, is the iSCSI target, is that a component of the engine, or is that a separate component in this sort of model? Yeah, iSCSI is part of the engine. iSCSI target that particular container actually gets spawned when a volume is created. It's not always running. It's per volume, there is a iSCSI target part that gets created. Right. Okay. In fact, all the green components are engine components. I have not mentioned the operators of the CSI driver components that are there. It just goes to say that whenever you create a PVC for a given specific storage class, the operators work out on launching these data engine components for that particular period. I've been getting into too much detail but that iSCSI target there is that located, is that co-located with the volume replicas or is that co-located with the attached container? That's actually controlled by the storage class policy that we can configure. There's no hard tying off that one with either the replicas or the application parts, but for performance or, you know, those kind of reasons users can decide to say that it has to be on the replication parts or it can be moving along with the application. Okay. Could you speak to where thin provisioning comes in to around that iSCSI target? Is that all done by the back end? Right. So in the way that thin provisioning works is based on the replica, it is handled by the replicas here. Yeah, I think that's a great question. So it also, for example, iSCSI does a synchronous replication. So if we have asked for like a 5GB volume, then we need to have 5GB available space on all of the replica parts. Now, it is possible for this replica parts to serve multiple iSCSI targets and store the data on the physical hard drives. So you could have potentially like a 50GB overall capacity at each replica, but you can provision 100 volumes. We monitor the provisioning, like the capacity usage at the replica and you can expand with additional disks on the replicas itself. And the new data operators that I talk about later on can actually help with, let's say if one of the replica parts was scheduled to take in data from multiple replicas and if that is going out of space, we can shift that replica to another node where there is capacity available for that. One distinction also is this is not a scale out storage. So for example, you cannot consume the capacity on different nodes and provide a capacity that's aggregate of all this capacity to an iSCSI PV. The capacity to provide the iSCSI is kind of what you get. And it has to be available on each of the replica nodes. So I think just just to clarify that point, right, whilst a volume, the capacity for a volume can't exceed the capacity of an individual node, the volumes effectively get distributed across all of the nodes that have available storage, right? That's right. Yeah. Also, did I was able to answer the question? Thank you. So now we get into a little bit of specifics and how these things are different. Just to kind of reiterate Quintan's point, the Jiva report that we have, that's a fork of Longhorn engine. That's one of the components of the Longhorn project. And the Longhorn engine works in combination with the Longhorn controller, Longhorn UI and the availability is really maintained by some additional operators which add and remove replicas to the controller. But in case of Jiva, there is a layer that's added on top of the Longhorn engine, which automatically reconciles itself or like, you know, reconnects to the controller on node restarts and node failures. And it does not depend on a control plane for the volumes to continue to work. That's where the major difference comes. And also, this is one of the reasons why we have put a limitation of 50 GB kind of a capacity on Jiva and similar limitation does not exist with Longhorn. And with Jiva, we need to have at least two replicas in online mode for us to read and write data into that volume, whereas Longhorn works with even a single replica in a read write mode. Because the control plane or like the UI controls on the quorum logic or like, you know, it decides on who gets to be the master. The quorum logic is inbuilt into Jiva. That's why we wanted to reduce the time taken for rebuild and make sure like this is tuned to have the two replicas available most of the time. The nice because he usually puts the volumes into read only and there is a manual operation required to remove that. Also, like Longhorn went ahead and made some changes in terms of like backup support and all that. This is an open EBS project for backup and restore. It uses well arrow. So all those and all those changes are where Jiva differs from Longhorn. And the other difference, like, you know, the change with the code is in terms of space reclamation for maintaining the high availability. Jiva and Longhorn both actually create some kind of an internal snapshots. Jiva ends up automatically purging them beyond like some threshold that's configured and Longhorn added that capability later on which was to purge those snapshots via the UI. So, just from a roadmap point of view, I believe if I recall around the sandbox time, CSTOR was sort of the primary engine where most of the development was happening, correct? Is that still the case? Or are you moving to the new My Store engine then? So, when we started the project, we kind of thought that there may be like one engine that may be suitable for all workloads but then we soon realized that that's not going to be the case. Depending on the capabilities of the storage node and the application demands, there was a need for different types of engines. For example, like I'll just get, there is a local PV that we mentioned here. This I think was introduced around the same time as the sandbox time frame. A lot of operations around just carving out a block device from the local storage that's available, typically discovered by NDM, was itself sufficient without adding other CSTOR or Jiva and that local PV itself has taken on its own journey based on the feedback from the users. Now, we support four variants of local PV in OpenEPS. And based on this feedback, we are continuing to support all the engines as the OpenEPS community at this point, but restricting the use cases to which each of those are suitable. For example, like Jiva is mainly suitable for workloads. Actually, we have built-in ARM support for that as well. So, for lightweight, when there are no additional block devices, I should get into that a little bit here, or actually the last line here. So, when there are no external hard drives or extra devices available on the node, that's when, and you need replication capability, that's when like Jiva is preferable. But if you have hard drives or SSDs or, you know, you want to be able to expand that on the fly, then CSTOR is preferred and CSTOR also has inbuilt capabilities around like, you know, instantaneous snapshots and clones, which we don't plan to add for Jiva, for example. And MySTOR was mainly intended for the performance reasons. With CSTOR, we couldn't drive up the performance to a large extent because of the way we were dependent on the ZFS technology there. So, we started working on MySTOR that actually solves some of those bottlenecks. It's inspired from the work that we have done prior on CSTOR Jiva and MySTOR is the new engine for hardware where NVMe devices are available and you have a lot of CPU power and you can drive up the performance. All right. Thank you. So, yeah, so the quick answer to that also is we continue, we plan to support all the engines. I have like a roadmap where I kind of mentioned what we are planning to do on those aspects as well. Another interesting use case that we continue to see is like how many people ask for read write many support and we support that via the NFS and it can pretty much work with all the block storage options that we have today. And we'll see in the adopters, in fact like CNCF itself uses the local PV plus the NFS to run most of the DevStats portals. So, any questions before I move on to the history or like the current state of local PVs that we have? With the local PVs, could you maybe spend two minutes to sort of differentiate how this is different from say, I don't know, the local parts type things which are kind of native to Kubernetes for example? Right. So Kubernetes, the local PV type or like the persistent volume source where we specify that word local, that's the same thing that's used in almost all the local PVs. So that source which forces you to specify and or like stick it with the node affinity, that's the PV spec that we use. But how do you actually give the path to that? Do you give a static device like the Kubernetes static provisioner which goes through a list of mounted devices? And then you go and create the PVs for that and then the Kubernetes storage class and internal scheduler kind of takes care of scheduling the applications to that. That's the core functionality. But on top of that, like how do you provision the storage to this local PVs? That's where I think OpenEPS has helped. And I, you know, I also noticed that this is kind of increasing. I saw a KubeCon talk around TopoLVM which also is doing something similar. So these projects like OpenEPS local PVs as well as like TopoLVM or like Rancher's host path provisioner, these are built around that core concept. But how do you make it easier for users to use that in terms of managing that local storage is where the differentiation comes? There are three things that I've listed here. There's one more thing that the fourth one I will add it probably by next time we talk about this. The main intent of using or like, you know, I was going towards local PV was the capability that we had already with NDM that would discover the log devices and the partitions that are attached to the node. So you can dynamically claim a device instead of using the static provisioner. That's where we started off. But then we soon found out that there are nodes where devices are not readily available and people want to use like some kind of a host path or a directory by creating a new directory within that. That's where local PV host path came in. The provisioners for these two are same and they're based out on the external provisioner. We're moving them to CSI-based things. ZFS local PV was kind of an interesting ask from the community where they like the concept of local PVs, but they also wanted resiliency on top of the, you know, basically protect against the disk failures. So you create Z pool and then each local PV is actually backed by a Z wall or a ZFS dataset. This is this one is mostly comparable with the topo LVM project that was presented at coupon. That makes sense. And the one that we are adding now is called raw file, which is the local PV host path or like the device or Kubernetes static local PV. All of them have one limitation in terms of enforcing the quota on the device. So typically like the volumes can grow beyond the capacity that's actually given to them. Applications can write beyond that. ZFS local PV can restrict that because it has a boundary management via ZFS and LVM with LVM. The new one that we are adding is based on the host path where we put up a sparse file that kind of helps you to contain the capacity used by the application. Hey, Kiran, just a related question. I'm not familiar with NDM, does NDM discover local disk and advertise them as local PVs? Look, how does NDM work? What does it do? That's a great question, everyone. So NDM discovers the local devices. Again, like it's highly configurable via config map on what kind of devices you want to discover based on path filters and properties of the devices. It actually creates something called as block device Kubernetes resources. It does not create a PV. It exposes block device resources and we can, for example, when you want a local PV or a C store engine to be created on those devices, these C store operators or the local PV operators can request for a block device similar to PVPVC concept, a block device claim. And NDM operator will pick up the node or the block device whichever is available and bound it to the BDC and the higher level operators can take that block device and consume it. NDM itself is independent. It is not, you know, it can be actually used for other purposes as well. For other operators, it need not be tied or like, you know, it's not, it's independent of the OpenIBS components. Just to add to that, I think NDM is actually an interesting component on its own because it effectively allows you to create Kubernetes objects that represent the individual disks. And there's nodes, right? So you can now sort of compose those disks and use them for, you know, different purposes either in the case of the local PVs or for the disk engines, right? Yep, that's good. Thanks. Thanks for the question about local PVs. Do you do anything about, are these, and just have to excuse me, I'm not that familiar with some of these other technologies, but is there any data management? So is a local PV by definition empty when you create it and all the data deleted when it goes away? Or do you have some mechanism for restoring the data into a local PV if it's not already there? Yeah. So the way it is today, it is an empty device or a directory that you give to an application. And once the application deletes that PV, it comes back to NDM operator and NDM can take care of deleting that data. And it actually marks it as released and does the deletion in the background and makes it as unclaimed once the cleanup process is completed so it can be reused by some other application. In terms of data services, if we want to put some data into the local PV before it is given to the application, one approach could be to use the volume data source mechanism. We have not done that yet. That's definitely on something that we have been thinking about. And the other data service that we provide with local PV is that again, like was a community asked us to make sure that we take backups out of this. So this is a, you know, we again use that a well heuristic based backups for local PV as well. In case of ZFS local PV we can be a little bit more smart because we can take incremental snapshots and send that to the backups. These are the two primary services. That we offer data services and ZFS local PV also as potential be actually experimented with it. We can provide encryption support on top of ZFS local PV as well. Encryption at risk. Thank you. Okay. So this is what has been keeping us busy. This is like just a quick snapshot of the things that we have done at a very high level. We can dig into each of these for further details in the upcoming calls or there is something quick I can probably answer now. The Ndm was at version 0.3, I think when we went into the sandbox, a lot of enhancements around that all sorts of clusters where we were deploying Ndm and we kind of learned a lot in terms of how the block devices are actually attached to Kubernetes nodes and the different variations in cloud environments with respect to how the block devices are represented. So all those things were factored into Ndm. Now we are able to successfully detect almost all types of block devices. The most challenging was around the virtual devices and partitions. We could detect it, but because there was no unique identifier like serial number or WWW and things like that and node reboots things would change in terms of path and it could become a little difficult to know what was the previous name to that one. So kind of adding support to handle those kind of situations is the main thing that I want to call out as part of Ndm. Just like each of these engines Ndm is its own project and it has its own maintainer reviewer and a lot of interest in terms of adding features to that. Right now there are a couple of alpha features that have gotten into Ndm as well, which we want to work going forward or in terms of metric support at the block level. Things like smart metrics on the block devices, how do we capture and get them out and put others, for example, like temperature or it could be like smart characteristics like error rates and all those things. If the device supports, how do we get that? Those things are in progress and there's also, it is controlled via the Kubernetes CRs right now, but there have been some asks about ability to control via GRPC API. So we are adding GRPC API layer also to Ndm to list and discover and perform some operations on top of Ndm. Jiva was, I think Longhorn, since we took from Longhorn it was, and we had tested it almost for like two years before we called it stable and a lot of users had actually already started using it. A few things that were causing the problems for the users around Jiva where in terms of volumes going into read only especially when the capacity was growing and also if you kind of create a 5GB volume and the application is such that it keeps on reading and writing the same blocks. The snapshotting technology used to maintain high availability would end up creating a lot of internal snapshots that would end up consuming a lot of space. So as users started using it for like one year and more than one year they started complaining about 5GB volume on a replica is taking like three times the capacity and all that. So that's where like we had like some initial work done on CLI work to reclaim the space but we automated that so that's gone as part of 2.0. A CSI driver is definitely where we want to move for this engine as well and that's available now for this moment. One of the primary drivers for us to go towards CSI driver though we don't support all the other capabilities like snapshots and clones is the ability to remount the volumes when they become available. Right now the ISKC PVs kind of need some kind of a manual intervention from the user if they get into read only to remount them so that additional capability we're adding as part of the CSI drivers. So CSTOR was in early beta when we introduced the sandbox we continue to keep that in beta for a very long time. The initial users that started using started giving us a lot of feedback in terms of data operations that they want to perform on top of CSTOR. The version one or like you know we were not for one spec that we went with was becoming very difficult to kind of support all those data operations so we went back and changed the schema a little bit based on how users wanted to actually use that. In fact like we didn't have some elements in the AML we got the AMLs and found that users were like putting some comments in the AML for creating the CSTOR pools. So we took those comments and converted into a new spec so the new CSTOR schema is up for beta now in and this time we went directly with CSI driver for this. So most of the things that we started off we came up with the new engines that we supported we are going with CSI drivers for the older engines we have started the work on supporting the CSI drivers as well. For local PV, host part and device there were a few configuration asks for example like especially like the device based one if there are NVMe SSDs and local SSDs available on in node how do we tag these block devices as NVMe or SSD and use that information to put let's say MongoDB on NVMe SSDs or some other application on SAS SSDs or that kind of fixes so those went in we are in the process of like kind of adding capacity based scheduling we did that for ZFS local PV but for host part and device as well. But then we also saw that there is Kubernetes enhancement around that so we are trying to see how to integrate with that work instead of trying to write all of that into this provisioners. My story is the new one, which we basically started off from scratch again based on our experiences with the old one. And this is a new engine that's written in Rust at the moment and it also supports CSI driver. So in terms of comparison, we just released 0.3 and it's in the early alpha where replication capability is built in with node level HJA supported we plan to add snapshot and clone capabilities also into my store. We already spoke about file local PV, the other important thing that we did was around the upgrades. There was post sandbox we support now seamless upgrades at actually they can be automated via Kubernetes jobs. That's been a really like this feature compared to like many of the others I think this is like on the top of using the pain of the users. A lot of community has come together to help us with building open ABS for arm and power PC. We still maintain it as alpha because we are not enabled to do pipelines. So other work that we have done as part of 2.0 is open source the e2e pipelines on, for example, like GKE platform, AWS platform, bare metal VMware run the open ABS every build and every release on different platforms. And one of the spin off of open ABS was litmus that helps us to stabilize or test the resiliency by introducing chaos. And in fact, like open ABS was the first chaos, the chaos thing came from, how do we test this storage. Since we are now distributed and it's running in containers. How can we kill all of this and make sure data consistency is maintained. That's how we have started that project I think it's taking its own wings right now. But a lot of improvements in terms of e2e also have gone in. Any specific asks on this slide or questions on this slide before I think I just have a couple of more. Hey, Karen, could you just talk a little bit about the sort of the ZFS dependencies are those an external repo or are they part of the IP that's that's being added to the CNCF and and if so how are we dealing with the sort of the licensing type issues. So for ZFS local PV it's definitely outside. So just like how you install for anything you can set up the ZFS outside of it. Now, when it comes to C store, I've actually had some conversations with Chris on this in terms of licensing and all that. Core ZFS itself is separated out. It's in a separate repository. So it can be pulled in as a library. There's some discussions on like, you know, maybe like that can be maintained as part of separate organization and not part of CNCF. I think we'll work that out. And that's the only I think like we have run the process scan on this and that's the only thing that has a CDDL licensing. The rest of the things are something that are compatible or like are actually apache itself. So just to clarify, it seems like some of these external code bases are forked and some of them we just incorporate directly. Is that true? Yes. And I think it's going to be very important to make very clear which are forks and which are just external dependencies and some of which are optional, I assume. And so if you don't use the ZFS stuff, then you don't incorporate that repo and then you don't inherit any licensing complications. But if you do, you need to be aware that you're pulling in the third party component. Exactly. Is that made? I think it's very important to make very clear which of them are third party external dependencies versus forks of such things. Yeah, maybe we can state the PR accordingly with that. I think that's a great idea, Quinn, so we can understand what we need to consider for trademarks and what we also need to evaluate if it is a true dependency. Probably also need to be looked at as far as. Yeah, definitely follow up with this list of repositories and that kind of dependency that we have on them. Yeah, it's a great way to kind of continue on that on the incubation PR as well. Hey, just a quick one though. So, is C store, I mean, components that are dependent on ZFS like C store and the local PV ZFS, for example, are they usable without ZFS components though? So C store kind of pulls in the dependencies, C store pod when you run, it has to have the, it actually has the code that is user space ZFS, which is like a modified question of ZFS. That's probably where we need to have further discussions and see how to deal with that one. In case of local PV ZFS, local PV ZFS started with supporting the CSI driver and additional options in terms of capacity scheduling for local PVs. So, ZFS is one of the storage options that you can configure. You could easily change that to like, you know, I don't want to create a Z wall, but I can create a directory there just like a host pod directory or I want to actually use a NDM block device to use that. So, the code that is there in the local PV ZFS is reusable for other local PV options that you can support. Okay, but for C store, it's a core dependency. C store is a core dependency, yes. It's a, yep. Okay. I'm not an expert, but I think we might need to, we might need to look at, ask the CNCF for some legal advice here. And given that, you know, the, the IP is not actually the IP that the project is dependent on isn't actually part of the project. I will go ahead with that conversations and probably like to the needful there, Alex. I think should be part of due diligence as well. All right. Thank you. This one just gives snapshot on areas of improvement or like, you know, things that we are already planning to do as part of the upcoming 2.x releases. We have a monthly cadence. So some of these things will come in this month, next month, you know, kind of course like that. The features are again like picked up based on the monthly product review meetings that we have with all the maintenance as well. It's open for end users to chip in and based on the GitHub issues that are raised by the end users. We picked this up. This is just a plan and one other thing or like, you know, a few other things that we want to definitely improve on is the end user documentation and website. In general, we have done very little in terms of open EPS advocacy in terms of, you know, whenever we now talk at QtCon or somewhere, people come and tell us that they've already used open EPS or they've integrated like gravitational open EPS that we get to know when a community user comes and ask. So we need to do some active work there and contributors onboarding. I think, you know, community bridge GSOC partitions have been helpful. We've onboarded contributors and reviewers as part of their programs, but we definitely can do a lot more in that regard. So those are like some of the things that we want to continue to focus on as we go forward. Just to kind of briefly list the different adopters, we, again, like started off like this exercise like, you know, once we started thinking about incubation, we opened up this adopters.md file and created an issue where people can comment. So there are around 25 users that have commented how they use open EPS. They're all listed on the GitHub, just to highlight some of them. This is one of the oldest and actually like one of the users that influenced us to continue to improve on the Jiva space reclamation problem Arista. This is their story. I think they were running on premise and they really liked open EPS in because of its ease of use and simplicity and it enabled them to move towards Kubernetes faster. That's there. It always feels nice to hear those kind of words from end users. So this is another thing that we recently got to know because of some slack channel. We are now in as part of the Kubernetes slack open EPS is the channel where we hang out. And we got to know this user was already using even pre beta stage 0.7 and has been upgrading and is currently at 1.10 version. This is a news case where local PV options kind of get highlighted. I think this is definitely a pattern that we see where workloads are going towards more distributed in nature and local PV is just on like tries and demand. It's just increasing. I did have a placeholder but if there are further questions, we can take it up. So the licensing when we when we went over this in Sandbox, there was some concern over it being compatible with the required meant for CNCF is that I mean, I still think we need to possibly work through that. You know, we've already talked about the dependencies as well, but Sure. At least with the presentation I haven't done any diligence. I apologize. It still seems like that's a problem that we need to rectify before it could be considered for Thank you. Sure. We'll take that up as one of the top items. To kind of follow up. Yeah, just to be clear, I think this was very clearly highlighted years ago. When the project entered Sandbox so I personally I'm a little surprised that that it hasn't been rectified yet and it sounds like it hasn't been done in two years not going to be done anytime soon.  I can't see this going into incubation. Anytime soon either because it's it's it's going to be hard requirement. So just to, I think my apprehension in answering that state was and we need like more is mainly from the legal expertise, but we did something. Clinton after the feedback where based on the suggestion from the CNCF legal team is split up the code base to see store and lip see store and lip see store is what was part of the CNCF donation and not the see store component. But I do want it to go through a little bit more of a rigorous license scanning thing and I've been having that conversations with Chris where we can come up with some solution is what he told. So we will work that out. And we're probably out of time now but it might be worth. I mean there's always a lot of detail in these due diligence is and especially around licensing but the high level requirements are actually fairly simple and pretty straightforward to understand. And maybe I can just summarize them in 30 seconds now the basic requirement is that the project should be usable without depending on anything which has a license which is incompatible with the CNCF open source license. And then any optional components that do not fall into that group ie potential optional dependencies that have incompatible licenses should be very clearly called out so that people can you know explicitly adopt them. Once they decide that they're comfortable with the licensing restrictions that they that they're adopting and it doesn't sound like you know and then there's a ton of detail underneath that to to make sure that all that that actually checks out but it sounds like we're we're if I understand correctly we're fairly fairly far away from even that basic requirement. Correct me if I'm wrong but that's what it sounds like. Yeah, I think notice.md has been added with that one. But I'm with you with you all actually in this regard, just as a CNCF member and contributor that we need to get this done and we have to do it the right way. So I'm not biased towards answering that question. We will follow up on this question Quinton I think that's the we've definitely done but I want some legal answer to be clear. These are like don't want to have any great answer around that. That's the only reason I'm visiting to continue on that. Hey, hey, hey Karen, I'll start. I'll start a thread with Chris and check and the co chairs and yourself and so I can actually loop you in. We can understand this. Alex, I can probably add you to the thread that I've already opened with them. We can take it forward from there. Yeah, that's that's that's fine too.  It's, it's kind of hard to move a project or a repo into into into incubation if if any of the dependencies aren't aren't compatible. So so we just need to probably sort that out as a step one at this stage. Got it. All right. Thank you everyone for and thanks Karen for the, for the presentation. We're looking forward to to working together on this. And we're just a little bit over time now so unless anybody has anything urgent, I think we'll call this meeting to an end. Thanks. Thank you. Thank you. Thank you. Presentation by the way. Thank you. Thank you.


So I've been meditating on and off for the last Five years probably. I'm 22 right now. So I started about 18 years old and Really really delve deep into meditation Like super super seriously only in the last few months I was going through the hardest period in my entire life Going through some major major emotional troubles and financial troubles and I Was going up and down and up and down all the time Now everybody goes up and down.         It's like a bug. It's like something if my mind is like a series of buttons There's the depression button. Somebody just and just Held it And now it's like it's like happiness. Yeah, but but it's held the the the depression button is still held Excitement. Yeah, but it's still held. So you're kind of stuck in a loop And this happens to you with positive emotions, by the way Having positive emotions is not very good either. I'm not there's emotions that don't have Negatives like joy like relaxation like honest relaxation Love True love not not oh my god like true love These are things that have no Opposites So they're good. They're natural, but there's things like being excited being You know super enthusiastic and stuff like that They basically guarantee That you'll have an equivalent crash sooner or later And if you think you're smart, you're like, yeah, no, I can I can be enthusiastic or I can be high energy happy And hold it for a while. Well, then you're just looking for Just building yourself up towards a bigger fall that will last longer Later because the guy who was really happy today And didn't hold it then tomorrow. He's sad he's like Oh, I'm sad now that sucks, but the guy who was like happy for a month And forced it. He's like, yeah, yeah Next month Bam his entire world crashes and He's like, but but I was happy. No Is it all fake?   I'm sure there's some stuff that I left out, but You know, I make five videos a day. So I think I think we'll be good and I'll just mention it If it comes up Thanks for watching