Welcome to the talk about Polygen, the title of the talk is Ending Flameworks with Polygen and we will see how we can use a software tool to end Flameworks. I don't see Amaya, she's wanted to come. She's just slipping off it. Okay, pull it in. Can we wait? No, no, because I need her for something. In the meantime, I'll show you something about Polygen. Things you can do with Polygen is, well, generate text. You wait to get installed Polygen and then you have it. You do Man Polygen data and you see some of the text that you can generate. For example, you can have Genius which generates Q&A. So if you want to add some Q&A to your pages for your favorite software tool, that's automatic. And you can have it do it for you. Very useful. Or in case you use Photoshop, you can get hints for it. Anything like if you want to make a video game, it will get some names for you. It's got like, if you want to make a playlist of hardcore metal songs, again. It can do great stuff. But that's out of the box, that's upstream, right? Okay, what I did, well, once is this. I went to a science and technology study conference and I've been reading through the titles of the proceedings. And so, well, they've been into my brain for a while and I put them down. All right, let's have a look at how Polygen works. Polygen works just like Bison. Okay, that's the side vertical here as well. You know Bison? It generates a grammar parser. So you can tell it, oops. You can tell it our grammar is made, so we have an expression, this for algebraic stuff. We have an expression which is either a number or an expression plus another expression and so on. And then Bison will generate a parser for it. So who's not familiar with Bison? That's how it works. You have, let's see if I can do it. Oh, so nice. With the arrow. You have a non-terminal symbol which expands to a set of possible instances of the grammar. And then, well, you have other non-terminal symbols, so a line is either an empty line or an expression and an empty line and so on. And then in Bison you attach, well, code to it. Polygen works exactly the same. You define a similar grammar, very, very similar. And what it will do is generate an instance of the language. Just as easy. If you want, you can use it to test your software. That's another use of Polygen, but it's too serious. So we won't do Polygen for testing. We want to end flameless with Polygen. So let's have a look on how to build a grammar for Polygen. Polygen has a really nice manual in user shared doc. So this is a very simple sample grammar. It generates answers from FTP master people. You can define, well, you have to define in Polygen two non-terminal. One is the metadata non-terminal and the other is the main non-terminal for the grammar. So that's metadata, title, author, language, whatever. It's useful when you submit it upstream. They will put it in the Polygen website. And that's the main non-terminal. It generates either nothing or no. It's nothing. So then we do Polygen... How do you generate underscores? Within quotes. Okay, very nice. Nothing should be more probable. So we have to control the probability of things. So we do the second... Well, we evolve a bit. Polygen is a really advanced tool actually. So we see M01. And we can actually control the probability of things. And we can say that most of the time they don't answer. On top of one part of the grammar you increase the probability. Actually what it does is it expands it in something kind of like this. That's two plus. I will kind of repeat that. It changes the probability like in that way. So you can look at Polygen and M01. Nothing. Doesn't sound too bad. Okay. So that's the idea. Well, general idea of Polygen. Let's see some more features. This generates taglines for my presentations. Let's have a look. So let's see some new features. So we have... If one word, unquoted word has a capital letter, then it's a non-terminal. If it has a small letter... If it's not capital, it's a normal word. Words are joined with one space in the middle. Okay. So you have the main grammar is this one. Actually, another feature is that you can add non-terminals to the metadata. So when you ask Polygens about the metadata, it's going to... By the way, Polygen... Man, Polygen data changes forever at least. Because most grammars reuse the non-terminals in the metadata. Okay. So we can have the main thing that calls other non-terminals. We can define as much as we want. And then every non-terminal has, well, different implementations. And then you can control the probability of things. It's pretty... All right. Simple enough. So how to get the metadata? Polygen, one only dash. And that's metadata. Okay. This is more like it. All right. Oh, I got my email address wrong. Whatever. Let's see some more things. And let's start into the topic of solving flames. Okay. This one. There was a problem with Damian women, with the people that approached you on IRC and tried to pick you up. So I generate a grammar that would solve the problem totally by making some subtitle. Actually, it helps running it through FMT. And since it's IRC, it helps to run it through BIF. All right. Once you put that in the wild, no one is going to harass you like that anymore. They will have to have a bit more fun. Okay. Let's see what came in. Okay. So we have a salutation, introduction, compliment, self-introduction, some compliment, and a proposal. The curly brackets that you see here. I saw much like that. And I discovered about it yesterday. How are you doing that? It's gromit. Gromit. A package called gromit. It's such a great package. And all right. These arrows means that all the items between the curly brackets can be swapped. So sometimes I have salutation, compliment, introduction, proposal, and sometimes I have salutation, introduction, compliment, and proposal. And then I have salutation, which is like A or high. And then person or... Curie. Could be also baby. Well, which again expands with some of the usual targets, sort of things, and some introduction and so on. Now, here's a new feature. This one looks a bit strange. If I don't put this one, what I have is that I have 50% of possibility of having cutie and 50% of having one of the names. But what I want instead is to have an uneven probability of all of them. So this greater than signs means expand person in line. So it basically gets replaced by all this. That way I can have... Yeah, I can group things and have a bit more control on things. All right, then we have other... Okay, minus, minus, minus is the opposite of that plus. And well, this will be like a real compliment, which doesn't usually happen on IFC, so I reduce the probability of it. And this will be like a real proposal, which means that people are serious and doesn't really happen on IFC. So again, we lower the probability of it. And that was one. So let's see. We have another pickup line. A bit more... There's a bit more advanced. It's kind of the same version as before, but it does a bit more. Well, there was another... Okay, here. It tries to make a nice, more articulated compliment. It can say, you are something. We can see it on...  Well, that doesn't happen that often to show, but... All right, well, whatever.  You are so pretty. You are the coolest. And, well, language is a bit crappy. You have, like, all sorts of concordance.  You can have to put N in front of something else. So poison allows you to define a non-terminal here with some kind of modifiers in the front. So we can define that these ones are names. Have... Are objective. These are superlatives. And then I can use them.  Or I can just use quality somewhere else. So I can actually choose to use them all, or to use some of them, choosing which one. And this is useful to start matching articles with names. All right, the rest is pretty much the same. We have another one, which is this hat sign. There. Good. This sign over here, it means to join without spaces. Poison by default concatenates with spaces. In this case we want to add a comma after the phrase, we don't want a space before the comma. And this sign does exactly that. It avoids having a space in front of it. Okay, so that was another future.  Okay. Right, and we've seen a bit more. So that was to exercise the carpet pickup lines. Now we have the problem with flamers. So a major one is the hot-bape one. And we can generate pieces of the hot-bape flame, and so on. And here we can see a bit more of the polygent features. For example, grammars can be recursive. So I can say that the main non-terminal calls itself. And I will have a repetition of things. So you can either say nothing, but that's unlikely. Or you have a phrase, sorry, which is followed by nothing, or a conjunction of some kind. And again, another instance of the language. And then, okay, you are, and here we see again the modifiers.  And you can actually get them all. You have optional words. So you can say, where is freedom of speech?  So the squares gives you a half, a 50% percentage of having it. If you want to control it more finely, and you don't just like 50%, you will have to do it with expanded form. And then you can say that people swear more often than not. But then, for the purpose of demonstrating things, I leave it like this. All right, and then, okay. And also, and then, but they are optional, so I can also have just end. This is already like something basic we have. All right? So with this one, with the feet, the hot babe flames. And then, what's the biggest problems in them in the world? Of course... Well, I swear, I went through five or six of his emails and just picked up braces and put them in. And so on, okay? So this is a bit more complicated, which shows a bit more features of it. Slash, this slash here, means a capital letter. I mean, like a piece of text has to begin with a capital letter, but then you want to recycle words. So you can just say, put a capital letter at the beginning. Actually, it's maybe clear if I just separate it. Obviously, it's confused with C. So I said, well, this course, it's all, you see, not capitalized. But then, I make it capital with this. Discourse, dot. And then a conclusion may be repeated. And then the discourse, it's, well, okay, usual things. And conclusion, well, objective and phrases. I pick up from the email. It's been really quick to put on this grammar. And you can see ways of expanding it to a more consistent way. So this is a bit of how policy works. We have another one here. The one for the algorithms. You see, I had to put pluses in front of the number because x book can expand in things that call it twice.  So I had to suggest that it should mainly choose numbers. And I remove a space before the minus sign in case of the negation operator. And I remove space between parentheses and the expression. All right. And number is a number followed by other number, again, removing the space. Option number, removing the space. Now you have an idea of what polygyne is. And if there is some flame that's picked up and gets really annoyed, just have to make a new grammar, post it to the list and ask people to do something different than what's generated by the grammar. Or submit it to me. I can put it together with the polygyne package, polygyne data package. Polygyne Manta is generated by a polygyne grammar as well. But then the only thing, I made it quick, so the only thing that's really generated is this. So it should be improved. That's a call for co-maintainership. Then other ways of invoking polygyne. Okay. Another one that I created just... for your entertainment. More ways of invoking polygyne. You've already seen that you can invoke polygyne like a moment. Okay. You can invoke it like this. You can pipe it through FMT to make it nicer. Well, you can pipe it through Biff, but then Andrew Safely doesn't deserve Biff, right? So it's more like of a chef. But then, well, he's serious. I mean, in what he does, doesn't deserve. And you can do, well, other things like, well, let's do the flamework. Odd, babe. Well, you know that, Kausen has many skins. So you can actually add, use different like spaces for it. I'm not going to use one at random because some of them are a bit gross. And so, well. And well, that's the idea. Or if you are lazy, you can just have polygyne do it for you. Of the polygyne, of the author of polygyne. Alvisa Spano is upstream. And there's a grammar with his biography. The polygyne website is completely auto-generated by polygyne. It's in Italian. Did I connect the cable? Did I do something? This has all the grammars. Well, it doesn't read that well. This is like from, well, there's an English version. But they used to have an English website. But then they thought they only had Italian users. And then they packaged it for Fabian. And they're hopeless. They have to bring the website again. They had all sort of cool stuff to use for cool components for the website. And so on. It's really funny. And so that's nice introduction to polygyne. How much time do we have left? Twenty. Twenty? So what's your current issue that needs to be solved in Fabian? It's the website. I mean social issue. Last night's drinking. Last night's drinking? Too loud. Too loud singing. Okay. At least on the mailing list. If you don't know it, you were in it. I was sleeping seven kilometres away. And you didn't even hear it. All right. It's amazing you couldn't hear it. So, well, I had put down some ideas actually for the polygyne both. Okay. We have our CP pipeline done. Update thread arguments done. ASA field answer done. Oops. Come back to me. Okay. This is other ideas I had. You know, writing change logs is a pain. Can we get a change log? Change log? I want randoms like a bug. Let's pick some sample data. And so we have... All right. It will be a problem to make the version monotonic. Well, I hope you will bear that with me. So package. You bear. Just think for future expansion. Semicolon. Finish that. And then we have package. Fantasys. Version. This is to avoid the space. You see there's no space here. Unstable. We always release to unstable. No. All probability is stable. All right. So... They still... Stable. Unstable. Testing. Experimental. Maybe I should... Proposal base. Yeah. Proposal. Stable is more probable. Yeah. Right. Can you think of another one? Stable. Proposed updates. Oh, do you have stable... Security update. Security update. Security dash updates. Yeah. Proposed updates. Proposed updates. Proposed updates. I was thinking more of a... Stable. Proposed updates. We still have money left? Yeah. Okay. That can not legal. Not legal. Not legal. All right. Taste trial. And then again, no space. Urgency equals... Urgency. Medium. Low. Critical. Critical. Critical. It might sometimes be completely unappercase. What? Sometimes they're all unappercase. All caps. People write them in all caps? Yeah. Archverge. High. Right? Can we capitalize it? No, but we can actually solve it very easily. Vim helps. I was wondering how come it's colorized, because you don't remember Vim having colors for a poison gun, no? But then it's for that, because of the changelog. All right. That's our fault. Okay. Beautiful. And then we have a couple of new lines. That's a bit sexy session, if I can now do the logo. Or do that. Okay. Good. And then we have many lines, items. And then we have the sick. Two black legs. What? Two black legs. Right. Okay. Two empty lines. Space dash dash. Maintainer. Would there be a problem with the capital for non-terminal high, medium, low, critical? No, that's quite okay. I think everything is capitalized. These are non-terminals. I have to expand still. High, medium, low, critical. Oh, right. You're definitely right. In this case, now that I think about it, it's polygen, right? So why don't do... We can. We do it, right? Actually, it's always good life because the test in strip is a women, isn't it? It's a great name. Right. Well, okay. Then we have the maintainer. Oh my God. Then we have one extra space and the date. Okay. Well, I would put that in the maintainer. So we have to fix items. You can see high, medium, low, critical. The other line also needs to be coded. No, that's fine. There are small caps. Oh. So it's a line and then nothing or items again. We have many items. Sorry? Look, plus sign inside one of the items. Okay, to make it longer, we can add a plus sign here. And... Okay. Then we have the maintainer. Right? Who can unmute this? Are there possible NMUers? Oh my God. It will do what then? If it chooses to say this is an NMU, then it would also pick from one pool or sub pool of the maintainer and it would also make the tag and NMU one half. Keep it simple. No, I think you can. You can do it with modifiers. You can choose to generate like a non-terminal NMU or a non-terminal non-NMU. Okay, my question was when it does it random, instead of randomly picking the maintainer, it could randomly pick if it was an NMU or not. And if it was an NMU, it should pick randomly if it was an NMU or not. Yeah, okay. And then a new one or a normal one and then propagates down. So I say that well, normal and then an NMU. Put something. You took all the things. Okay. So what else changes with an NMU? It's only version terminals. Yeah, yeah. With version and date at least. Right. So version. It's always zero dot. Yeah. Zero. Without space. Zero, none. Another number. Okay, let's do it like this or NMU, or sorry, or possibly NMU. Let's concatenate it. Dash. Just a number. It should work. How powerful.  It's really like I'm amazed by my upstream. And then we need to make the line. It always closes the bug. Or insults people, maybe? No. Closes the bug. Introduce the new one. Right. Introduce it. Introduce it. Support that. I need to recycle the digit. Yeah, but it wouldn't. Oh, there are usually six. Yeah. Closes, number, things, and so on. That's how they do it. There's also dot space. And then... Actually, I think the bug would be a separate kernel, so you can repeat it. Right. Yeah. There's a couple of spaces. In the web. Oh, no, they're okay. I think. You haven't specified them yet. All right. You're very right. Cool. Actually, I only need one. The other is added by Polygon. Oh, I can do both. And the Polygon, not to bother. All right. Then we have an object. And then... Context. Well, let's make it simple. The fix. Use. Work. Work. Work. Okay. Marco is never very vulgar, except making it so. The object. Support. Temp.FS. Detection. Report. Whatever. Context. Context. USB printers. And that's easy. You can add mount to the verge. And U-mount. U-dab what? You could add mount or U-mount to the verge, because it's U-dab buckets. All right. Then one can add other things like... We have the line. Let's ask Polygon what do we miss. Line 15. There's an extra dash here. All right. This is quoting. You're missing the answer. 44. All right. The copy. You're missing the answer to the beginning of the lines. Right. As there is there. And then we need the date as well. Okay. Let's comment. The comment syntax is strange, isn't it? Because Polygon is written in Ocanel. Which is the only language you could use to write this stuff. Because if you try to do that in C, you spend like eight years implementing the same thing. It has the complexity of GCC. Now Ocanel is a really high level language. Really good for writing all sorts of... It's kind of like prologos. No. It's more like Haskell. The sort of functional, but... And then, okay, date. Always work on a Sunday, always Saturday. And then... And then, all right. We do... What? Yes. If it's possible to run shell commands inside the grammar, but it's not implemented yet. Because I wanted to involve Polygon. To recycle other grammars. But then upstream answer me that it plans to do it in a more proper way by actually compiling the stuff or inside the other in a smart way. It's okay. I respect your wish. Do you want to use Polygon to generate the grammar? Okay, let's cut it short. Was it? Digit, digit. I mean, it's a... And then it's plus one. Because we are in... Actually, if it's suffiled, it should be plus zero. So we should actually do it with a context as well and pick it up. Next version. Yeah, next version. Hey, this doesn't need spaces in the beginning. Worked. There's one line too much. There's one line too much? There's lines too high. There was one place on Sunday. That was the line too much. Okay, because it's up and then right. And so it's up, up, up, up, up, up, up, up. That doesn't say. There it is. Hey, you learned how to do this. To a stable. What was it? I'm stable. Back to the closest. It's all just for missing. Goes back down, though. All right. I didn't add it in the end of line. Actually, we need to have a... Capitalize the first letter. And the version number. Oh, right. What's the cards? You didn't encapsulate them all. Digital K. Version. Concert, concert, and num. Okay, here. Yeah. You have really to hurry because we have to leave that building on time and everything you make longer, you cut from the last one. All right. Sorry. So I think we're done. We've seen how to post one.


CARE is a humanitarian and development organization. We were established 70 years ago where there is a crisis where CARE comes in. After the Marshall Plan was successfully delivered in Europe, it was decided to help other peoples and other regions of the world with humanitarian or development issues. We support very much their mission and their mission is to defeat poverty and to improve social justice.  So at this stage I think there are more than 65 million people that can't live at their home. And what we see worldwide is that there are a lot of refugees, not only in camps, but in cities. We want to understand these kind of issues and how we can help in a better way, in a more effective way. The cooperation with CARE came about when they started a big project on urban refugees. The problems in Greece started, they were at the border at Idomeni, but they couldn't stay there because there were no facilities, there was no safety, there was no sanitation, there was no help for them. And so we focused further on this topic. We immediately found synergies and benefits from doing activities together to help the refugees, to help the people in need in Thessaloniki and Athens. What we have done over the past months is we have had six students in Athens and Thessaloniki. We provide them the access, it's very difficult to even find the refugees, also helping with the relevant questions to really understand those needs. They have composed a very big questionnaire of over 100 questions and they managed to interview more than 300 refugees from all nationalities, from all backgrounds. It is a responsibility for the work that we do has also a practical impact and can be used by organisations. Local governments, national governments and NGOs like CARE can take better decisions and know more what is going on. So to understand that and to make sure that you address those root causes is I think really important for an organisation like CARE. It also fits with how students these days are not just wanting to learn about data science but also how they can apply data science to address social issues. They say this is a life experience I never have had and this is an experience that stays with me for the rest of my life. If you listen to those stories, if you ask these people the questions, I can only imagine that it will change your perspective on your own life but also on our global community.


So you know those open source libraries? Well, they kind of disappeared a little bit. So we're going to be talking about the unexpected demise of open source libraries. Or it's going to disappear, and we're not going to be talking about anything I don't know. So next step, we have Larentel. So I'm probably going to be the only person who's stopping you and the after party up happy hour thing. So no pressure there. So yeah, the unexpected demise of open source libraries or basically where do libraries go to die or libraries die hard, all of those kind of titles of us using open source projects but maybe not knowing exactly what's happening there. So open source has fundamentally changed my life. As a teenager, for me, I was going on to things like the Linux kernel mailing list, hanging out with people, learning about open source, using tools, basically finding collaboration and communication paths with like-minded people. And it enabled me to learn and do many, many good things, I hope, and just put them out there. One of those was the projects that I actually built about 15 years ago. I uploaded it to an open source page called SourceForge for some of you who might remember it. And to date, I've actually went back and seen how much that has actually been impactful. And I've actually seen this project called Deloradius as part of the resume of some people that have their skills and experience laid out on LinkedIn, which is pretty cool. So it actually changed my life, but it also allowed me and enabled me to basically improve and change other people's life to the better. So I'm glad for open source. My name is Irantal. I'm a developer advocate at SNCC, where we build developer-first security tools, basically help empower developers to not just find but fix security issues along the way, ID is, Saturday and all of those things. Also doing a bunch of stuff on the ecosystems of Node.js and PM, open source all around. So if you have any questions on this in any different way, I'm happy to chat about it afterwards just to find me here. So talking about open source, this is a pretty interesting topic and the success of it. I think it's been successful so far. So I went back and found some information that was really interesting for me, one of which was almost half of the projects at around 2015 on GitHub are MIT licensed, which is interesting by itself. But actually, this is all the projects, everything else. If we look down specifically on things like ecosystem, language-based ecosystems, like NPM, like Ruby, Python, PyPI, things that people store there, we're actually getting to about 90% of those projects being used using licenses that are very, very permissive. And I think that is something that has indeed contributed to the fact that open source has really been growing and advancing at a really increased state. So one question when we're talking about here and the unexpected demise of open source libraries is have we used so much open source that at this point we're actually taking it for granted? And then what happens when we do? What happens when we use open source and just consume it? It's just another S-bomb entry. So with that said, I want to share with you a few stories that really happened in the ecosystems. This one specifically called Left Pad is a story of how legal actions were driven and taken by a commercial body towards an open source maintainer that at the end of it, back in 2016, at the end of this action, it had actually caused widespread disruption of build and systems impacting projects like the Node.js project, like Babel and other in the JavaScript ecosystem. So what exactly happened there? Well, if we take, for example, a regular software development team like this one, well, there are some fallacies there. For example, we have Murray, Jonathan, and Mina doing open source development, which is great. And they're doing also internal development for the company. And it's been scientifically proved that to be a developer and what it actually means is essentially you take some coffee. That's right, you know it. And some code goes out. And so you know about this one already. There's another thing that we've been witnessing recently, and that is that they use a lot of open source packages. Up to 90% of those statistics have been shared out once and about every time. And they use a lot of those open source packages. One of them, by the way, is this person called Azur. He's an open source developer and maintainer of many, many modules. One of them, this team actually uses. So in essence, one of the fallacies is this team is not really a three-people team. They're like a four-people team. Actually, many, many more people because we're using a lot of open source packages. Behind those projects, behind those packages leave other maintainers and other developers that impact us. So let's talk about Azur for a second. Azur is an NPM developer, you know, building stuff on the NPM registry for JavaScript. He has this module called KIK, which accidentally is also a name of an instant messaging application that's been there and a commercial entity that exists on this kind of like a platform. And so he was asked by this legal entity, you know, several emails, exchange, and so on to basically give up the name and use some other name, just, you know, give it up and allow this other company, use this KIK to own that actual name. That didn't really work out as, you know, could have guessed. So as a protest of, you know, not really agreeing with that, then NPM, the registry itself, the team behind it, not able to actually bridge between the two sides. And eventually what happened is he did not want to get a lawsuit or any other kind of like legal actions. So went on and just as a protest action removed all of his modules from NPM. He had many of them, about 250 back then. Just removed all of them and said, you know, bye-bye.  And it's downloaded just a little bit, about 2.5 million times. And so when that got removed, essentially a lot of projects that used it as a nested dependency somewhere down their transitive dependency tree have actually broken up. And this is how this story became to be how one developer actually broke whole ecosystem. So some learnings from this story.  So obviously one of the ways of doing this is to put something like a caching proxy, like something that actually takes both the fact that it also saves up, not just the fact that something could be a network error or just begun from the registry for some reasons, but also the fact that you wanna speed things up and do those kind of installs really fast internally. So Verdecio is an open source private proxy server that does all of those kind of things. It's pretty fast, basically the way that it works. You kind of configure package managers to use that as the upstream or as the actual registry. And then that one is Verdecio itself is used as that source to which upstream are you allowed to actually contact and bring all of these things in. So Verdecio is really cool and it's a great open source software for us to do this. And I think a lot of companies have learned this in the hard way of using those internal proxies, not just for speeding up reasons, but actually to prevent those cases from happening. But also the fact that when this happened, when all modules got published, this was an unprecedented event in the NPM ecosystem because no one thought about the potential threat attack that someone would actually go and unpublish their modules, especially if they're so popular and everyone used them. But when they do, it just impacts everyone else. So it actually brought in NPM to introduce new limitations as if you cannot unpublish your modules if it is popular or used by other dependencies and so on and so on. So we've learned from it as an ecosystem as well, but we've had to learn the hard way. Another story is this story about someone who actually stopped their project from being alive, from being maintained anymore, just because they wanted to be a bit more greener for the earth. So I don't know how you would feel about it, but I'm pretty sure like for a lot of maintainers like gaining the popularity and reputation of using an open source project is something they probably don't wanna cut in just midway as you're developing it. But for this person, not so much. So Meet François is the developer and maintainer of this Faker PHP library. It's essentially a library that allows you to fake data, to random generation of data for tests, things like that. It's pretty cool. It's been there for since 2011 and it has been used to be an SDK for like different languages like Python and also provide different locales like over 70K. So it's like a very, very popular project when you download it as a DevTool, it's three megabytes of size, pretty, you know, not that much if you compare it to like end-to-end tools and like bigger things that you probably pull in. So this is a project that became so popular that by 2019 it had actually been downloaded over 120 million times, many, many contributors, many, many products being dependent on it. But the thing is, at this point of popularity, if something, if you wanted to be conscious of climate change, would you actually consider the size of your library an issue? Sure, maybe it's an issue for front-end and optimization and stuff like that, but for climate change, that is something that François really had difficulties agreeing with and working with. He did not want to be a person who contributes in a bad way to climate change and he actually figured out that potentially what's going on there is about 11 metric tons of CO2 emissions just by all of those downloads. That's a lot and he said, I do not want to do this anymore. So what happens next? Well, first of all, do we get new maintainers? How does open source work? So François had, he's a passive sin and understood like he has been on the open source ecosystem for a bit and he had like really bad experience with transition in his projects to other maintainers. One of his stories is he actually owned another, maintained another project called Propel, a PHP ORM library and when he handed that off to a friend to continue maintaining it, that friend handed it off to another friend, which handed it off to another friend, which they ended off to another friend that said, this codebase is not really good. I'm gonna rewrite this whole thing and do a 2.0 version. That was six years ago, nothing came out of it. So he had a really bad experience and he did not want to just try to continue the project by handing it off. So he said, I'm archiving the project, archiving the GitHub repository, no more releases, no more issues, nothing else if you want, you can forget. You know, he just added a comment. You can forget, it might be better than Faker, the PHP library, just remember it might not be so much fun. So here's another open source dependency just unexpectedly dying off. First of all, appreciating maintainers I think is a really good thing that we should adopt and that relates to the whole sustainability aspects of open source and discussions around this. And of course, you know, health support, the climate change is an important thing and it's great that maintainers and developers are conscious to this. Our last story is about how Mr. M, this person, did not want to do any more free work as part of their open source packages. And I'm pretty sure there's some people here that are contributing to open source, maybe even maintaining it. So we're all using open source projects, but how much are we actually giving back to people? How much are we actually giving back to the projects maintainers themselves and developers? I promise that, unlike the other two stories, I'll end this fairytale one in a positive note. So meet Mr. M.  One of them is this called Faker.js. This is by the way not the same project. The idea is the same, but coincidentally it is of the same name, just not the same one. Different maintainers, different projects, by the way, different outcomes. It's pretty popular as well. This one as well is used to generate random data used for tests and things like that, but it is pretty popular as well, 12 million monthly downloads and just like many other projects on the ecosystem, most of this work is unfunded. It's maybe sustainable, maybe not, but no one contributes to the project. So this person, the maintainer behind a project called Faker.js, went on in 2020 and said in a GitHub issue, I am not going to support this project or you anymore, no more free work from me. And I think as you can see maybe a bit by the reactions on GitHub, most of the people actually agreed with this and support this because it's a project that's used by Fortune 500 companies and we're not doing that well of a job contributing back to all the peers and the colleagues that actually work on open source. So this was his statement and the question is, was this heard? This was back 2020. It wasn't that heard. One of those things that happened is that by 2022, the beginning of January, this person, the maintainer, actually pushed codes into the repository that essentially nullified everything else. They forced push a commit. So all the source code had existed for this Faker.js PHP library was completely removed only if you archived it or forked it before or clone it or whatever, nothing exists. Also they pushed a new version called Faker 6.6.6, understand where this is going, which indeed broke the ecosystem as well. It had nothing in it. Essentially it was an empty project. If you try to use it, it would fake. It would not work. And so he'd actually lived up to his statement, right? Said no more free work, so no more free work. You're not getting any of those packages anymore. Question is, did anyone really care? To understand that, we actually need to dive a bit deeper into what happened there. And so Faker, this project, if the maintainer actually pushes a project with 6.6.6 major version, that is broken, but is this actually affecting all of us? Not exactly. The main stable branch or version, same range of Faker, the library on NPM, actually major version five. The way that NPM works behind the scenes is where usually it auto fetches, minor and patch versions, depending on how you configure things, but you have to be very explicit to just take everything in with a wildcard. And people usually do not do that and good thing that they don't. So the fact that a new version <IP_ADDRESS> is published doesn't really impact many, many people because the amount of people actually use Faker as a new project or automatically get it by builds isn't that big. So something like this wasn't really hurting anybody, even though we wanted to make a statement as this open source maintainer did. But here's a note. This person, a maintainer on NPM, actually owns many other libraries, not just Faker. For example, colors is another important and very important to the ecosystems, but also very prolific one. And it's so popular used by Nessa dependencies that it has over 100 million downloads a month. So Mr. M actually owns both of them. Here's the maintainer of both of those. So what would we do? Perhaps inject something into colors if 1.4.0 is the latest stable version that gets those 100 million downloads a month. If we release 1.4.1, for example, a patch version, probably a lot of automated download tools will actually start getting this. And this is exactly what happened and also developers at least in JavaScript from what I've seen are very keen on just getting the latest and bleeding edge which is, we'll leave it to the end to talk about it. But what's in 1.4.1? We usually don't even look at a source code to realize what we're getting. We just want to be at a bleeding edge. Want to get the new features, the new bugs, new bug fixes. Well, here's one of the change logs. I'll go ahead and zoom in for some of you. This is what was added in 1.4.1. I realize that maybe not everyone here are programmer, but essentially by the for loop and a case of minus of less than infinity. You can understand that this is a denial of service of a loop that once you include and use colors 1.4.1 in your project, the moment that you do and this is getting used, essentially, this is a denial of service attack for anyone because nothing will actually continue working with your application, especially Node.js being single-threaded and so on. So this is what happened. Now remember we talked about Mary and Jonathan and Mina at the beginning and we said they are part of open source ecosystems. This is the fallacy. We do not understand, but they are. So let's see how far this went out. Well, Mary actually works at Amazon and she is working with AWS CDK, which got impacted because it uses 1.4.1. So all of the users who used Amazon CDK have actually had this problem coming in. Remember Mina? He is actually working for Heroku's Ocliffe. Essentially, that also uses colors behind the scene and if you were using that, you would be pulling in automatically potentially this denial of service attack to your application. How about Jonathan? He's working on Jest. That's like a test runner for JavaScript. Very famous one built from Facebook or Meta these days. So that's also been pulling in colors and all of these projects and many, many more. So we are very much part of open source and we cannot deny it anymore and all of those things are actually impacting us. So if something goes bad there, regardless of security vulnerabilities, like projects dying or in some way getting added some malicious code into them that just is bad, we are impacted by this. Let's see further out how impactful this is. So as they later published version 1.4.1 and version 1.4.2, you could see that really, really fast in a matter of hours and days the amount of downloads per each version. We're talking about more than 500,000, half a million downloads within a day or two. These and 1.4.1 and 2 are the impacted one. So it shows you how fast and how impactful this has been on the ecosystem if you have been impacted by colors. Well, let's talk about colors incident learning. What could we have done to have prevented this? Well, first of all, if you use a log file and don't just install everything all the time, then you have a lot of assurances, like a checksum, like a Shah, and all of those other things. But essentially, it allows you to not just get every install, new versions, Semver, whatever, that just went out yesterday, but you're always getting deterministic installs. There are specific commands in NPM that you have to follow both in CI and your development environment to make sure that indeed this is what happens, because you can still have a log file, but use different commands that override it. This is like a bad practice. The other thing is, if that would have happened to you and you wanted to mitigate it, if this is a nested dependency, not a direct dependency, which you can just update, if this is a nested dependency impacting you, then you cannot really do anything to change it. If another maintainer pulls in, for example, Jest, that pulls in colors, 1.4.2, you cannot do anything to tell Jest to do 1.41, until this feature from NPM, now supported by other package managers like Yarn and PNPM, and that allows you to specify this thing called overrides, which we used it also for, as you can see in the reference here, for the node IPC, which is, you didn't cover it here, but that's like another interesting protest work. Let's call it like that. So, this is not a way of doing that. The other thing that I actually want to share with you is, there are many automated dependency upgrade tools, like the one you use and see in GitHub, and there are probably other vendors that do it. The thing is, you have to be very conscious on what you're updating. We're just talking about now, about packages that have been harmed or just stopped in one way or the other, but they might by the maintainer. That's not even like an external attack vector, someone taking over a package. That's like someone that you're supposed to trust within the ecosystem, and all of those automated dependency upgrade tools that you actually get all the time, updates and updates, think about for a second, what if you could have been able to just delay the upgrade, right? And so, this is something that we actually looked into about three years ago, back at Snake when I joined, actually looked at many other malicious incident that happened, and we looked at the data frames for how much time it took for a malicious package to actually be known. So now, unless a dependency is needs to get a security fix, like unless this is a security fix that needs to go in, all of those updates that you will get from the sneak bot, for example, will be delayed by about three weeks, which is what we found to be a very reasonable time for you to update, just for new features or new bug fixes if needed, automatically, but also be a bit on the safe side, give you some buffer time, so the community can vet it and things can get worked out by a bunch of other people and more eyeballs looking at it, if to quote, para quote someone else. So, I said this is gonna be a fairytale ending, so what happened here? I'm gonna go back not to collars, but to Faker.js of the same maintainer. Luckily for us, Faker.js on the JavaScript ecosystem is a very big project. Like we said, many Fortune 500 companies have been using it. So the beauty of open source is that in the case of Faker.js here, this maintainer of collars and Faker.js, another group of people, amazing humans who came up and said, hey, you know what? We're gonna fork the project and it is very influential. There's like a lot to do. People use it all the time. It's a very good project and they have continued it and provided the roadmap and everything that they updated the documentation. It actually brought a lot of different forks and different people who maintained like documentation and stuff like that aside, brought them together so they can actually cooperate on a new place for Faker.js. So big shout out to this team. Big shout out to open source. To everyone here who's doing that and to that team specifically. I wanted to take in just an example of the metric. This is from a screenshot from yesterday. You could see that from just the last month. The activity that has been on the project has been to close down, merge 83 pull requests and more and more. So it's amazing to see also the continuation of projects in the open source ecosystem beyond them being sometimes discontinued in ways that are unexpected. So to end the whole stories here, I'm gonna give you a bit more takeaways unrelated to the specific stories. Well, first of all, I think when developers essentially look for packages, we use the currency of stars, the currency of downloads. If I want to look out for a package, for example, an HTTP library, a client-side, a client library for HTTP calls, I might go into a place like NPM.js and probably the one thing that I'll use to judge is how many downloads are there for it? Actually, it looks like this is a trending up graph event. So it looks like requests here, this library gets in about 15 million downloads by other users, by our developers. So I don't know, maybe it's safe, maybe I'll use it. Maybe, but there are more perspectives that we could look at around the spectrum of what makes a project, what makes a community for a project that are good indications that we shouldn't deny them. So on this one specifically request, actually got deprecated a few months after this screenshot thing. So what led to that is if we could have looked at something that gave us more information. For example, I open it up retroactively now, for example, in the sneak advisor tool, which is a free tool that you can use to look at packages and get more package health information. And you could see, for example, we look at maintenance and we see that it didn't have any commits or any releases in the last two years. So yes, it's getting downloaded 15 million times, but it's not really active. That's a signal that something could actually turn bad. It doesn't mean that it's bad internally or in any other way, but you probably do not want to continue new greenfield projects and might want to consider moving on to other HTTP client libraries on NPN that are not this. Let's take another example, that's the other side of the flip side of the coin. This is marked, a library, a JavaScript library that translates markdown and converts that to HTML. It's a pretty cool library. I've been on the access them also for a long time. And it had its share of vulnerabilities too, but you could see that many of the latest versions, first of all, there are many of those throughout 2021. And even though many vulnerabilities are being triaged to the project, they're getting fixed and new releases are going out. Also, you could see the pulse on the maintenance side of things, showing you that commits are happening all the time, showing you an idea that, again, is not bulletproof and doesn't mean that this is not gonna be discontinued tomorrow, but it's a good indication that says, this is a pretty positive, healthy project. Whoever heard about this mom, right? But I want to give you a different perspective. I want to lend a new opinion here for you to think about here. Not just the software bill of materials. That's fine, we talked about it a couple of times. That's okay. How about the fact that projects are not materials. They are built by humans, by people. They are very dynamic in nature. They change, they evolve. And so to think about humans as materials is a pretty bad concept, I think. So how about we talk about SBOH? How about we talk about the software bill of health? Not just what we use. Not just the vulnerabilities of it, but all those other signals that we may have in the project and around the project and how it's actually being used and consumed and does it have tests and so many other kind of signals that we can get into how healthy the project is as an open source project for us to use. So hopefully some of this has resonated with you. I'm here for a bit. Thank you for having me and use open source safely. It is great. It's amazing. That's it. Wanted to end with a bit of an ending here. This is the one. So the only things that last forever are not really dependencies, as we've seen, and not really air miles. So I hope you used all of those when flying here on COVID and all of those things. I think it's cat videos on YouTube. That's the only thing I could think of that probably lasts forever. And so with that, really thank you so much and thank you for having me. Any questions? Eager for the happy hour. Not sure yet. Okay, cool. We'll have fun people and enjoy the conference.


you today. So today we're gonna start another little art studio or art room series that has to do with organization, which is the bane of any creatives existence, no matter the product. You always struggle with the right way for you and the way you create and what you create with in the space that you do the creating. Like always, it's always a thing whether you're a painter or a needle crafter, no matter what your creative endeavors, it's always a struggle to store your tools and supplies and to store it in a way that makes them easy to use, easy to access, so that you remember they're there to be honest. For me, one of those big things that have been a struggle for me for years and years are my stencils. I've tried everything. Boxes, bins, file folders, accordion folders, binders, all sorts of things, none of which were completely satisfactory for me to be honest. Now I'm gonna start inserting here some pictures of different storage ideas that some of my friends use, including Shelsey, Peg Robinson, Vicki Brown from Messy Table Studios, Cindy Utter and a bunch of others. I'm trying to put their names at the bottom and I will try to slow down the timing of the pictures so you have a really good chance to look at what they're doing. Some of them are going to do videos on what they do for storage and if they do do a video I'm gonna link it in the description below, so go check it out and some of them hang their stencils up on shelves and rods. Some of them use bins and baskets. Some of them have them in a drawer. Some have them in accordion folders and that works for them. There's no right way and there's no wrong way. That's what I want to tell you about this. I just want to give you all different, and as do they, want to give you all different ideas and ways to store your stencils. Today we're going to talk about stencils. There will probably be other videos coming up if you have specific things you want me to give you ideas on how to store. Leave something in the comments below. Alright so you'll see in the pictures some people use binders, some people use baskets and bins. Like I said, some use o-rings to hang them up. Some use peg has a small clip it up. If you don't know what the clip it up it was made a very long time ago. It's since discontinued. It was kind of pricey at the time when it was new and she has one of the smaller tabletop versions. They did make a large floor version which was I believe at the time like $150. I've always wanted one but they don't make them anymore and they were really expensive. So we're going to go back here to the table and you'll see all these binders. So this is what I was using and my most recent incarnation for stencil storage were binders. I had different sized binders for different sized stencils and in the old space which was a very small little bedroom. This worked although they were in a closet so every time I wanted to use one I had to dig it out and then you know that gets to be tedious and a pain in the neck and so then you only use the couple that are just out all the time and you never use anything else. Yada yada. You guys get the idea. So this wasn't perfect but for the time it was a good solution. Now I'm in a very much bigger brighter space and I have a lot more room to spread out and have things more open and accessible. I don't have more things than I do in the old space. In fact I have less because I got rid of a lot of stuff but what I do have is more open and accessible to me and I can get to it easier including the stencils. So this is what I did used to use and they're a combination of scrapbooking binders with the sheet protectors in them, small photo albums, Avery, small Avery. These are Avery binders, small ones, but wide like these are I don't know two and a half or three inches two three inch binders. Now these I've never seen in this store. I get them in Amazon. I'll try to remember to put a link below. Some of them had just your standard small binder dividers and sheet protectors in it. These were the ones that my personal design line of stencils were in. And inside of them I used these or Tim Holtz sleeves and these actually are very strong sturdy plastic. They've been used for a few different things over the years and you have to repunch the holes because they only come with two. But they work really well actually in here. They're a little bit thick for the binder you can see but they hold the stencils very well. And then for the really big ones I got this really large binder. I think at scrapbooks.com again it's a scrapbooking binder made to hold 12 by 12 sheets of scrapbooking paper. It holds the big stencils very well. So I transitioned out of these literally last night. I was doing one of my many Google searches about what I could possibly do with the stencils which is as I said earlier the constant bane of my existence and crafting and creating. I'm always on the lookout for something that was more affordable but comparable to the floor model clip it up. And I will try to find a picture of the clip it up and insert it right about here somewhere. I stumbled across across something the other day and a couple of weeks ago and I was like aha that's what I need. It did take a while to get in of course as soon as I ordered it there was some issue with weather and getting it over here to the West Coast but it's from Chicago so there was some flooding and anyway I did finally get it and I set it up last night so I am going to cut to a view of that it is a retail floor display intended for hats and scarves. Coming from a retail background I have been thinking for a long time there must be a retail display out there that looked like the clip it up or something comparable that would work just as well. I just hadn't found one until I stumbled across it a couple of weeks ago. I'm going to leave a link to the one I purchased in the description below. They do have a countertop version that only has one ring on it it's a little bit smaller it's 14 inches instead of 16 but it would fit on a countertop and if you have a smaller collection of stencils that might work for you and I don't know because I didn't order that one but maybe you might be able to message the company and see if they could send you a spare ring and you might be able to put two rings on it. I have no idea but you might ask if that's what's going to work for you but let me go over to the setup of what I have and I'll be right back. Okay so how pretty does that look in the corner? So this is the floor display from the marvelous manufacturing company it is a as I said a retail display and it comes with four rings you actually can get it with five I opted for four and every single stencil I have is on there. I have my little ones on the top these are all the ones for my design line and then more of the similar size like six by six and six by eight and a few homemade ones and then they're really big ones at the bottom. I do have a few of them that are singled out and sections that if I use them I want to make sure I tell you all who they're from like my friend Mike Deacon and they're not for sale. I do have a few that are singled out like these from my friend Mike Deacon so if I do use them in a video I want to be able to tell you all where you can get them. For the most part I haven't been great about keeping track of whose stencils or whose but I do know some of them so I'm going to try to mark those that I know but I am loving this. I can literally just go over here.  It's going to make it so much easier. I can see everything that I have. Now it didn't come with the clips. The clips were purchased separately and I will leave the Amazon link to the clips that I bought and I do I do actually need more clips because I don't have enough. I had to sort of bag some stencils because yeah I run out of clips so anyway so you want to you know I the clips come in bags of 120 and I need another bag of clips so I won't tell you how many bags I've ordered because that's embarrassing but let's just say if you have about this many stencils you're probably going to need three bags of clips. Just saying. So I hope this is one thing that gives you some idea of what you can do with this particular product in your creative space stencils whether you're storing them in binders or you're going to get one of these style of racks to clip them on to. Now you do occasionally see the clip-it-ups the small ones and the big ones on eBay. Every now and then they'll sell it on eBay for a really good price. Most of the time it's way too much money but it's also something that you want to maybe keep an eye out if you like this idea when you go to the sort of creative and scrapbooking garage sale type things that some of our local mom-and-pop scrapbooking shops will have once or twice a year. So you want to keep your eyes open also maybe at garage sales or thrift shops but or maybe stores that are going out of business but it's a really great idea. You can use something like this which is also a retail display from my craft show days. The problem with this is your stencils would fit back to back so then it's not as convenient to get them off of the rack I don't think but look at your retail displays. This one with shipping was still less than half price of the original clip-it-up and that included shipping. Without shipping it was $46 I believe so that's a lot less expensive than the clip-it-up. So I'm going to link my friends videos in the description below again really think about what weight works for you how do you create in your space how do you access things like your stencils what's going to make it easy for you to get to them and remember to use them maybe just having a basket of them under your work table works maybe having them in a drawer works maybe putting them in an accordion folder works you have to work with your space and how you create and make it easy for you so that you actually use them and you don't have a product just sitting in a closet that you never get it to so that's it for today don't forget to like share and subscribe hit that little bell icon if you want notifications of new videos when they come out and in the description below or all the links I spoke of plus if you want to follow me on social media or support the free content here on YouTube or over on Facebook click on my link tree because you're going to find a bunch of different sites that you can do all of those things on including my Etsy shop red bubble patreon my tip jar all that stuff so check it out and the most important thing go out and have a great day everybody do something nice for yourself because you deserve it enjoy those stencils and I'll see you later bye guys


Sideways doors, silver doors, handle doors, look at these doors! Left, left, left, left, left, left, left, left! Exit doors, turkey doors, enter doors, epic doors! Left, left, left, left, left, left, left, left, left, left, left! Oh! Every door, passenger door, every other kind of door, maybe just a silver door, because that's the only kind of door that we have here. Look at these doors! Come down to Jim Silver store store of doors where you can buy silver doors


Hi, everyone. So I am Konal, founder of Analytics Vidya. Analytics Vidya is world's largest analytics community. And we started this journey about two years back. Before I actually take you through what I wanted to discuss, just a quick show of hands. How many of you know about Analytics Vidya? OK, and how many of you want to be a data scientist or are already a data scientist? Sure, thanks. So as I said, so Analytics Vidya is actually a community of analytics professionals. What we want to do is create a community where people come in, share their knowledge for free, and essentially contribute to learning of the people who are there already as part of community. We reach out to about 3,000 people month on month in various modes. So blog being our most popular mode. So on the blog front, we write various articles. So this is our home page. There are specific resources on Python. For example, we have a comprehensive learning path on Python, which you can come and see. Again, it starts from bare basics, why you should use Python to what are the tools you would need, and then what are the libraries. We do various medias again. So starting from infographics. So this is an infographic on machine learning algorithms. So that's the first part of what we do. The second part, essentially, without which a community cannot work, is discussion portal. So in terms of discussion portal, again, it's a very active discussion portal. Typically, people would get answers to their queries in less than three to four hours. And there is a constant interaction, constant feedback. So this we launched about seven, eight months back, so January to be precise, actually. And it's building up nicely coming along well. These are some Python-related questions again. And finally, lately we have started some data science hackathon. So for those of you who know Kaggle, it's a shorter form of Kaggle. So we run two to seven days competition. And again, the focus is on learning. So even the people who are on top of leaderboard actually share their solutions with the larger audience. And then they learn and they improvise. So at the end of the day, everyone comes out by learning and doing a lot more learning than what you would do in, let's say, a period of 15, 20 days. So why am I here? Why am I sharing this platform? The idea was that I saw a lot of interest in last two days in data science. In all the talks which were related to data science, I could see a lot of interaction. So I thought I'll just come up, show what we have been doing. I would love to interact with you. We would love to take your questions, help you out. Or in any way, we can help spreading data science in India or globally. We will love to do that. Thanks.


Okay, let's get this started. Hello, everybody. I'm Rosela's blended oh in Susie I lead the team that takes care of SDN and an avi and I was a core reviewer for Newton few years ago today with me Miha, yeah, I'm Miha Rostetsky. I work as a software engineer at Susie nowadays I'm mostly working on psyllium projects, but I used to also be active in open stock cola some years ago and today we are going to tell you about improving security in your container infrastructure on the stock using cure and psyllium and About our plans to integrate those technologies and the little experiment we Did nowadays to prove that there is a good path to try And we will start our Presentation from explaining what is BPF filter. Are you familiar with it? Who knows BPF? and eBPF okay, so BPF is Berkeley packet filter and this Mechanism in kernel by which you can run programs on the virtual machine inside kernel and those programs Monitored Cisco's and they can monitor and filter packets and in our case in case of our presentation BPF is used mostly for Filtering out packets based on rules we defined before and BPF is a very programmatic very general you need to write a source code in C and then by LVM and C-Lank compile it by code Which is which can be loaded by the kernel by its jit and verifier and then after Getting the program via jit. It's running in the kernel virtual machine and Celi is a project which brings BPF to containers so Celi um has several components The biggest of them is Celi um agents, which is ironic on the on each note and It translates rules regarding some container or some network namespace To a BPF program which filters traffic specially for for some container So and Celi um also provides a CLI for the user to define those rules and There is also an integration with Kubernetes and Kubernetes has a concept of network policies by which you can define which IP addresses or Which labels or which namespaces can connect with each other? so Celi um has a functionality of watching your Kubernetes cluster and translating network policies into BPF programs Celi um's as a project started in the times when Kernel still didn't have a defined path to Start replacing IP tables with BPF But nowadays there is an intuitive insight in kernel called BP filter Which aims to be an alternative both for IP tables and NF tables and Replace all far-owing stack in kernel with BPF programs, but Celi um started earlier and it's fully ready and usable solution to Bring security to your containers by BPF programs instead of using IP tables So how many of you have heard about Celi um before? Okay I will try not to scare you with this microphone the volume is pretty high So let's now introduce the other component of this integration that it's a career Actually, the idea for this talk came out during a team brainstorming You know me how has been contributing to Celi um for a while and the rest of the team is pretty familiar with open stack So we just thought it would be great to use the power of Celi um in open stack How can we actually do that and then we thought okay, we can try with career so we'll just explain you today how we did that and Yeah, the future work that's still we need to do So going back to career Let's take an historical perspective Many of you might remember a few years ago like containers were not as popular as they are now and you know open stack it's been great in dealing with the amps and their metal but Yeah containers were a kind of new thing so there was There wasn't a unified Infrastructure to be able to connect VMs and container So one of the earlier solution to do that was the approach that Magnum took So basically to run a container inside the VM Which you know gets the job done, but it's not optimal Because you know some as the unsolution they are they are already native for container containers and you lose that when you run them in a VM and The other drawback is that you know Like in neutron we already use a tunneling to take care of the network isolation and most Container networking solution are also using tunneling like for example flannel So you ended up having a double tunneling like the neutron tunneling and then inside that the Tunneling by flannel for example And we'll see in the next lines how career avoids that and I'll go back Sorry. Yeah, I just wanted to say a few words about the main components of career So we have the career controller that runs on the node where the Kubernetes API is running and its main task is basically to watch for every crowd operation that it's performed through the API and Taking care then of creating the corresponding neutron resources like for example You create a pod in Kubernetes and then the career controller will make sure of creating the corresponding neutron port the other important task is that It will then annotated the Kubernetes resources with the neutron information and This is done because the other component the career CNI Which I think it's outdated now. It's the CNI demon This component will work will be in on the worker node So where a cubelet is running and it will watch for annotation on the resources So when it sees a new one it will then perform the Interface plugging and we'll see in the next lines how that works Thanks So in the graph here, I put two containers. They are on different networks The blue one and the red one and they are running on the same VM So how do we achieve the network isolation using career? So career is Utilizing the trunk port which is a relatively new concept in neutron was introduced two years ago To be able to get target traffic up to the VM because before that it was not possible to get Tucked traffic up to the VM. So using the trunk port You define the trunk port and then you can define several sub ports and every support is associated with a specific Milan So in the case of career So in the network topology, we have the VR int which is you know The usual neutron bridge that takes care of you know, tagging and untugging the traffic according to the network Then we have the trunk bridge which is the bridge created by neutron to represent the trunk port on That trunk port Career will create two different sub ports One to take care of the traffic of network one and the and the other one for nectar network to and they will use Different villains. So what happens when the when the packet goes through the trunk bridge and Exit one of the support it will be tagged Correctly, and it will then reach the VM Tagged and I will be delivered to the proper container So why are trying to integrate Serium of career? first reason is that we Like to bring support for network policies I'm currently career has support for network policies, but it's using mapping for security groups and We think that's trying out Integrated with Celium which will filter out the packets on the host or VN where containers are by BPF programs is an interesting Idea to try out And so we did some experiments And in this experiment Curio Kubernetes controller is running as a deployment. We didn't use curio kubernetes CNI or CNI demon, but instead we rewrote all functionality which curio kubernetes CNI is using to see Liam CNI plug-in And in Celium we use the direct fruiting mode. So we thought Celium not to take care of Transitioning the packets between nodes why we did it like that because of course Copying functionality from curio kubernetes CNI to Celium CNI doesn't sound good, but at the time we tried this experiment Celium Was pretty monolithic and it still is but in the next version will change so Celium CNI takes care of IPan and creating the network and then creating an endpoint and Taking care of generating BPF programs for that IP address and we had some challenges with that approach Yeah, we needed to Inside Celium CNI plug-in Read the information which curio saves in the annotation about networking and creates open-v-switch VIFs and the schema of How the whole solution works looks like that so when user is creating a bot It's gets created in kubernetes API as an object to keep that is creating sandbox and then cause the CNI plug-in to set up Network namespace at the same time curio controller which observes Objects curating kubernetes requests the port creation for the pots then Neutron is called and Neutron creates a port and then curio controller gets the information about that Neutron port and Saves those data in site pot annotations in kubernetes API and That's how at the end CNI plug-in is aware of all the networking configuration But I mentioned that I Monolithism of Celium is going to change it will happen in version one point one point four which Will be the next version the idea is the couple Celium into three layers networking load balancing and policy and and make it able to use only some bits of Celium so Celium will allow to use any other solution for providing the networking and Will allow to provide even just load balancing or just security policies on top of that There is also an ongoing work in Celium to Use that kind of the coupling for Celium flano integration because there are a lot of people who are using flano in their kubernetes clusters and they would prefer rather to Use only those parts of Celium which deal with network policies instead of replacing the whole network stuck in their cluster and After that the coupling is done in Celium that will be a perfect opportunity to implement also cure support upstream and In clean way which should look like that So the mechanics of creating port and how cure controller talks with neutrons stays the same but then curious in I would take care of Creating VIPs and reading new from data and then Celium on top of that Getting the data about what IP address was assigned by Curious C&I will be able to create Bpf programs for free filtering packets and observe the kubernetes API for network policies and Now Rosela will talk about Packet from special yeah, it's the same graph with some modification So just to get you a visual understanding of what's different when using Celium So as I was saying a few minutes ago With vanilla career Basically the packet traversal is you know, let's say the container is trying to send a packet outside Then you know from the container the TCP packet will be created It will go through the virtual interface. It will go to the trunk bridge. It will go through the support will be tagged It will go to the integration bridge and at that point it will be filtered like either using IP tables or the obvious firewall If you're using Celium Then as Micha was explaining You have this BPF program that it's hooked in the kernel. So when the container issues a Cisco, so let's say a connect That program will be executed immediately So you see that we are jumping a few steps and the other advantage is that The BPF program is much more flexible compared to IP tables, for example, and It's also extensible like you can basically Stitch few programs together and then you could also make use of xdp to you know increase the performance farther And that this is the time for demo in our demo we have Kubernetes applications which simulates Star Wars the station death star and star fighters tie fighter and x-wing and Those tie fighter and x-wing will run as a spot as spots in Kubernetes will label them accordingly to make it clear that tie fighters and Empire star fighter and x-wing is Alliance star fighter and From those spots, we would try to use death star API to land the ship and Celium will take care of preventing Alliance star fighters in landing on death star so This is the Installation on death stack. We have installed death stack with cure and Kubernetes Which is running along so and that's Kubernetes installation should be considered like Bermetal installation But using open stack So we created pots We are waiting for Those pots to be created. Is that font size good for you? Okay, x-wing is running death star is running tie fighters running and All of those pots have IP addresses Assigned and we will check now whether each of them has a corresponding new from port So first you are checking whether there is a new from port for death star pod and as we can see We have such a port now, let's check the star fighters and They all also have new from ports So the communication between cure and Neutron was successful and Now we have no network policies applied. So everything is permitted But firstly, we will try to ping All the pots between each other to see whether we really have a network connectivity and now we will try to land x-wing on Death Star application and It's successful because we have no network policies yet. We can land any shape But now we are going to create a network policy which will prevent it So let's try to do this URL again from tie fighter It was it's successful, but from x-wing it will hang Because the pf program is filtering out a traffic to from x-wing parts to the death star application so This is the end of the demo I will show you the young files definitions of applications So the death star is a savings of the employment and tie fighter and x-wing are Just normal pots. They have according labels or organization empire organization alliance and The definition of the network policy Allows to Connect death star only if Organization label matches empire So what's without that label cannot connect to death star and the future work we are going to do is to be because in that's very simple experiment We didn't touch the topic of load balancing and we will of course after the coupling of our bits of Cillium I mentioned we would try to implement everything That's all what we wanted to show you today. Thank you very much for listening And yeah, do you have any questions? Yeah, I've seen that the kind of network policy that you're using doesn't look like a default one But it's a silly network policy. Is that some kind of CRD you are applying yourself or what's the difference with the default one? It's a CRD. You can use normal network policies. It will work as well with Cillium But Cillium also provides a CRD with network policies which also support L7 rules because Normal network policies in Kubernetes Only support L3 and L4 rules, but with extended Cillium CRD you can also Provide rules for L7 but career is there any kind of modification you have to on for instance today a default network policy driver To run this or how did you implement this? Well, I Didn't use CureCni at all. I rewrote everything to CilliumCni and I used the version of Cure which didn't have network policies at all. I used this table Okay, then maybe we'll get to speak about that later because we have basically refactored all the network policy support You know, anyway, so we'll talk offline after that. Yes as for as far as I know nowadays Please correct me if I wrong but Cure has an implementation of network policies based on mapping to Mapping to security groups in OpenStack. We do definitely have that but we have refactored that So we are now using CRD. It's basically to minimize the amount of coastal neutron And I was thinking that maybe you could we could be making that but that's a depth. I mean that's A depth session if they've talked out we might be writing this offline because maybe not everyone would be interested Okay, now so that's the gate and thing Yeah, that sounds like a good idea to consider. Thank you Are there any other questions? Okay Hello Um, I have one question you're using pot annotations for transporting basically the neutron information Is there any security implications like that if you modify the annotation as a user that you can like Move into other networks or something because I'm not sure how If the user is able to edit pot annotations, then yeah, there is a release I think so it's all about setting up airbag and possibility to edit things Yeah to prevent unauthorized users to modify annotations But I think that's a little problematic because you can't with airbag you can't prevent people from Modifying annotations and they have to modify the pot. I mean they are the ones creating the pot Yeah, so everyone who who can modify the pot unfortunately Okay Thank you So that's yeah, that's that's a good question and good problem to consider in future Just one question Can we use other neutron plugins along with cilium as a layer seven policy? Are you planned and that experiment we were using only OBS plugin But of course when implementing upstream We'll try to support everything which curry supports But that experiment was obvious only and other questions If not, then thank you again for listening to our presentation Thanks, everybody


So welcome everyone to our session on BPF where we hope to go beyond the buzzword and explain how this call the next technology can have a real impact on your daily Kubernetes experience. My name is Andy Randall. I'm the business guy at Kinfolk, an open source company based in Berlin. My name is Albon Cricky. I'm a co-founder at Kinfolk and I'm the director of our labs team, which is actually very active in various Kubernetes and BPF related projects. Yeah, and I know it's always a struggle working out who's speaking in a joint talk. So to help you out, just remember that I'm the one with the British accent, that's Andy. And I am Albon, the one with the French accent. So first of all, hands up those in the audience if you've heard of BPF or EBPF hands up. Okay, I see a few of you out there, a few at the back, maybe not so sure. So for those of you who aren't so clear, the concept of EBPF or extended Berkeley packet filter is pretty simple at a high level. First of all, it allows the user to write programs that run in kernel space. Now, as Brendan Greg of Netflix put it, this is a fundamental change to a 50 year old kernel model, because up till now you've been used to writing programs that run in user space. Now you can actually run them in the context of the kernel. Now, these programs are not completely unrestricted. There are certain hook points where they can attach into the kernel and data structures and helper functions that are available to them. And lastly, they run in a restricted virtual machine environment, which acts as a sandbox. And all of the code is verified against certain strict conditions to ensure that, for example, the code will complete and not hang in the kernel in an infinite loop. So you care about this because as a Kubernetes application developer, you're probably not going to be writing BPF code, but you are going to be using capabilities that BPF services to use. So the first of those is potentially some of the fast networking and customizable networking capabilities that you get available with BPF. Next, anyone who writes applications needs to debug applications and BPF provides some pretty cool tools for surfacing things that are going on in the kernel that help you debug either feature problems or performance problems. Lastly, from an operations perspective, BPF gives you some capabilities that really help when it comes to monitoring applications and also particularly monitoring them from a security perspective to check that you haven't got attacks in your cluster. So these are all pretty good reasons to care about BPF, I think. So at this point, I'm going to hand over to Albon to talk to you a little bit about the history of BPF and how we got to where we are today. Thank you. So it started in 1997 where we had the first batch of BPF. It was used for TCP dump so for network capturing. And that's pretty much it until much later in 2014 where we have the new eBPF for extended eBPF. So Alexi Starover-Toff comes up with the idea of a universal in kernel virtual machine where we are kind of different eBPF program running together for a variety of applications. A bit later, in 2015, we have the Iovizer project established that comes initially from Plumgrid technology from Plumgrid and then it gets moved into another Linux foundation, Umbrella. A bit later in 2016, we have XDP for Express Data Path. And that's the idea to apply the BPF program very early in the network stack so that right in the NIC ideally we can filter packets on a block unwanted traffic. So it makes it very fast. In 2017, we have in the Linux kernel 4.11, we have new data structure like LPM maps that make it possible to filter large amount of IP or CIDR on the network traffic. In 2017, we have a feature called SOC map. That's a new BPF map that's used for intra-host network to make it very fast by bypassing the network stack. That's used in Cilium on Istio when we use Istio, the service mesh together with Cilium, for example. In 2018, Facebook announced the project Catron, that's a BPF load balancer, and Facebook announced to the world that they are using that on their networks. In 2018, we have a new BPF helper function for filtering events by C groups that's used for containers. For example, I used that in trace loop and inspector gadget to figure out which trace came from which pod and which container. And lastly, in my section in kernel 4.8, sorry, 5.8, we have a new kind of BPF ring buffer as a communication mechanism between the BPF program in kernel to user space. That's a new ring buffer that is much more efficient memory wise. So I picked a few of those items to show how it's relevant to Kubernetes, for example, with Cilium or Istio, for tracing tools on Kubernetes. So let's take these capabilities that I've talked about and look at the landscape of tools that are out there that you can use. So the first layer that I want to talk about is there's a set of low level tools, which a lot of them have been around for quite a while, where you get very low level access to BPF capabilities typically on an individual host at a process level, something like BPF tool which actually ships with the Linux kernel, or the BPF compiler collection, which we'll talk about in a minute. So for coming at BPF from a programmer perspective, and you want to be able to write programs to inject into the kernel, then there are libraries available that you can use that shortcut a lot of these capabilities to make it a lot simpler, whether you're running in Go or C++ or Rust or Python, there's probably a good library for you out there. So when we get the next layer up, we start to look at more complete solutions and less kind of individual APIs and libraries and in security and networking space is one of the main areas for the BPF is used. This is one of the first BPF enabled networking solutions in Kubernetes, but Calico has added it as well recently, Catron, as I mentioned, Valco is used in for security applications from Sysdig. So a lot of things that you can plug into Kubernetes there. The last layer in visibility. This is, I think, you know something that anyone from an operations perspective should be looking at these tools to say, you know, how can this give me greater insight to what's running in my cluster. So things like we've scope or Hubble actually provide the capability to see in real time what what network connectivity is happening, how things are connecting in the cluster. There's a few other tools as well which don't fit into any of these categories, things like the Cloudflare BPF exporter project or net cost, which actually analyzes your network traffic to give you some idea of what it's going to be costing you to run that in the cloud. As an example of one of these tools, Hubble is from Cilium folks sits on top of the Cilium CNI plug in and surfaces up metrics and and also analyzes kind of a graph view of your traffic in the Hubble UI and look something like this, you know, nicely displayed to see what the network traffic is. And there are other tools that do the similar thing so we've scope actually quite a while back, the team at Kinfolk work together with we've works to author the agent that collects the data using BPF about network connections. So here you can see which pods talking to which pod, all thanks to BPF. One of the nice things about this is as a part of this project, we developed a library called TCP Tracer BPF. And, you know, that's available so if you wanted to build something similar, a lot of the base capabilities are there for you to build on the BPF compiler collection we've mentioned a couple of times because it really is the granddaddy of all these BPF tools. The nice thing about BCC is, it's very kind of Linux style so these are command line tools with arguments that do one thing do one thing well allow you to snoop and control your system. You know, really quite easy to get up and running and using any of these tools and I recommend you check those out if you haven't already. BPF trace is kind of the cousin to BCC, maybe less well known and less frequently use because it does have a bit of a learning curve, but it's great for quickly writing custom BPF functions it's got a lot of flexibility for tracing and debugging your Linux apps. As you can see with BCC on BPF trace there are a lot of tools really powerful one that come off the shelf, but they don't know anything about containers or Kubernetes so if you run Kubernetes and you want to run that on your Kubernetes cluster. There is no easy way to use them. But if only there was a really easy way that these people who running Kubernetes clusters could take some of these tools and apply them in Kubernetes. Yes, if only that's where can folks inspector gadgets come in comes in. So you can think of inspector gadgets as a Swiss army knife of BPF tools or gadgets for communities. Some of them come directly from BCC with a tiny wrapper on top of them to make them Kubernetes aware and some other are developed by default independently outside of BCC. So what do we need to make BPF tool communities aware. We need a few things first the granularity that we care about. We don't want to trace at the PID level process level. We don't necessarily want to trace all the process on the system. We only want to trace process from a specific parts usually. That's not something so easy to do because BPF programs running in the canal they don't know anything about Kubernetes spots or Kubernetes labels. So we need to do a bit of work there. Another thing we want to do is to aggregate the information by labels so we don't want to see the piece or to select individual process but aggregate using Kubernetes labels. The last thing I want is to have a cubes detail like experience so developers will not need to SSH on a specific node. They should not need to know in advance. This is the node where my product is running. They should have cubes detail experience where they have a common line interface directly for that. So ideally that's a cubes detail plugin which is exactly what we do. It looks a bit like that. So at the bottom left I'm on my laptop. I use a cubes detail gadget. That's a cubes detail plugin so it execute the cubes detail gadget plugin. And then it only talks through the Kubernetes API. It doesn't SSH when you know it use Kubernetes native concepts like birds on the mindset and so on. From there, Inspector Gadget deploys a board on each node. That's a demand set. And each of them will be able to execute the different gadgets. And it will execute, for example, trace loop or those from BCC. And those will install a BPF program in the kernel. And the kernel will be able to gather the information and send it back up to the CLI to cubes detail gadget. So what do we have today as Gadget in Inspector Gadget? The first one here is capabilities. It's a way to get information about capabilities that are exercised in your pod. The use case is, for example, you have a pod that needs some privilege. Maybe it need cabinet admin, can season me, but it's quite difficult to know if you are not deep into this level of kernel knowledge. So what often people do is just to give all the capabilities, but that's not so great security-wise, instead of just giving what you need. This tool allows you to run the pod and see what actual capability are exercised by the pod. And then you can pick that information to write your pod stack on PSPs. Other gadgets in Inspector Gadget are open-sloop, exact-sloop, bind-sloop. They tell you when a file is open, when a new binary is executed, or when there is a new TCP pod open by your pods. So you can see this information in real-time as it happened. Other tools are TCP top and TCP tracer, so that gives you the network traffic happening inside the pod and what is the most voluminous traffic with TCP top. And another is TCP tracer where you can see one new line for each new TCP connections to see what's happening in your pods. Profile is a CPU profiler that this one comes directly from VCC as well. It's useful to see why something is slow. Sometimes it's not so easy to see which pod or which thing is slow. One example I had to debug is when there was a Kubernetes system with way too many IP top role for some reason there was a bug in all versions of Kubernetes. And then this generates the network traffic to be slow, but that was not so easy to debug. So in this tool, we could see what are the network, sorry, the kernel stacks on the user space stack that are the most often called, and then I can see what's happening there. The next loop is a call it a flight recorder. It's a record of the system called done by all the pods and keep them in a memory in a ring buffer. And just in case something crashed, then the user can ask what happened and see the last few season calls exercised. And the last one I'll talk about now is a network policy advisor. The use case is when you come to develop to implement security on your pods, but you come to a project without any previous knowledge of that, it's quite difficult to know what is supposed to talk to which other pod. So if you use the network policy advisor, you can run your pod, see what kind of network activity is there, and it will automatically create network policies that you can look, see if they make sense and if they make sense, pick them up. It's a lot faster than creating them from scratch when you don't have any knowledge of the project. So let's take a look of a couple of those tools in actions. Let's see. The first tool I want to show you today is execs new. So that's a inspector gadget tool that allows me to see what kind of program are executed inside each pod. So let's get started. Qtl gadget has to run inspector gadget shows the tool execs new. And then I don't want to filter. I don't want to get information about all the parts, but only a specific one, I will select the namespace default. And then I will use Qnity's label selector to choose only the pods with the label run equal cooking. So let me run that. I have only one node in my cluster on Miniku. And now it starts to record the new program being executed. There is nothing to display yet because I don't have any pod to record with that label. So now I will create a new pod. So Qtl run with some options. And I will select the image fedora. I give it the name cooking so that it will have the label cooking. So it will be selected by my inspector gadget tool. And in there I will execute a script, a shell script with a search. And I will use this anti-pattern which is to call shell script on executed for the purpose of this demo.  And I pipe the shell script into a batch directly. So here I don't know what this script contains. So hopefully inspector gadget will tell me what it is doing. First it takes some time to download the fedora container image. And when it's ready, it should, here it is. I start to see the commands. And here I see it will execute grep, cart, org, mkdir, rpm, and so on. So I see it downloaded rpm file here. And then it's installed with rpm and so on. So in this way I can use inspector gadget to select a specific pod with labels. And then see what it's doing with xxnoup in this case. But I can also use opensnoup to select to see what are the new files being opened. Or bindsnoup to see if it's listed on a TCP port like for AT for example. I can see that in real time in my terminal. Okay so hopefully it will finish soon and then I will show you the next demo. Let me stop that first. Okay the next demo will be about the network policy advisor. First I will show you that I don't have any pod running in the demo name space. And then I will deploy something there but first I will use the network policy gadget to record all the network connection that will happen. And try to generate a network policy from that. So let's start with kipctl gadget network policy. And the first step is to use this sub command monitor to record the network traffic. And I will only select a list of namespace. So in this case only one. And I will output the result in this file network2is.log. Okay so what it's doing is installing the BPF filter. And every time there is a new connection, a new network traffic in that name space it will record something in that file. For now there is no traffic because I don't have any pod. So let me deploy something. So kipctl apply in this demo namespace. And then I will select this Kubernetes manifest file. As you can see it contains a lot of services and a lot of deployment but it doesn't contain any network policies so far. So all the pods are free to talk to each other. The only thing I need to do now is to see what traffic is gathered and what network policies are generated. Let's see if my pod are created. kipctl get pod. Dash end demo. Okay most of them are already started. Let's see one of the difficulties to get started. So that's good enough. Now let's see what has been generated. I will stop the recording file. The recording gadget into this file. And just to show you how it looks like it just a piece of JSON that lists the different TCP connection that happened there. But you don't need to worry about that. You don't need to read that file actually. What I do is to use inspector gadget with the network policy. With this report that will generate the network policies. So attack this network trace file that I just generated. And then I will create network policy. So now I can look into this network policy and that it contains just the list of network policies that has been generated by inspector gadget. So I can see for example that the card service is about to talk to redis on this TCP port and so on. So it created a network policies for all the different traffic that has happened. And what you can do now is to see if that makes sense. And if it does, you can use that as a base to what your network policy that's a lot faster than writing them from scratch. Thank you. So that's it. Thanks, that was fantastic.  I hope you got excited by that and inspired to go out and try it on your own clusters. Thanks for coming to our talk today. And we look forward to your questions.