Well, thank you guys for doing this. I know it's probably getting late over on the other end of the world So thank you. Thank you. Thank you for I don't know wanting to hang out even if it's kind of late in the evening And I know it's gonna just gonna get later Yeah, thanks for having us. Yeah It's okay. I think it helps that we're in lockdown. So we have some interaction. Yeah Well, I would like to make this as cool and casual and informal as we can be I like I hate the word interview right because I don't think that's right I just think it's hey us just hanging out and chatting having a conversation Is that cool with you? Yeah, yeah, the stuff that I Sounds pretty wild to me the stuff that I put together isn't by any means meant to be like a strict script Or we don't have to stick to that like you guys can answer any part or talk to any part at anything And we'll just kind of roll with it. It's a flow whatever and obviously if there's anything that you would like Don't want heard or said or you accidentally released some deep dark spoiler We can obviously like edit out as long as you're still cool with this going online Yeah, so bad you got the NAB us password ready Should we just kind of like roll into it get right started and can I badger you with questions or? Yeah, yeah, sweet Okay, so just to kind of like set the stage Give it like the hill foundational baseline for the whole conversation. We're talking a lot about try hack me your baby Can you give me a little bit of a background on what try hack me is? How do you define it as the creators host maintainers developers, etc? So whenever whenever we're explaining try hack me to people We're always the way we describe it is that it's a learning platform that makes it easier to break into security So our idea for try hack me has always been that We want people to come to try hack me whenever they think of learning security. So track me should be the first port of call Yeah, just like I think one a shoe and I met like a few years ago And it's the cut sort of the common thing of you know a problem that we actually wanted to solve We were thrown into a learning environment, which just wasn't really helpful was really buggy didn't really work Didn't really teach us anything And so if we wanted to find a fix for that solution and that was you know having a yeah and all in one Sort of virtualized lab for learning and breaking into security. Yeah So that was kind of like the idea or inspiration behind it Like you just wanted they there wasn't something exactly like what you wanted already out there Yeah, so Bennett like Ben Ben mentioned Met we met your attention and obviously part of the internship was doing some security training on stuff like web apps networks And we like we found it quite hard to learn because it was at that point. It was just here's an IP address Here's an application, right and like it's quite fun doing challenges sometimes But and you like like going down the rabbit holes like really good fun But like when you're learning it's a very inefficient process You'll have like a million tabs of Google open You'll have like a million 10 or windows open running a million different tools and that that's not very That's not very efficient learning process. So this this all started out as a side project More for us to you know, easily to easily learn and then and then ever since and I think that was around Around August September and then the first or at least the first public version released in November December, right then Yeah, yeah, we had a felt like a prototype within sort of three four months. We pushed out and Yeah, it was like this really really weak sort of basic application where you could deploy one machine It was like and I think there was even rooms there We had like a credit-based system and that's when we met dark. Actually, I think you you came in duck We you shot us a message and said hey, this is really cool what you're doing And I think it was just after the credit-based system, right? Mm-hmm I I joined and I could still still see the messages and chat in the juice shop room Well, we had chat still of the credit-based system That's how I knew about it because otherwise I had no idea that we used to have that. Yeah Yeah, the iteration. It's gone through. It's crazy. Yeah I think that's like a great thing as well for which I hack me We're not afraid to constantly change things like make different iterations You know and push out to the community and getting feedback from it. Mm-hmm Did you guys get Did you guys get the feedback in response that you were hoping for or was it a surprise? So I think it was all it was all honestly quite surprising because we never we never intended for To have me to be used by a lot of people It was always just for me and then to like get machines on their hack easily here So when we saw people starting to use it, we're like, oh, wow, this is really cool people Actually when you use try hack me So so it was quite exciting, you know to have built something and to have like people actually using it Yeah, definitely I guess I know Assemblance of that feeling kind of like when you create something and then you see real people using it It's like well, that's a that's a pretty cool the feeling feeling Yeah, you find lots of bugs like that you never thought would even be possible Yeah Yeah calling your website try hack me doesn't help at all Yeah, yeah, actually the community part of it wasn't like it wasn't actually ever sort of never thought about doing a Like having a community aspect to it. It was only until we met dark star John You know, you were like, right, I think this is great Let me take, you know control of the community aspect of it and build that up And I think that's what really sort of gave us the initial You've seen that's when the trouble started. Yeah Exactly Yeah, there was yeah, I think the community's been key in all of this and it was all up to dark or from dark that we Yeah started it really. Oh So the whole user submissions and like community involvement I guess submitting things and adding to the whole environments and all of the rooms that try hack me has that was kind of your Hard charge dark. Is that right? To some extent when I joined and I I put it up on Twitter a while ago like the initial messages I had with Ben that I thought With doing the submission process. I had thought it had been a very community driven thing I guess looking back on it now I and maybe a shoe or Ben can speak to this that it I guess it was originally more driven by just adding these boxes up There and having it up there so that you could teach others with that I thought it was just a hey, you want to throw something up here You go for it and I just kind of went wow and I started throwing a bunch of things up on just Go ahead Well, yeah, yeah So like like dark said like like the feature of like having people upload their own boxes was something that's quite intended But I think dark drove a lot of the a lot of the engagement behind the future and a lot of the processes So stuff like how do we actually get people to upload? How like how do we how do we encourage them? How do we get people to push up specific content? I think the whole process and getting the community to engage with uploading their boxes was all his like his baby through a large part and I guess a lot of that on my end was just working with people and You know having friendly competition with people in the community to create better content that Initially, so I started out just a normal user and I was just uploading content of the site And I wanted to be the number one on the scoreboard and for a little while I was and I still claim that that's my throne as it should be But we had one of the users come along paradox that he's one of our moderators now that I was making content and he would push me to be better on it that he would come in and say Can you clarify on this or he'll come through and start challenging me for getting first place and things like that And he did dock me off the throne but we won't talk about that Nice Yeah, it ended up being a lot of just continual improvement on that end as far as the community goes and Taking what I had learned previously. I before I had gotten involved with try hack me I had done streaming independently Ended up dropping it because I just didn't enjoy it in the long term For what I have been doing it with and taking like the base rule set that I made with that Kind of form the base of like the rules for the community and just built from there So dark you were kind of hey the community guy is a community manager. Is that the right I guess role or title or whatever I have a weird title. I I think I'm technically content director right now and kind of community head. Yeah My yeah, yeah, let's go with that. I don't know. Do you guys want to clarify? It's kind of nebulous most of the time Yeah, yeah, I know I think I think that's perfect You I guess you also tend to wear a lot of hats So it's I think community depth of the I think content also very strong and then yes Yeah, you help us a lot here in there. So It's like a very overarching role. Yeah, most of the time the contents I guess it's split play you're pretty evenly between us I'd say as you you do a lot of the content as well, especially with our creators program Directing people and helping us keep the vision there of you know Maybe we don't have content on a specific topic How can we push that forward and push for change in that positive growth? I think that works out really nicely because it really is shared between all of us So a shoe and then what do you guys kind of consider your role or what do you actively work on? What are you working on now, etc? So I think Ben does a lot of the dev work and I and I do a lot of the I guess the business side of things But it's quite cool because because of this complimentary set of skills. We tend to do a little bit of everything So so I'll picked up like an insane amount of like dev skills from Ben He's like he's like God it through a lot of it and She all the business side of things and sort of yeah dark sand the vision has come from a shoe Absolutely, so it's quite cool that like we get to work on a lot together Which is like which is like the fun part of like try hack me is that we get to learn about so many things that are outside are outside our like skills of interest or talent And I as to what we're currently working on I guess they have been jumping here in case I miss anything, but one of like I guess the major thing is always is Scalability, so I'm sure you've noticed like or you've even seen that like through discord We get a lot of comments and a lot of feedback from people So we just want to make sure that their experience like with stuff like performance is Best so we I think the past few weeks is the very Fixing performance and scalability issues right then. Yeah. Yeah, it's something that I'd not had Thought I could have come around so quickly it was one of your videos John that thought of when you had a lot of popularity And we just saw things break and we're like wait what?  It's just skyrocketed into the number one video Yes, that was a huge driving force so, you know, thanks for that was Great video. But yeah, as you said scalability sort of, you know, better in the user experience Like number one is like the great thing was our community like speaking to them and seeing what you know Can be improved and made better on the platform and I think that's primarily the focus right now cool cool when you guys were I guess getting started what sort of Rooms or activities or what things that you try to put in place to like really make them I know the baseline of the platform to encourage other people or just like here. This is our first impression for the world What did you put in there and why? So I think if if not One of the first rooms, I think one of like the earliest in rooms or the OAS to shop Because because we know that like at that time we were like interns and learning about security stuff And that's like and and to shop is like the holy grail of web security So we were like oh One if you had to set this up on your machine It's like it's just time-consuming and two is that people whenever people are doing to shop There's no like guidance. So we want to show people that look you can actually Come access a machine with a click of the button and actually learn how to do it so I think you shop was one of like the first ones and then I I can't I can't seem to remember Basic pen testing that it was blue and that was the very first room dark did basic pen testing in the OAS to shop I remember the OAS to shop the guy who made the VM The OAS to shop came on and gave us feedback. Yeah, I just remember him saying like how cool it was That's awesome I Just shovel where that I put out that so originally I started with blue and then I started making the Red primer series and the the primer series on the whole that I had the idea that you know what? What is gonna be a good core and it's gonna be you know covering the basics is Everyone has to learn the basics. Everyone has to learn how to use something like metasploit or end map You know, where do you what do you use if you don't have end map gonna be up a creek for a little while, but When did you guys start to see the platform kind of like us take off or like when did a ton of users? Okay, start to work through rooms start to create rooms. When did it just skyrocket for you guys? Do you feel like it has at this point? Yeah, so I think I think the past that well the early the latest growth spurt has been was December 2019 So that was when we first challenges the advent of cyber challenges So that was that was 30 days of very intense box making and we had and like yeah, since then it's been it's been really really crazy I remember the day we released the first Christmas challenge and we already had like two thousand people on the page at like 8 p.m. Oh wow Yeah, and we were deploying a challenge and Ben and I were like live and we're like, oh my god This is so stressful and we're on fire. I think ever since then yeah ever since then it's been it's been going quite well cool 2019 December 2019 right so like just about the start of this year almost right Yeah, yeah, and I think before that we were we were focusing a lot on students So we ran we ran a university CTF challenge just more specifically in the UK So we had that in I think in I think twice across that year once in October and once in March And that and that did like quite well right then I think you can yeah more to that No, no those went really well They haven't I think the October hack back to our university event had 27 unis And everyone competing to get to the top that was that was really good. Yeah, great fun. That's awesome Can you tell me a little bit or I mean whatever you're totally comfortable and cool with sharing I don't want you to give away like any like trade secrets or stuff behind the curtain But what what can you share? I guess a little bit about the infrastructure What kind of the tech is behind it to make the whole platform work between spawning virtual machines in the cloud But behind bringing machine up and down kind of a whim what what tech is behind that if I may if I may so a lot of the Infrastructure is hosted using it's all using cloud infrastructure I think that's been a huge advantage not having to maintain and sort of manage our own servers That's been I think one of the like a key thing You know to the success of all what it is at the moment a small success is I'll try hack me at the moment You being able to you know have these base images that you can just sort of spin up and terminate instantly It's been it's been great that there's so many sort of cogs to make this whole thing work But a lot of it has been configured in the cloud As for our application it uses a variety of different uses the mean stack without the angular and Yeah Sweet is that kind of the You said mean stack is that right with you said without angular? So is it a lot of JavaScript or what kind of languages do you typically work in for when you're building for when you're creating? Yeah, good question. So it's primarily no JS and Mongo and express and there's we have other sort of sort of scripts involved Yeah, as a small piece in the whole puzzle to make the platform function But the majority of our applications been programmed in no JS. That's cool But is that kind of your your weapon of choice? Is that like your strongest sword because I am so bad with JavaScript and no That's sort of it was like I'm still at university and it was like one of the units covered a couple of years ago And I remember making like the proof of concept was I was like, okay I had experience with it like a couple months back and sort of went from there and it works So we were like cool when node.js is up and coming and sort of becoming more and more popular And so we just kind of ran with it Sweet yeah, that's probably a really good one to choose considering I mean like you said It's just becoming like the number one thing. Yeah, like the industry standard Yeah, yeah, super popular and the amount of libraries in the community and sort of the feedback around no JS is it's great as well because now having a stack overflow page that you've got a horrible problem. It's really helpful nice nice How much of the how much of the build out is automated? I mean between like you said between working in the cloud And I'm sure you've got obviously your node.js and stuff in there How is it deployment completely automated or are you still clicking around in whatever GUI for? AWS or is your digital whatever whatever it's in Yeah, yeah, so are all our all our deployments the cloud are very API driven nice So they're yeah, so which makes them like almost fully automated apart from the Debugging here and there which makes it which makes it quite easy to build into a web application Part of me is thinking about how that API stuff goes Because I when I think of okay spinning up some my cloud infrastructure I'm using cheesy things like terraform or decorating things with Ansible and vagrant or What what does that look like on your side or is that just hey strictly restful API calls? Yeah, yes as restful API calls So when we initially started we initially saw when we actually heard about using terraform But I'm terraform was mostly just for orchestration of infrastructure as I think you use you So it didn't really fit our use case and because a lot of our vulnerable machines tend to be base images, right? The only process for deployment is that it's just using a base image Which is why you don't need to like labels on these images or you don't need to have these orchestration tools on there. I Guess that's a good point, right? Hmm. I think that we've been experimenting with Sort of more like the minimal viable product we've got and sort of you know seen that it's really great But like things like having networks which are coming up very shortly Sort of deployed and configured in the cloud. It's a little bit more complicated So as we see, you know, sort of the future of try hack me that the extra functionality applied there It's going to be a little bit more complicated than just you know one API call to know it's like deploy a machine That's really cool. I'm sure a ton of people are super excited for the for the networking capability How would that kind of look if I may ask right even on even on the front end? Would would you okay let kind of the player a designer like draw a little network map and put machines here and there? Or how do you envision that all coming together if again, not I don't want any trade secrets, but whatever you're cool No, no, that is good. The fantastic thing is just give you a bit background like that My my dissertation I worked on the network So I had a lot of time to be able to research and I could fit uni and try hack me in together That's awesome So we've always built up like try hack me to be as flexible as possible So a shoe mentioned that it's API driven not just the calls to our cloud infrastructure, but the platform itself You know in the whole room sort of model that we've got being able to clone repurpose and you know Enter in different rooms again flexibility and I wanted to apply the same stuff to network So that networks can be used within cough that can be used as I like a more CTF based Idea that could be implemented as its own course. So how it's designed at the moment is you can physically Have your network you can upload it using the same sort of Principles that is done now you can give it some extra Details and exactly like you said it just sort of creates the network for you And you can deploy once it's uploaded and you've added in the extra extra details You'll be able to just deploy your own network and tear it down and bring it back up relatively You know simply simply so so yeah, it's yeah, it's good. I'm excited for it I've spent a while making it all sort of functional and yeah, I think it's gonna be great I think it's gonna add a lot more Teaching a lot more for security concepts through networks gonna be great That sounds super cool And I mean I love the faith I love hearing that you're excited about it like hey That's gonna be a really great win for the community and all the people that want to create and build and play you in that And I know the same I may remember you telling me about how excited you were about the king of the hill Functionality and that has been incredible, right? Well, how was king of the hill? What building that making that the reception? What what can you what can you help me understand in that whole new release on the king of the hill game type? So I think I think it initially sorted out as so Ben and I are just constantly throwing ideas at each other And one of these ideas was um, it's just like messages at like any time of the day Which is quite cool because we just we tend to get like very overexcited Just was I just message Ben and I was like oh We don't actually have any anything like collaborative on the platform because it tends to be someone just enters a room And then and then Ben comes back saying that oh like I would make it even more fun Is if people collaborate, but there but it has a competitive element to it and and then that sort of Broad about this whole idea of how do you actually develop something that's fun to play? You know you get to work with other people but at the same time that the element of competition involved and then I think I Think Ben Ben mentioned earlier that or when we were talking that someone in the community was just talking about how like these competitions are done Right then I think mm-hmm. Yep. I'm gonna say on that probably Yeah, yeah, so I had the initial idea for like a competitive hacking game And yeah, we spoke about it in the community and that again keep out of speaking to people and never been there in the chat Is that someone else said coming but it was it's awful I think it was definitely one of the key members of it in the community said like hey, we do this at school You know we've done this at college and this is how it's worked And that was me that sent you the documentation because I do red team competing against Blood teams that are given a bunch of boxes to defend and it's eight hours of them getting mentally abused by us I Yes, so yeah That's sort of the initial idea of like speaking to people get getting that idea And and then we just kind of ran with it and we did a trial when it went down really well and off the sort of doing the tweaks Releasing it was it's fantastic. We have a lot more ideas because it's just finding the time to implement it How do you guys plan out? I guess I guess that the box or the whole vulnerable room that they kind of move in between Keeping it okay an easy difficulty so that anyone can work with it Or a harder more difficult box what what sort of mental mind mapping does that take I can probably speak to this one For King of the Hill, it's a little bit different General box Mentality I actually just gave a talk about this at Sarkon nice where you have an idea of you can you have a kill chain that It's the same as like a kill chain on an assessment that you can work backwards from it And you almost make yourself a map of how you want to have things planned out where you look at major points where you have your entry point you have your privilege escalation what enumeration you want your users to be able to do and Depending on the level of extra just things you throw in there like extra ports that are open extra programs that are installed And running on the computer, you know that it escalates your difficulty and you can tweak it from there I King of the Hill I know Ben and issue can probably speak to this quite a bit But you take that and you essentially spread it out laterally So you have four or five different kill chains per box that each users a little bit different that you have to get up to that room level One of the main ideas was that we wanted this defending element for King of the Hill Right, it or like it wasn't just hacking into a box That's how do you stop other people hacking and and and part of that part of that defending aspect was we need to have multiple ways In because if you just have like an SSH connection or an RDP connection Someone could change a password to that service or turn that service off right which wouldn't make it fun So so the idea was that if they're like multiple services or multiple programs opening when someone becomes king or when someone roots the box They'll be like oh my god I have I have like five six different things I need to turn off while other people are hacking into those five six different things So that part is what we thought would be quite fun for people taking part in King of the Hill Yeah, we issue and I when I we did we met at an internship in London and there was a day We both pwned one of the boxes on another platform and we're both kicking each other off trying to defend it So it goes back as early as that. I was like, why did my connection die lug back on us like a shoe That's awesome has the gameplay kind of been what you expected it to be between okay It's both the red team adversarial stuff that people are doing to break in But even the blue team stuff because I always feel like when I'm playing King of the Hill like man I I stink at getting the defensive stuff put in place, but it's not like anyone can bring Oh, here's an here's an elk stack. Here's my bro logs It's it's different and then it's a okay quick one-hour game And you've got a little poor man's environment to try and patch and keep secure Yeah, so I think Like so far in track me. We actually don't have a lot of blue team content Yeah, I think a majority of the blue team content other is Jones But so so we've we we've actually explored the idea of having like you know having like a sack running or having steam solutions running where people could Look at logs that that would be quite cool But so that that's something there's definitely like room for in the future, but in terms of the gameplay I think it's been fantastic. We like We we release co-op and like Ben said we love to experiment things trial things just push things out all the time And the reception has been like quite fantastic There's tons of games running every day and we've been getting a lot of feedback that have been enjoying cough What was the game? I'm sorry has almost been like a The meta has almost evolved like an arms race for the red team side of things where you see people starting off with you know Of course people wanted to use things like auto phone scripts because there was a small pool of boxes initially That's not something that we allow at this point But you know watching this arms race evolve that we can go through and react to it and say that Okay, maybe we don't want to allow some of these things But it forces people to get better and faster at They're I guess hacking in that aspect and we can go through and actually develop the blue side or blue team Set of things from that where we can see you know What are we missing with content that we need to add at the site or specifically? You know what needs to be there in the box for the blue team to effectively be able to defend themselves Blue team is really hard man. It's really hard to get blue team stuff in It is harder than red in real life We've seen that over and over again that I mean historically blue team gets its butt kicked by red most of the time Which is I mean it's something that industry on the whole has been facing adjusting to having blue team be a little bit More balanced against red team You can't just use the chatter binary to lock one of your password files Especially if you like you you put a little time bomb in the chatter binary that blows up your shell like some of the box Yeah, people have been really creative with how they've made their cough boxes or even like you know The players have been able to upload their own binaries and have their own tricks. It's really great to see. That's cool What was the what was the driving force and kind of making coughs free to play because I remember hey at the beginning This is just hey subscribers only let's kind of test the water here Well, did you always want it to be free or was that I like hey, this is a huge success Let's give this to the world. What were your thoughts with hold that that's all strategy I think I guess then like jump in here in case I just don't remember correctly But like whenever we went when we did release call We I think we were thinking about monetizing it But then because we tend to run things as experiments. We were like, oh, you know, well, we'll run it for free So we had this one week experiment where we were like, okay Let's leave call open for free and let's see how it goes and we got like a crazy amount of optic in terms of the Active users playing call that week. So we were like, oh look people are enjoying it, right? So like why should we lock it down or why should we lock down the base model where people can go back against their friends That's really cool the the conversation of Monetization and funding is always like a hard one to have but it's really interesting from a the creative standpoint of like what what should we Gatekeep is not the right word, but I guess that's what I come to think Because like oh look, we're all want to do this right we all want to train We all want to like teach people we all want to learn and exercise and sharpen our cyber security skills So it's it's weird when you want to make that a endeavor and money For yourself, but I think you guys have just hit the hell hit the nail on the head like to try hack me It's really simple and what it asks for and what it doesn't Yeah, it's like Monetization has always been like like like you said a very very tough conversation But but like it's quite cool that having like the two or three of us like on try Hack me means that we get to make like our own decisions, right? Or we get we get to like sort of do whatever we want and the fact that you know We can make like certain rooms free for a week or we can make all three but like that freedom is quite cool quite exciting So I never asked and I wanted to I've been myself up because I forgot to but this is this is it right?  Is there anyone else that is helping out or do you is it? What is it?  It's just us. I'm hoping to bring somebody else on We're just gonna write up a job advertisement to help out with the development and sort of grow from there. I Mean wow, right? That's pretty awesome and kind of incredible and I'm sure it's got to be a good feeling to be at the spot Where you can put out some I guess some job advertisement because that's got to feel really real Yeah, yeah, it does and it comes back to the you know the awkward conversation of monetization You know having a freemium model it allows people to come and learn for free There's so much free content on the platform as well as description only so making I think yeah making use of that Great Cool With that with the community tie-in that you mentioned right so I know you guys have the discord server And I kind of want to zoom in on that a little bit I know that hey That's for the community that cultivates the whole environment and I do you use that to get some feedback How do you enjoy that platform? Is that the right platform for you? Does does that do everything that you need to or what other forms of feedback do you get and how etc? Um So yeah, I think Like so so John drool was like I think the biggest driving force behind the discord I think Ben and I was always it was always Like our thought process was always like oh we want to talk to users like that's how we want to build out trihack mean I think before that it was it was the very standard methods of surveys emails phone calls where we could that It's actually how like we got dark involved We were like we saw him as like sort of an active user and we're like aren't you mind hopping on for a call for like 20 minutes And you know now we're here So I think discord has been insane in terms of like how we communicate with our users because it's not just Getting opinions on on features and boxes really but we also have people filing bug reports Filing feedback through discord, which is I think really really cool Sweet Yeah, I mean I have my cheesy discord server and I just love it What Can I ask I mean looking forward what other things do you plan on bringing in to try hack me? So I know you mentioned okay. We've got the networking capability coming soon I know you want to pour more into cough There's one thing that I saw and it was a community question as well when I was kind of poking around asking people What is that university coming soon tab? That's been for a while. Yeah Create there's so many things have happened. That was put on hold. It's we want to have You know be able to have universities Sign up and sort of have like a separate leaderboard so that students can compete against each other We work with one university in the UK and it was requested that we had a like a University only leaderboard so they could you know sort of hack against their friends and see who's the best and I think like this whole gamification and activity type, you know Sort of model works really well. So having a sort of all your friends at uni sort of to compete against. I think it's gonna help Yeah, help them sort of you know move forward in terms of their learning That's really cool. I certainly would have loved that when I was at university It would have had a lot more fun I Know as the oh, sorry No, no, sorry, and now I'm just gonna say what I guess one of the other things we're trying to work on is We're just trying to like improve user experience heavily because we know that you know Like one part of learning is having content available and having this whole platform and infrastructure available But we're trying to see we're trying to build most instances into try hacking I would sort of make it easier for people to learn Because I think and I think this this might purely be like a user experience thing But like we we want to look at more ways that try hacking can actually aid in learning And that's that's been quite a quite difficult Conversation or just difficult set of questions about to ask ourselves because it's so broad But something I guess that could have a very big impact One thing I know I'm looking forward to And this is new for me because Obviously, we I was really happy and had a lot of fun putting out P kill my my new room that I sent over to you guys But I felt so bad just nagging you Ben Like over and over again. I'm so sorry. I need to push a patch. I Rather repatch You know done the intended way so never worries I think as you said that things like that like extra features to be able to just you know Update your machine that's you know, you've deployed and sort of replace your image with it I think it's gonna be great and really helpful. Oh, yeah, absolutely I think from from the creator perspective because I just feel bad pinning you No, not at all. No, no, it's it's something that's Requested heavily and people I think it's really interesting to see the unintended ways, you know to talk when you're pulling the machine It's yeah, I think it's I think it's really cool So being able to patch it straight away. Thanks for getting on that. You seem to be hot on Oh, it's funny. Oh Lost him for a moment. Come back This is awesome guys very long. No, I'm sure I'll catch it real soon Thank you guys again and again for doing this. This is this is super cool having us. Yeah, thanks for having us I Guess as far as other content goes I know it's been brought up a couple times in a shoot you mentioned this but expanding blue team content It's another big thing that we're looking at expanding that defender aspect just because it's it's something that it's very There's a lot of widely taught, you know, this is how you can break into things There's a lot of you know a fair number of platforms that are teaching that right now But having that increased coverage on blue team I'm looking forward to that Ben's back. Hey, sorry about that. What did I miss? Dark was just filling in for he's like, oh, I guess I get it. I better take the mic Yeah, I'm quite excited for the for all the blue team stuff because it's basically us She's going to be like dark. Can you please like look through all this or dark? Can you can you just like look through this million page doc and like what blue team stuff we can get out there? But yeah, it's gonna be a really weird conversation of yeah So I can I stand up a full sim environment on like these boxes and I just need you to give it like I don't know. How's it?  Yeah, sweet All right, cool. I know what we're kind of get. Oh, sorry Oh, no, it's in terms of other things that we're looking into think Yeah, I don't know if it was said or not, but the more learning pathways for sure and continuing the sort of pushing out more More content. So I feel bad because I feel like I've neglected like personally I haven't looked too much into the whole learning path Feature and functionality has that had a lot of success. Do you feel like that is good at where it's at? Does that get the reception you want? Mm-hmm. Yeah, so so so we've seen that a lot of people on try hat me like using the learning path. So a lot of the Yeah, so I'm trying to think what the breakdown is But I think all of them are are very are very well received because the main point of going to learning path is because The rooms are so modular It's so easy to slot them in different ways and a learning path for structure of going into a particular room And I think I think people like that structure, especially when they're learning content But that they've probably not seen before so learning paths have been quite popular.  We're not gonna give you any guidance Just here's an IP address go and some people are used to that and really really like that But I mean as you said for the beginners the newcomers the people that are just like hey I'm getting my feet wet and I don't know what to do That's why I always kind of tout try hack me is like they'll literally hold your hand and walk you through it And they they're upfront with the write-ups Yeah, yeah, yeah, the idea has always been to sort of to make the learning more efficient and and Yeah, so some people do like the IP address structure But we've seen that you know putting the content out there and that being handled is quite like it's like Efficient for learning and then what what we tend to see is people people who go through a bunch of walk-throughs Then move on to challenges once their skill levels have been raised So the whole walk through this is challenging fits in quite well and having that flexibility on the rooms on try hack me He's worked has worked quite well Super cool I'm sure the learning path is just like mag like multiply that because now you've got hey Here's a collection of those guided rooms to learn and you don't accidentally hop into a little challenge Cool. We're almost at like a little 45 minute mark I don't want to be holding too much of your guys time Is there anything else that like we haven't discussed or we haven't brought up that you really want to bring some spotlight to Or you want to chat about? If not, that's totally cool I guess I guess just just just one thing to mention is that it's probably off late But for going back to when we picked up from act from like really massive growth is that we've had a really good team of content creators come on and so Yeah, John Ben and I just want to look at the submissions So just want to give want to give them like a massive shout out so people from the community and other people developing content Because I think it's quite cool that or at least the way we do content creation is quite interesting because it's not just Box submissions and rejections It's sort of people submit boxes and then it's probably a lot of it is dark guiding them throughout to build better box Is how to also part of that is how to write better technical content And then the end result is having really good content from out which our users appreciate So I think massive massive shout out to people that have been pushing out content off late Yeah, I have a firm belief that any submission and this goes for anyone that wants to submit content on that It's okay to fail any submission you have a goal there And we can get it up to that the standard that we look for and you can learn from that You know, I've mentioned this. I think the other day that you know submitting a box Making something like that it is, you know, it's a resume out of it It's something that helps build you up because you're learning a lot And even then as you were seeing John that finding those unintentional paths You learn a lot from just having something you've made get beat on Absolutely I have to reiterate what a shoe was saying massive shout out to everyone in the community that does content Devon our community staff members as well I love I love the way just you guys handle it because I mean it's super surreal seeing and talking with the three of you Because we know like hey, man, that's that's the core. That's where it's at But all of you have just been super supportive and always approachable It sounds like that's what I've always heard when chatting with the community like We love hanging out with the trihack me guys because they're nice and friendly and they talk to us I don't know. I'd like to think we all bite just a little bit. Oh, yeah That's true in coffee at least we'll bite it when we're playing the game Sweet So my last question that I like to end on for some kind of stuff like this is um Obviously we all want to do this sort of thing, right? We love cyber secures our passion We love teaching we love learning. We love sharing the knowledge. So if someone Wanted to put together their Like their their dog and pony show their trihack me their form of teaching people and helping the community grow What is your best advice to that person? What do they need to know what languages what program what techs hard skills and soft skills What do you tell someone that wants to do something awesome like this? I think Try to figure out how you learn how well how you feel that you learned information security How did you what is effective for you and work off of there and ben an issue can expand on that Yeah, I think I think more on to on to building building like a product or platform, right like trihack me I think something that's worked really really Well for me and ben is like being like these are driven. So I it sounds super cliche, but I think A while back we started feeding in like data points into term in terms of like how users use the platform What do they engage with best and having something to refer back to like having a tangible thing to see Is really really useful because that like sort of helps you guide like your vision for the product, right? Like you can see all users going to this room or users using this feature How many users like it and like and you can use that to basically improve your product And that's something we we've learned off of like I guess the past year Yeah, user engagement. I think for me, I think it's huge. I think you know the way you deliver content the way that you know If it's a video that how you speak if it's the type of hands-on material that you're providing user engagement I think I think it's really key and like if you want to like build something like try hack me or your own version I think having you know, it doesn't have to be really fleshed out You can sort of learn, you know, node.js or you know, python You know different frameworks in order to like make and push something out As long as you know, even if it's just one machine and you're you know, just sort of, you know Using that as your you know minimal viable products. I think that's great But for me, it's you know to answer that question. It's definitely user engagement Awesome. I love that answer. It's all it's all about the people. It's all about the people who use it Yeah, exactly. Yeah, super cool Okay, okay. Well, hey, that's everything that at least I had in my mind that I wanted to chat with you guys about Um, I don't think I have to tell you. I'm obviously a humongous fan Uh, you guys have been doing phenomenal work and you keep killing it and I'm happy to It's grateful for the opportunity to hang out with you guys and spend some time together So this has been a blast. Thank you. Thank you. Thank you. Yeah. Thanks having us. Yeah. Thanks having us. Thank you


All right. I guess we'll get started on time. So we're here to talk about Edge Kubernetes OpenStack and LovNFE. So I think that wins the buzzword bingo for the day. So I'm Randy Levensoller, architect at Cable Labs, working on SDN and NFV. Hi, I'm Shamik. I'm from Aresint. I lead the Aresint and in general the edge computing strategy as well as our solutions based on open source and virtualization. I'm based out of India. Yep, and we work closely on some open source projects together with some installers and we'll talk more about that in the presentation. So just want to start with setting some context for where we're at today and where we see kind of the service provider industry going in the future. I mean, if you're not familiar with Cable Labs, we're a non-profit R&D organization that does research on behalf of the worldwide cable industry. So kind of we see today and some of our members are fairly far along within this lift and shift, which is where you think about OpenStack, a lot of the first generation network functions. They look a lot like the physical devices. That's one of the sales pitches, right? My VNF does everything my physical device does, which is great if you don't want to take advantage of the cloud. We're starting to see some of these infrastructures as a service, platform as a service, network as a service as this evolves, and some level of automation. We're getting a lot of learning through this phase. Next phase, which is kind of where we're at in the next one to three years and when I say where we're at in that timeframe, that's looking at deployments for the network architecture, a lot of the technologies available today. We are seeing some proof of concepts for cloud native and even some of the more advanced features, but really not seeing the wide adoption yet. And so we're seeing kind of the next phase, containers, cloud native, and actually seeing smaller VNFs, not quite microservices, but moving towards that direction so that we're actually able to take advantage of the cloud infrastructure or Kubernetes. And we'll talk a lot more about kind of where we're at with Kubernetes. And then you're kind of going on into the future, right? Once we have everything automated, we have microservices and moving on to mandoservices to show up throughout another buzzword. We really want to see autonomous networks, self-healing networks. We'll really be able to collect a lot more data, have more learning from that data, and then be able to do the self-optimization, self-healing networks. Which you can do smaller portions today for where you actually have that end-to-end automation. But really kind of once you get everything across your network supports, programmable switches, open flow or P4, some kind of SDN controller managing the switches, that's really where we're kind of excited and seeing things going in the long term. And again, I promise to talk Edge. Of course, what does Edge mean, right? I think everyone has a different definition. So for us, Edge means everything from the customer premise. So this could be your cable modem for a residential customer or a stack or set of servers if you're more of an industrial user. Access nodes, so these are like, they're the little pedestals or things on telephone poles. You can put compute there or kind of your Edge, which is a head end or hub for cable, central office in the telecommunication space. This is where you have some air conditioning. You can actually use traditional racks. So you get a little bigger, a little more oomph there, but still space is limited than kind of core and regional data centers and hyperscale. And typically, we'll move to the next one. Talk about why do you want to look at these different areas. The more centralized you are, the less expensive it is to run. But you also have more latency. If you look at trying to render things for VR goggles if you host it and it's 50 nanoseconds away, you're going to get queasy. So you want something that's really within 40 kilometers or less, very close to your house for some of these low latency applications or high bandwidth applications. You really don't want to have that bottleneck anywhere on the network. And so that's what we're looking at when we're looking at these Edge applications where move it as close as you need to for that application. And HA comes into this as well where you may run more stuff on the customer premise just to have everything within that availability zone. And some kind of rough level things about why would you look at placing an application in these different areas. So again, some use cases, I'm not going to go through this too much. But basically, we can do more data plane stuff closer to the Edge. Some of the access networks, 5G is one of the things everyone loves to talk about. We're not going to talk about that too much today. But that's something that you can run in the Edge when we talk about the infrastructure, but a lot of the next generation cable infrastructure can also be run in a virtual environment at the Edge. More central things, you have your mobile core, your voice. And then lots of things like Netflix and all can be streamed centrally. And then you have your CDN closer to the customer site just to kind of give us some little more context. And we have more and more requirements as we move for these network virtualization, low latency, high throughput. We care about how many packets per second can we get through. How close are we to line rate? If you move things from physical hardware to virtual hardware, you don't want that to be a slow down, right? If you're playing game, doing your gaming at home, you don't want to add a half millisecond to your ping because, hey, we virtualized. You want to have that same speed if not a better scenario by having a smarter, more intelligent network. So we really do focus on a lot of this. And some of it may mean we're doing less oversubscribing of compute nodes and things like that so you don't necessarily get the same VM density at the edge for these applications as you would in a traditional cloud. So again, probably won't spend too much because everyone here knows KVM. But hypervisors and kind of open stack based virtualization is really good for general purpose compute, especially if you have a large data center, gives you a lot of flexibility, a lot of security, a lot of isolation. It's a lot more mature than things like Kubernetes. But there's a lot of overhead that comes with that. It takes a long time to spin up a VM as compared to a container. It just, a lot of extra management. And if you're managing thousands of sites, when we're talking about the edge, we're talking about thousands of sites at the access edge and then millions of devices or millions of individual sites when we talk about the customer premise. So we can't do things that need any active management. All needs to be passive. You need to be able to roll it out and have it all automatically managed as much as you can. Again, open stack. We're at open stack, so I really don't need to cover the slide. Sorry. I left this in from another presentation. Let me hand it over to Shamik now. Yeah, hi. So I will be speaking a bit about the edge computing in general and how this particular market is picking up and what kind of problems we are trying to solve in edge computing, particularly with respect to the open source project that we have recently started. So as you can see in edge computing, there are many domains. There are many verticals where edge computing makes sense, primarily not just in communications, in consumer, in rail infrastructure, and energy as well. And if you look at this today, one of the fundamental problems in edge computing today is to find the right use case for edge. I mean, if we can save 10 milliseconds or 20 milliseconds out of public cloud to edge or from a central office to edge, then does it really the right thing forward? Because edge is expensive. There is a lot of hardware to be run there. It is highly distributed. You will have thousands of edges. So is it prudent enough to run everything in edge? Or what exactly we should be running in edge is perhaps one of the key problems today that we are trying to solve. It's also important, perhaps, that we also need to look at edge computing from a developer's perspective. So application developers who will develop these applications like AR applications or applications for cloud gaming, which essentially needs to perhaps run on edge for latency purposes. They all come from a completely different world as compared to the telecom or the communication providers. So getting the edge more developer friendly, to get edge more accessible to developers so that they can seamlessly transition from maybe public cloud to edge, or, for example, they can start their own edges to try out a few things. So that's one of our critical foundation for this project that we are doing, where developer is a first class citizen in edge computing. Let's move. So when we started this particular journey, we also wanted to understand which are the applications that make sense for edge. I mean, as I mentioned, not everything will be requiring an edge. Some would perhaps can run on public cloud, or it can run far away. Maybe it's the latency tolerance is high. But then what exactly should run on edge? So based on a few observations, we came up with this kind of a list that, of course, the latency is important, otherwise most probably don't need an edge. The other, perhaps, important aspect is the power. So today's AR glasses are large and bulky, and it gets heated up pretty soon. And its prices are pretty high, right? $3,000 or $4,000. Now if you have to make the AR glasses more accessible for users, then to bring down the price of those devices. And if you have to bring down the price of the devices, then you might have to offload the computing from the device to edge. So there must be some ways, some innovative ways of offloading computing from device to edge. What kind of battery consumption and application requires today on the device, and how you can actually move that consumption away from the device are some of the key challenges that we have to solve. It also means that you would be doing a lot of computing on the edge. You would be running, perhaps, neural networks. You would have to run a lot of data-related algorithms. So the edge also needs to have an architecture which can support data designs as well. So it has to support, may not be big data. I mean, that's a misnomer, but perhaps you need to support streaming analytics. You need to support a lot of capabilities which today can run on a device. But once you move it to the edge, you can do much more than one device. So today, a device can render one particular application, the same application when you're running intents and thousands of them on the edge, then there must be different architectures to handle that. So this was a kind of an exercise that we did for some of the applications. Not all applications really require edge. But what this particular set, I'm not going to go through the list, but what this actually means is that why we perhaps need a microservice-based approach towards edge compute is the simple reason that a developer today's developer, an application developer, doesn't build everything from scratch. So they use a lot of open source. They go to a marketplace to get some microservices to build their applications. Those microservices could be built by somebody else. For example, if I'm doing a face recognition application, a very specific algorithm could be built by somebody who has nothing to do with edge as such, but built a face recognition algorithm. And as an edge application developer, I might want to use that. Now, can I use it in the same structure, in the same format that is running in a telco edge? Most likely, no, unless there is a way to consume APIs. There is a way to launch those applications and microservices together. So that's the critical factor. Nobody today develops everything from scratch. How do we get microservices which are not exactly meant for edge to work on edge? Can there be a standard method for the infrastructure? Can we do serverless? Can we do container-based architecture so that these microservices can be consumed by applications? It's something critical for our journey as well. Apart from this, there are some other fundamental problems on edge compute when it comes purely from the developer's perspective. Now, today's developer might be running 10, 20 virtual machines or 50 virtual machines in a public cloud, and is able to serve, well, hundreds and thousands of consumers. So how did this happen? It happened primarily because of two things. So you could store the server side of the application. You run it on public cloud. You can disseminate the client side of the application through app stores. So basically, you can reach out to all consumers today. Now, if you expect this application to be just lifted and executed on edge, most probably it won't be the right economic model. Simply because 10 virtual machines today would become 10,000 if you run all those virtual machines across all the 100 edges or 1,000 edges. So the developer's cost per application increases drastically if he has to move towards edge because he cannot possibly lift and shift it. So there must be some ways of re-architecting the application so that the overall cost of moving applications from cloud to edge reduces. So that means that there must be a way to run applications which can be started on demand, which can be switched off on demand. It could be a microservice running on the edge, which is corresponding to one device.  But it cannot be a simple lift and shift because it's simply economically not viable. You cannot possibly ask a developer who is running 10 virtual machines today to run 10,000 for edge. I mean, it simply doesn't work that way. The other big challenge for a developer who is developing an application is that, how do they actually access the edge? I mean, they can't, right? It's somewhere deep inside the operator's network.  It's on the tower somewhere. So today, the application developers have no way of accessing that. What they can do in public cloud is that they can just go log into the virtual machine. They can check what happened there. They can find monetization data, how much money they made out of the applications. But they would not be able to do that when you run this deep inside the edge in the deep inside the service provider network. You cannot possibly expect the application developer sitting somewhere in his home to connect to an edge running somewhere deep inside in the countryside and somewhere in the city and is able to monetize or monitor his application's behavior. So there must be a method to bring all these data together and aggregate the data and able to show the developer some view of how his application is performing. Remember, it's an application that's on steroids. Like, it was running 10 instances in a public cloud. Now it's running maybe 20,000 microservices across different edges. So the kind of aggregation required for an application is simply too high. So this is another problem that perhaps down the line we have to solve it. Right. So what is our approach? We are not trying to solve everything in the first go. We are going step by step. So this particular project actually is trying to solve one thing. That's creating an environment for the developer to develop such applications. And what's a model is that we have the Edge provisioning platform, which is what we call as the Edge provisioning automation. And then there is a runtime stack software, which is based on OpenStack and Kubernetes. So it's an automation platform which helps the developer to build their own Edge and try out applications without having to worry about the access. So this access could be anywhere. And the access location could be anywhere. But the developer still gets an easy way to run his application in the runtime. What we are trying to do now is to find ecosystems and partners and figure out use case development. Because each use case will bring certain change to the stack itself. I mean, if you run an AR application, there would be a specific need of running certain components in the stack itself. And if you run a robotics application, there will be something like a point cloud running somewhere. So you need specific customization of the stack in itself based on use cases. So that's another aspect that we are looking at the community to build different use cases to figure out what's the right stack for each of these use cases. And finally, of course, we are looking at the operator community to try this out and give us feedback in terms of what kind of capabilities are required, what kind of proof of concepts we can do with this particular stack, and using the community's applications. And that's what we are trying to build the way forward. So this is our building blocks. There are three tiers here. We haven't built everything. So we have started to build it. So on the customer premises or on the edge locations, what we have are edge hardware. It also has a platform, open source-based platform based on OpenStack and Kubernetes. This is what we call SNAPs. Runtime platform, which is running on the edge hardware itself. On the second layer, the management layer, we are trying to build two things. One is an operator device management and provisioning, which is basically an automation platform where you can spin up a customized edge stack, whether it's a Kubernetes OpenStack or Kubernetes with certain layers specific to a use case.  And you define the stack, and that stack can be provisioned on an edge hardware. So that's the work that we are currently doing. That's called the SNAPsX on the management side. What we are trying to build on top of that is an orchestration system, which is totally distributed. This is going to be in a roadmap. We'll start working on it very soon, where the orchestration within an edge is contained, self-contained. And then there is a central model which is going to monitor a very few things. For example, get monitoring and diagnostics data from different edges. Now, why we are doing this is primarily because the orchestration has to be decentralized. You cannot possibly run everything centrally anymore because there'll be thousands of edges. You cannot run an orchestration platform in the central office and expect them to manage all edges. So we'll have to split the orchestration capabilities in such a way that there is an edge-specific agent and then there is a central management. It sort of talks to each other minimally and ensures the authentication of the application, the user, the consumer. Remember, there are many stakeholders there. There is a developer whose application must be authenticated. There is a user who must be authenticated. The edge itself needs to be authenticated. So all the decentralized way of authentication is very important. And that would be our next big challenge to solve. Maybe we'll pick up some of the existing open source projects and try to use it for solving this purpose. But that's in the roadmap. That's the next domain. On the top, you see something called a customer-facing services. These are primarily not what we plan to do. This is specific to an operator, specific to users, how they want to use it. People might want to use it as a science project. They might want to use it as a live production environment. So we don't know that. And also, we don't know how each individual user would like to use it in terms of monetization. So we have kept that open, but any suggestions are welcome. We have a list of email addresses where you can send us your suggestions on how we can take this ecosystem further. OK. So talk a little bit about these specific open source projects, you know, SNAPs, which originally stood for SDN and NFV application platform and stack. As you can see, I got tired of saying that at conferences. So we started saying SNAPs. Man say this is very similar to OPNFV. And it is. And that's kind of where we started. But we really wanted a platform that was a little more stable and focused on targeting what we can do with the VNF. So something that runs a little more. It comes pre-configured. There's less decisions to make when you stand it up. If you look at a lot of the NFV demos, it's like we're using standard. Plus we did CPU pinning, large memory pages, DBDK, and things like that. And we've actually integrated all those in the base install. Done a lot of the network configuration and some of those kind of fun things up front. So you don't have to do that. It's an open platform. We've had issues in the past where we've tried to collaborate on one of the four-fee versions of OpenStack. And even though we were working with the vendor and the partner, and we had all the licenses, and all it took a month and a half to get the right license key to run on our site and not the customer site. And that's why it's nice to run in pure open source. We haven't modified the upstream packages much. We really are basing this on COLA and COLA Ansible today. So we're just trying to make that easier to install, make some more of the decisions on how we configure it. So it's a little more consistent for doing proof of concepts and testing. You can see kind of our main projects we have. Boot, this is where we install Linux. We set up the large memory pages. We do the CPU pinning. We set static IPs, all the stuff that most of the installers, at least the open source ones, don't do today within Kubernetes. OpenStack, so we have SAF, we format the disk. And just to make it a little bit easier, a little more usable to do, we have a couple members.  And we're seeing a lot of vendors run this. And so that way they can actually show that we have this interoperable. We actually had at our summer conference virtual CCAP. So the data plane for the Cable network running on Snaps OpenStack and a commercial version of OpenStack were able to run the same set of VNFs to demonstrate the interoperability, had one orchestrator managing both. So that's kind of where we're trying to go with this, right? It's a lab. It's not meant to necessarily replace what you do in production, though. It could be hardened to move to production. Kubernetes, this is where we've been spending most of our time these days. Like everyone, right? It's new, it's shiny. We're really looking at Kubernetes on bare metal because you look at a lot of the Edge OpenStack cases. It's Kubernetes or containers with OpenStack on top of it. And then, oh, by the way, you're going to run containers and Kubernetes in the VM. So you kind of have Kubernetes, OpenStack, Kubernetes, just so you have a very flexible, secure environment. And with the Edge, especially the closer you get to the customer premise, you really don't want that extra overhead. So that's where we're looking at Kubernetes. So some of the stuff, right? We have data plane acceleration that we need here. So for packet processing with SmartNX and DBDK, transcoding, you have to have your TV coming in. You don't want to have to transcode it on your TV. We use GPUs. On cryptography, there's dedicated device Nix or stand-up cards and some SmartNX where you can actually do the cryptography on. Yeah, and this is some of the roadmap that we have planned. And that's what is the call to the community. Of course, the orchestration that we talked about, the distributed and decentralized orchestration, is perhaps one of the most important things to build for Edge. So we would be working on that. We also need to solve the problem of monitoring and diagnostics in a heavily distributed Edge. Like, there are thousands of edges, and application developer needs to have one single view about monitoring and diagnostics on its applications. So that's the other next problem that we want to solve with this particular project. We also need perhaps a service mesh which goes beyond a single Kubernetes cluster. It needs to talk to East and West. In Edge, there are use cases where we feel it's important for edges to talk to each other, particularly from an application standpoint. And we perhaps have to build that kind of an architecture next to ensure that the applications are resilient and able to move between edges in a certain way. Even if it is stateless, it still needs to move across edges. Security is very critical. And these are some of the capabilities that we would like to add. Many of them are existing open source projects which we are evaluating and trying to see if it fits to our set of requirements. This includes image scanning, policy engines, authorization, identity-based networking. There are plenty of security work that still needs to be done specific to Edge. And then, of course, serverless. I mean, that's another keyword. And it's important from Edge perspective because we want to run completely even-based architectures and off-load computing from device to edge seamlessly. You know, talk a little more about that kind of inbuilt tooling and monitoring. You can't kind of take all the data from all the head ends and centralize it if you do. We've seen what's just doing that with traditional hardware environments, which probably have a little less metrics. It often takes a half hour to consolidate the data across multiple regions and geographies. So we really want a lot of that done at the Edge and then only sending the interesting stuff back to consolidate globally. So that's where a lot of this stuff is a little unique to the Edge because you have thousands of data centers and you can't necessarily have that large heavyweight solution to do it at every data center, but you still need to do something. And then, of course, we have a few things to do on the cloud native side. The developer ecosystem still is fairly, I would say, at a nascent stage and we need to build that. That's one of the things that we are trying to do. We also want to standardize SDKs in such a way that the application developer doesn't have to worry too much about how the system is functioning, how the access network is. That's not an application developer and AR developers strength. You don't expect an AR developer to understand NFV. So how do we actually abstract that kind of complexity to an application developer is also some of the key problems to solve in Edge. And then, of course, there will not be a case where you're running everything on Edge. So there will be situations when you're running something on Edge, something on public cloud. There must be a kind of federation that needs to be done across Edge and public cloud. Kubernetes federation is one way of doing it, perhaps, but there could be some other ways as well. So that's another area that still requires exploration. Yeah, I think we'll skip these next couple slides and trying to be complete in our stuff. And talk a little more about our Kubernetes release. It is a certified cloud-native compute foundation installer. We just completed the 111 certification. And then we'll have 112 done fairly soon. Again, we'd love to have collaboration. We've had quite a few people start to pick it up already. Pretty much everything's on GitHub. We also have Project Snapse OO, which runs a bunch of the test tools and is used by a lot of the certification and other tools within OPNFE, Snapse OO. And again, that's hosted by OPNFE. Yeah, we're seeing new challenges with NFE and Edge for operators. But we want to take advantage of what's out there today. We don't want to reinvent everything. We want to make something that's accessible to a broader developer ecosystem. You should be able to write an app and have it run on a telco or a cable operator or all the operators' infrastructure and take advantage of those low latency, high bandwidth opportunities. And we have our own little IRC channel, but mainly do things over email. So again, reach out. And we have plenty of time for questions, maybe? Any questions? You mentioned a unified approach for the solution. But what do you think of extending it for these workloads targeted for Edge? Like allow users to leverage the same solutions as infrastructure uses and maybe even Kubernetes on top of it. And there is really a lot of things that can be unified there. What do you think? Yeah, of course. Eventually, perhaps having a unified project is excellent. But I think there is place to have multiple initiatives as of today because everybody is trying to solve a different part of the Edge, different problem of the Edge. We are trying to build a platform which is easily installed and you can easily run applications from a developer standpoint. And that's perhaps a domain that we are trying to solve. We don't know what's in future. Maybe there will be some kind of an unification of projects. But as of today, we are trying to solve a certain set of problems that we defined. And eventually, based on usage, we'll take a decision on that. Yeah, thanks. I can think at least of the one thing which is really common for all, probably it's state synchronization. Is what? State synchronization, like database or message queue. And here is a nice space for unification. Exactly. Thanks. Thank you. So you had mentioned a couple of pain points around policy engines and authorization. Can you speak a little bit more to the requirements that you're finding in Edge and how those systems in OpenStack and Kubernetes are lacking in that area? Yeah, and we've seen a lot of progress over the last few years in this area. But it really comes down to how can you get that low-latency IO performance. So you need to coordinate your schedulers around both compute or acceleration and network and possibly storage. And you really want to have, if you have for your VNFs that are service-chained together, you really want those on the same system and then have them on a different server for doing the other chain. And you want that done automatically, but also could work across multiple if you have to. Some of the other things, looking at things like RDMA to lower that overhead of doing that communication between the VNFs within the data center, where you don't have to have all that overhead of the IP stack and the latency of establishing a connection and sliding window and all of that when you have a much more reliable connection. OK, so like round. And there's another thing about the end-to-end authentication. So you have a device which could be a drone and an application running on it. And the application has been written by a developer who must be authorized to run on the platform. Then you have multiple edges. And then you have the operator. Then you have storage. Then you have so many different moving parts in edge. So having a common way to authenticate applications and authorize applications to work in a seamless manner is not going to be easy. I mean, it apparently looks so. But we'll find it out soon, whether the existing infrastructure is good enough to handle so many different authentication chains. And then, of course, there's access network. So like identity sprawl. Like too many user accounts, essentially. Yeah, essentially. Multiple user accounts and multiple trust domains are needed to span those trust domains. Got it. Thank you. Because if you were at these servers, you could see everyone's internet access in that area. Hello. Hello. You have the rule map to do the management and the orchestration. And my question is, do you have some rough idea about this type of dynamic orchestration between different edge nodes? I'm not sure whether I got your question. So it was the different. Some rough idea about the dynamic orchestration between these edge nodes. Yes, so we did some analysis on that. We still don't know which project to adopt there yet. It could be on app. It could be OSM. It could be something else. But currently, most of the orchestration systems that we have today is extremely centered around the VNF ecosystem and not exactly meant for edge applications where the application developer doesn't have to fill hundreds of pages of VNF descriptors. We don't want to do that. The dynamic orchestration between edges is very important in the sense that there will be mobility. There will be applications moving. A device will move. Unlike in case of NFV in edge, you will have the end devices who will be actually moving around. There will be orchestration from one edge to another. You might want to move back the application one step behind and move it back to the central office. At times, the application might even work better with public cloud. So we need to think of a mechanism where applications are placed at the right location based on the right attributes. Today, this problem, I don't think, is solved as of now with any of the existing orchestrators. But we intend to work on that in our roadmap. Do you have some specific open source component? Have you already looked at some open source component now? Not yet. So what we have in the open source component is already in our presentation. But there is our email addresses. You can send us your queries. We can try to answer them as well, whatever we have today. We have run OSM in the past on the platform. We'll probably get ONAP running at some point. It's just we need to designate support to support ONAP. And we didn't have that initially, as well as we have to clear, I think, all of our VMs off of one of our labs to spin up ONAP. And another thing is that in edge, the cores are valuable property. It's expensive. So we don't want to waste cores on heavy orchestrations. So we want to do very lightweight orchestration on the edge, but so that it does most of the stuff that you need to do within the edge. And then the heavy lifting can still happen from the central office. We don't want to waste cores in edge by running too many orchestration overheads on edge itself. Oh, thank you. We'll be around for a few minutes if we have some questions. Thank you.