Hi, everyone, and welcome to the first Manage 201. So one of the things we wanted to do within the Manage team is to share our knowledge more widely, because a lot of the things we do are really expertise or really narrow focuses, which one person shares. So myself, I've been working on SAML a lot, but Imra has been working on SmartCode authentication. James Lopez is the expert on import, export, and Rubens, the expert on billing. And on a lot of these things, we don't have much visibility into what other members of the team do, and even less for people who are not on the team. So we thought we'd create these 201 sessions where we kind of go into quite a bit of depth about how we work on these areas, and also that will help other people who want to contribute or to review the things we're working on. So today I'll go over quite a lot of things, but starting with what is SAML, why it's important, a description of the protocol, how to actually get started working on this, and then I'll go on to some other things like the security implications of some of these things. So what is SAML? It's a single login solution. So I'm sure you're all familiar with things like Twitter sign-in or Google sign-in, which we use ourselves at GitLab. And what these systems help us do is not have to remember lots of passwords for all these different services we use. So it means you can just click one button and you're signed into the service. But there's another side to this, which is for the company, they get a lot more control over how a user authenticates. So they might enforce that everyone uses two-factor authentication, or they might control which services you can access, or even remove access when you leave the company. So you go to sign-in and it no longer works, and the company doesn't have to worry too much about that because they set up their system and everything just works. So SAML stands for Security Assertion Markup Protocol, sorry, Markup Language, be some, otherwise. And the keyword here is assertion. So this is a confident and forceful statement of fact or belief. So SAML lets the organization make a confident statement that this user is James, his user ID is 27418, his email address is jadewasjames.gitlab.com, and he's a member of the Backend Developers Group, and he's on the managed team. So we kind of, so that's what an assertion is, and that's kind of the key to understanding how SAML works. Getting more into the detail, it's an XML-based protocol. So you see here we've got an assertion, it's got lots of different sections. The kind of interesting ones are these attributes. So we've got a UID attribute to user one. Here we've got an attribute that maps to group one and an email attribute. Another really key thing here is the name ID, because that's what we use to link a user. So when the identity provider, which is the service you put your password into, returns its response, it says, this is user one or this is user james.example.com. And so that means which user this is, that's the name ID. So if you're kind of flavor of how the protocol works, people normally would show you this kind of diagram. So you've got the user, you've got the identity provider, and you've got the service provider, which is, in our case, GitLab. But that's way too complicated to follow. So instead, I'll give you a quick demo. So at a very high level, we have this kind of My SAML button, and that's, in this case, I've called it My SAML, but that'll be the organization name. And so a user who wants to authenticate, they click that button, they get redirected to the identity provider. So here I'll put in my username, user one, my password, which is user one, password. Identity provider, you're running it locally on your machine, right? Simple SAML, is that what it looks like? Yeah, so I'll go into that a bit more later. But in this case, it's one I'm running locally on my machine. But in other cases, it'll be Azure, act ADFS, ping identity, one login, and behind the scenes, the company will have their own system where they've got all their users. And so it will, to the user, it's just the end of their password, but behind the scenes, it could be a number of different things. Yeah, I'm just curious, is it in the GTK to install this? So I'll go for that later. Okay, thanks. But yeah, feel free to jump in with questions anymore because there's a lot to take in. Sorry, yeah, so click sign in and you've ended up on GitLab. And I'll show you that again, and we're signed in as user one. I'll show you that again, but using a tool to show you what's actually going on underneath. So this tool is called SAML ChromePanel, but there's also SAML Tracer for Firefox, and I'll have linked to those later. So, and this time, of course, we're already signed in on the identity provider. So when I click this button, I'll immediately be signed in and redirected back, but there'll be a lot going on underneath, which the user won't see. So we've got these kind of two key parts. First, the request was made to, in this case, simple SAML to the identity provider, and we've sent them a request saying, please tell us who this user is, a bit about them and are they signed in. One of the things here is we've said the issuer, so that's who we are, so they can look up which service we are and respond appropriately. And another thing we've sent in this request is where we want the user to be sent back after this is all over. And that's because this is a browser-based protocol, so everything's happened in the browser in these requests. One other thing to note is that this request actually looks like this, because it's base 64 encoded, so that's why it's really helpful to have a tool that kind of decodes that for you on the fly and gives you this nicer thing instead. Where's the user insertion in there? Or... Yep, perfect question, because that leads on to the next part. So the identity provider responds with its SAML response. And back to that key part, it's the security assertion markup language. It replies with an assertion. Can't find it in here, so I can see the end of the assertion, but the stuff is all in here. So one of the things is the name ID. So in this case, it's a really simple thing. It said, this is user one. And it's also replied with some other stuff that I screenshotted before, so it's replied with the user and with the group. One other thing that is important for security is that this is a signed response. So it's replied with a signature value and also a copy of its certificate, so we can check that it's signed correctly. Just to go over some of those concepts again. So you've got the identity provider, so that's where you sign in. That's often called the IDP. So if you see that, it just means the identity provider. The service provider is us at GitLab or any other service like Office that you might be signing into. You've got the request, the response, the name ID, the issuer, the attributes, the response. Cool. And I just put some of these in here, but in case anyone wasn't watching the demo. The next thing is how we link accounts. So when a user's signed in and they click that button, we link their account using that name ID. So we say, the user currently signed in is the same person as name ID one, and we remember that link for future sign ins. With Instant Samal, users can also automatically be created. So if user two signs in, we can create an account on GitLab for them. And things are a bit more complicated on gitlab.com. And so I'll describe that. And any questions so far? Cool. So on gitlab.com, we've got another problem, which is that instead of configuring these, your Samal provider in the config, there's one, there's a Samal provider for every organization. So every single company that wants to sign up and use this needs to configure their own identity provider. So instead of having this kind of button on the left, which says my Samal, when you click it, when you go to user sign in, you have a page for each group. And so you get something like this on the right, saying sign in to my org. And so the configuration page currently looks something like this. And so instead of having it in config, it's configured per group. And that's an important difference because we then need to look up which group someone's signing in for and then respond appropriately. So just to go over that again. So Instant Samal, we've had that for a long time. It's in C and thanks to CERN for doing the initial implementation. But this group Samal for gitlab.com is something we've been working on this year. And it's very different. One of the things is the trust model because that identity provider that someone configures could actually be a malicious one, a one we don't trust. So I'll come back to that a bit later. And yeah, so I'm gonna have a sip of water and I don't think there's any questions. Go ahead. So how do you get started with all of this? Six months ago, I just said you need to set up Active Directory. You need to configure ADFS. You need to configure some users on that. You need to add certificates and then you need to set up some LDAP transform rules. But you don't have to do any of that because we have a Docker container in GDK. And so I'll run you through how to configure it with that. So the main thing or the first thing, sorry, is adding it to the actual config. So with instant SAML, it would have looked something like this. So you're saying what the fingerprint for the certificate is, you're saying where you want the user to be redirected back to to process that SAML response, where you actually want the users to be sent to for the identity provider and who we are. There's also this name ID identifier format which is less important, but we often want that to be persistent, but we'll leave it like that. But for gitlab.com SAML, it's just add the name group SAML and then you configure it through the UI. The next thing is to run this simple SAML PHP service. So we'll do that with Docker and I've forked, someone had a Docker image all set up and ready to go. And so we've made a fork that was minor changes that aren't too important. And we configure it like this. But first we'll go in and we'll create a group and I'll just sign in as root to do that. In reality, you wouldn't sign in as group. Sorry, you wouldn't sign in as root, you'd sign in as a user on the organization. So we'll just create a group and we'll call that. And so when we're configuring the identity provider we'll need those bits of information. Sorry, that path. And I will just stop the one I was running before because I'm gonna reuse the same port but in theory, you could have lots of these running. Then we're configuring the two things, same as before, where we want to be sent back. So that's sometimes called the assertion consumer service URL, which is a mouthful, sometimes called the callback URL. And we're configuring the issuer which here is called the entity ID. So there can be a lot of confusing terminology like sometimes assertions are called claims but it all starts to blend in and make sense eventually. So next we'll go in and we'll configure SAML for this group. And going back to the GDK docs, it tells us the certificate fingerprint for that Docker container and the URL we need to use. So I'll fill that in here. So we've got the certificate fingerprint and here we provide the information that we used when configuring the Docker container but our users will often be configuring that on Azure or on ADFS or on any of a number of services. Now, one of the differences I mentioned with group SAML is users don't sign in from the kind of global GitLab.com page. They sign in on a different page for each group. So we go to this one and we've got a warning saying, you're about to link this account and it will grant you access so it can sign you in to GitLab. So these will click authorize and be redirected to simple SAML and I'll just use user one again. This Docker container only has user one and user two. So, oh, what have I done? The thing is I demoed this earlier when it worked. No, really. Yeah, actually working. I've seen this error when like my web worker wasn't quite working right. See, I'll just restart that. One thing that I do that not everyone does is I run a worker in a separate process to the rest of GDK and that means if I'm debugging I can jump straight in there. You know what I've done there. This isn't a problem with UDK. This is the way I've configured the Docker container. I've put its callback URL to my demo group but it should actually be groups, my demo group, SAML, callback. So we'll go ahead and we'll, so down. So here it should have been callback and here it should have been my demo group. So it does show that this is the URL that the identity provider will send users back to. And so, yeah, configuration mistake. Let me just go back to the SSO page and double check that we're listening on the right socket again. And so user one, user one pass. If I were to inspect this with the SAML panel we'd see very similar activity to with the instant SAML. I think I'll click a bit late. Cool. And so it's saying we've now connected it to our account. So the user does not need an account before that or do they have to have an existing GitLab account? So allowing this on gitlab.com with sign in is a work in progress merge request. So I'll just describe how it is at the moment but if when you visit that page, you don't have an account it'll redirect you to the identity provider because it doesn't know if you have an account yet. They'll redirect back. We'll show the registration page and if, or the sign in or registration page if someone signs in then it'll redirect them back and they'll be the account will be linked for the first time and if they register when they do that it'll be linked. So we don't have a automatic registration process but because of the way we're doing the redirects they'll get shown the sign in or sign or register the page if they haven't linked an account before. But one of the things we're looking to add in the future is something called SCIM and so that allows organizations to add and remove users automatically so that they don't need to worry about this as much. And just to include a link to the tool that I was using and there's also SAML Tracer for Firefox. Any questions? Again, cool. So now I'm gonna go through just some areas of the code so that if you're reviewing a merge request or trying to look for security flaws you have a vague idea of where to look for things. There's quite a lot to go through so I'm just gonna start jumping into the code and kind of showing you different areas. So feel free to jump in with any questions. So we'll start with the identity model. So this is used by LDAP and OAuth sign in but one thing we do differently with Group SAML is we also store which SAML provider has been used. So when we're looking up who someone is we can say they've signed in with this SAML provider and they've got this UID, sorry, this extern UID which is the name ID. And so with that combination of we know who it is from the name ID and we know which identity provider saying who that is we can identify the user to look up. The SAML provider itself is just those things that we configured in the user interface. So that's the certificate fingerprint with a link to the group. And through those identities it lets us find all the users for that SAML provider. And you can see that we've got this name ID format that would have been configured in the configuration. Here that's configured in the SAML provider. So one of the key things about how this works within our code base is that we use Omni-Auth. And so that comes in before we kind of hit the rail stack. We, Omni-Auth does it stuff in rack middleware and it has its own kind of way of doing things. So it has a setup phase and a request phase. So that's where we create that auth request we saw earlier and then it has a callback phase. The thing we do differently for Group SAML is we, in that setup phase, we look up which SAML provider this is, do some verifications on that and then use the appropriate configuration for that provider. For instant SAML, we just use the Omni-Auth SAML library as it is. So just to be clear, it looks like the Group SAML stuff is only in EE, right? Yeah, so with C, it just uses the plain library itself. And just to give you, like when I'm developing this, a lot of the time I do have to come in to the actual library and have a look at what's happening here. And that's all configured here. So we've got the request phase. It uses the configured settings. It does a bunch of stuff with the fingerprint. And it also calls through to Ruby SAML, which is a library that does the kind of lower level validations and things like that. In both cases, the user then comes after it's been validated that that's a valid response of a correct fingerprint and everything's correct about it. Only then do we get through to the Rails stack and to the callback controller. And we kind of have, so it's, we kind of have two different paths that can be taken here, ones when the users are already signed in and ones when they're not signed in. And so it's a bit weird because we do two different things and everything's in two different places. But the kind of key thing to wrap your head around is that we have an identity linker. And in here, we say if the user's signed in, then we either link their account, if it's the first time or if it's already linked, then we do that check in here as well. And this code here is actually shared across different OAuth things, across held app. And so it's something that we override specifically for Group SAML there. If the user is not signed in, then there's this whole other bit of code. But it also handles things like logging, audit events, setting, remember me tokens, dealing with two-factor authentication. And by having that shared with the other ways users can log in, it means we don't end up accidentally forgetting to do two-factor authentication with SAML or anything like that. The OAuth user classes are where that's then handled. And so we have one for SAML and one for Group SAML. I want a branch where I don't have that implemented, but it does very simple stuff. Looks at the user and just returns if they're signed in or not. So as you can see there. So that's a very high level run through. Look, those are the key things to look for. Cool. Security. It's in the name. It's the security assertion markup language, which means that there's a whole lot of things that could go wrong. The main kind of ways that SAML helps protect us is it's got this protocol that's very well thought out, very well specified, and then it's signed with that certificate. And so you've got XML signing and you've got all these validations that different parts of that XML are correct and as we expect it. But that also means that those are all the places where it can go wrong. So attackers can bypass the XML signing. Sometimes they'll do this by creating a second element with the same name. And depending on how you pass the XML, you might get the one that's actually valid or the one that they've inserted. There was an attack where they use XML comments to change a user's ID. So instead of being user ID one, they can make it user ID one, XML comment thousand. And then using that, you can have it so that when you look at the XML one way, you're user one, and when you look at it another way, you're user 1000. And by doing that, they can choose which user that they want to be. And so there's all sorts of very clever attacks based around this. And so, yeah, I've got a slide here showing all the different validations that Ruby SAML does. And so you can see it validates the response state. It validates the signature, it validates that the issuer is the correct thing because if it wasn't GitLab but Office, they'd send that SAML request, they don't be completely invalid. It validates the time on the response. So when we get back a response, it only has a short window in which it's valid and that prevents replay attacks. But then it has another time that says this user is definitely this user for this many years or months or weeks. And so they can, so the identity provider can make these assertions with different time periods as well. Yeah, every single one of these is something that an attacker can work around. So we have to be careful thinking about those. Luckily, Ruby SAML takes a little of that away and does it all for us, but there's a lot of things we also need to be careful of. And I'll just jump to this slide for a bit. One of the complications with this multi-tenant SAML is that the identity provider might not be trustworthy. So when we're configuring in the config, our own identity provider, we know it's trustworthy, but when we allow any user on gitlab.com to add their own identity provider, they could then trick users into visiting some site where randomly on the internet being redirected to their malicious identity provider, which could then send back a valid response all by itself and then, hey, that identity provider can now sign its own users into GitLab so they can then go on their computer and pretend to be you and hijack your account. So although we haven't changed much, we have to be extra careful and we also need to protect against social engineering because we've got that warning page saying, are you sure you want to link this account? But if someone doesn't know what that means, they might give permission to someone else to sign them in. And some of those attacks I've been talking about, they've allowed uncontrolled access to Office 365 and to GitHub, so they're not just theoretical attacks, they're ones that people have actually gone out and found ways to, for research purposes, enact. But yeah, so we need to make sure we're very thorough when reviewing this and think about all these different possible scenarios. Questions? James, is there anything we can do to kind of approve or like pre-check the identity provider? I guess not because it could potentially be self-hosted and something that we're completely unaware of upfront. Yeah, so it's something, we could place an organizational process around this and saying whenever someone enables SAML, we'll check their identity provider. But another kind of attack that maybe kind of not like that as much is that someone might set up an identity provider just to play around with things and then that might have a vulnerability that can then be, if someone can get access to any one of these valid organizations, they can start attacking users as well. So it still wouldn't completely protect us and it would add a lot of process overhead. So it's not perfect. So just to back up here a little bit. So, you know, with LDAP Group Sync, basically you've got groups in your LDAP provider and then as soon as you log in, we automatically sync users that belong to the group. Is the same thing happening with SAML here? So it's something we'll plan to do but at the moment, they configured the identity provider for the top level group and we don't do anything other than adding the users to the group. But because we've got those assertions and they can say this uses in Group X, we've got plans to then use that and say they've asserted their own Group X, let's add them to Group X. But that's something we need a bit of design around and also which permission level they have because at the moment, they'll be added as guests but we might want to allow them to add people directly as developer or maintainer. Okay, so the user flow right now. So let's say this gets deployed to GitLab.com next week or whatnot. People set up there, they have a group, a GitLab group. They set up their own SAML provider, right? And what does that give them now? That gives them ability to just people to log in and get access to that group or they start to manually add users to that group? So what we've already got on GitLab.com is that ability that when someone signs in with SAML, it automatically adds them to the group. So if I sign into that MyOrg SAML provider, I automatically get added to that top level group. What we don't have on GitLab.com yet is the bit that actually signs you into GitLab.com. So they need to enter their username and password first. So it's kind of halfway there. Ah, got it. So it syncs to the existing group but it doesn't automatically log them in. Yep. Got it, okay. Yep. And that kind of leads us on to the things that we want to do next. There are also things that we're going to need security review on, things that we're going to need UX review on and things that I'd like to be able to sign to maintain as well and having to say, I don't know what this is. I'm not too sure. So these are all things that are relevant. So as I mentioned, the actual sign-in bit is something that's work-in-progress merge request. And as part of that, there were these kind of three other merge requests allowing users to unlink their accounts and providing that warning up front. There's also this metadata configuration because one of the things that SAML administrators like to do is just get a URL for some XML, auto-configures everything for them. And the next stage is providing administrators with more power over enforcement because they want to be able to say, all the users on this group all use SAML so that no one can be added or accidentally get added. That's just a normal getLab.com user. Another thing that we see people wanting is the ability to say that that user is currently signed in with SAML. They haven't signed in with their getLab.com username and password. And so that lets them for audit reasons say, yes, they've definitely used all our policies. They've used two-factor authentication and we haven't removed them from our system because they're signing in with the right provide with SAML. And then two other things are making those memberships expire if someone hasn't signed in recently and then allowing them to mass-add users and remove users when they leave the company. And to do that, we can't really use SAML because it's a browser-based protocol. So if someone's left the company, they can't exactly log in and go through the flow and be trusted to do that. So we need to look at another technology, SCIM. And then kind of next area for improvement is the stuff we're doing up with setting the access level. So whether they're guest, report, maintainer and deciding which subgroups to add the user to and also making sure that those notifications go to the email address they are provided in that assertion rather than just their personal email address. Finally, just some of the area we want to work on after that is making things easier for the user. So things like sign out a better registration flow and just items which are polished in the UI. So things like displaying the active sessions on the user profile correctly. To recap, SAML is great for organizations, managing users, great for individuals, only having to click one thing to sign in. Group SAML is the thing I've been working on. And we've got a lot more to go, but it's going to be very powerful for organizations which want to move to GitLab.com instead of self-hosting. And so that's something that for sales is really important and something that's really important for a lot of organizations. And so it's a, yeah. Just some resources that are linked to. You've got the full SAML specifications in all their detailed glory. Some links to the doc. And you can reach out if you have any questions after in GManage on Slack or myself or Jeremy Watson's the PM for this. Any final questions? So I'm just curious in terms of like people, why would someone choose using a SAML provider over like an LDAP provider? So they're kind of a different beast in some ways. So for GitLab.com, one of the reasons not to look at LDAP is it sends passwords over the internet. So someone using LDAP, they put their password into the UI and we don't really want organizations having their users give us their password each time and sending that over the internet to their server and it's a very different approach. For self-hosted things, when it's all within a contained network, sometimes it's just the convenience for the users. It's one click. If they're already signed in to their office account or their gyro account or anything else, it's just one click and they don't even need to notice it. They don't need to put in a password each time. Yeah, it solves a very similar problem to LDAP. Cool. If you have any other questions that you think of straight after we finish, just as I said, ping on gmanage. It takes a lot, James. Thank you. That was awesome, James. Thank you. Thank you. Thanks, James. No worries. See ya.


Steve Newman reports on one person already using the brand new system. This is something that you guys have requested for a very long time and after two and a half years of living here I finally feel like I've made it my own and decorated it how I want and I'm ready to show it to you guys. Alright so let's start off with the bed and the couch area. Honestly if I was redoing this from the beginning I would have really liked to have a double bed in this room but unfortunately with the space as you can probably see from that wall there till the door it's basically just enough to fit a bed and a couch. When this is my couch over here I've recently changed it and if you've been following my channel for a long time then you'll know. I'm really proud of how I colour matched the pictures on the wall over here with sort of the grey and yellow vibe with the grey and yellow cushions. This couch is also quite big it fits three people quite comfortably and if I have family or friends staying over it opens up into a full double bed. This coffee table over here it's glass and has two levels so it's really good when we have multiple friends over and we have lots of food to put on the table or if we're playing board games we can have people sit around it and still have a lot of space to lay everything out. So there's nothing too special about the bed itself but it does have a very important function which it has a lot of storage. So down here is basically where I keep everything that I can really keep in my room because there's an enough space. I've got an ironing board of my suitcases, bags, clothes that I don't wear that often, anything that I need for fixing the house like screwdrivers, drills, nails all these things that I definitely wouldn't want to have laid out in my room but I still need somewhere in the house. I love that this room has floor-to-ceiling windows as well it really lets the light come in especially in the summertime. I know London doesn't have that much light but I'll take as much as I can and there's also a small balcony out here which is fairly rare for London. We have two chairs where we can sit down and sort of enjoy the weather when it is a little bit better but now it's really cold so I'm just gonna close it and then a little behind the scenes that you probably will never see again in a video. Is that down here I keep all the filming gear that doesn't really fit anywhere else in the room. Two sets of tripods, I also have a C stand. Okay next to my bed I've got a radio alarm clock and this has actually changed my life. It makes waking up in the morning and getting out of bed at 6 a.m when it's pitch black outside so much easier. When you're waking up really early it's so much better to wake up to the sound of somebody talking to you, your favorite radio station, some music whatever instead of that classic iPhone alarm. Over here on the wall these are posters that I actually bought back in 2013 in my first year of university in Toronto and I really really like them so I've kept them around. Also I'm really proud that I managed to put them up all by myself. They look quite neat and I feel like they're equally spaced so I'm really proud of how they turned out. Another really cool thing about my room is that I have this TV directly opposite my bed and that's really nice because obviously after a long day at university studying or in the hospital on the wards I can come back home and relax in bed while watching something on the TV and it's on this really nifty bracket that allows it to get turned like this so that if I ever have friends over on the couch or me and my sister are in here chilling eating lunch or whatever we can view the TV from any angle in the room. So down here is basically just like a TV stand even though the TV is mounted on the wall there's obviously some things that we need down here. So this used to be filled with a PlayStation, Xbox and a Wii but I didn't really use them as much so I've just left my PlayStation down there. This is a set of Alex drawers from ITEO which I'm sure you guys have all seen before many many times. It's a great set of drawers because it holds a whole lot of items so it's very good for storage if you have a lot of things laying around. I mostly keep knickknacks and stuff that I use fairly often here in the top drawer. I've also got an entire drawer for things related to filming and YouTube videos so mostly hard drives the remote control for this light over here and stuff like that. And then down here I've got this big white box which basically holds all of the clutter that would be underneath my desk had I not had it. This is also a great way to hide all of your cables and make sure that it doesn't look very messy underneath the desk. And these two drawings over here I did in my first year at medical school. This guy you can see on the left is half bones and on the right is half muscle. I did this before I started my YouTube channel when I was looking for something to do in my spare time and I wanted to express my creativity in some way. They took me a really really long time to complete but I'm proud of how they turned out. Maybe one day I'll color them in but honestly I'm just scared to do it because I feel like I'm gonna ruin it if I do. So moving on to the next part of my room this is the productivity area where I do all of my studying and all of my work. Let me just check out the way. So I've done a desk setup video before and I'm definitely gonna update it because I've got quite a few new things that I want to show you guys but a quick overview. This is a massive IKEA desk. It's absolutely huge but I love to have a lot of working space when I'm sitting down at my desk and before a thousand of you ask me in the comments down below this is a 34 inch LG monitor. I've got my laptop over there in clamshell mode so it's just sitting on this stand here connected to the monitor and then my iPad sitting over here on another nice space gray stand and yeah I use these two things in tandem when I'm working all the time mostly my iPad to take notes and my laptop to do anything else that I need. Over here I have this small wooden tray just to keep the things that I always put in and out of my pockets so currently it's holding my wallet my AirPods and just some loose keys. Check it out. Moving on over here to the left if you guys have been following this channel for any period of time then you'll know that this was for when I got 1,000 subscribers on the YouTube channel. Here we are at 80,000 now. I always keep this on my desk just as a reminder of how far this YouTube channel has come. I said in the video where I showed this that if you could guess the correct number of M&Ms in here I would send you a postcard and nobody has guessed it till today so take a good look and tell me how many M&Ms you think are in here. If you guessed it right I'll send you a postcard. This desk lamp is from Ikea. It's also really great because you can move it around a bunch till to up and down left and right whatever so it also takes a very little space down here on the actual desk which means you can have more space for other things to put and then the final thing that I want to talk about in this desk area is this lovely lovely neon sign. It took a lot longer than we expected to get it done so I'm really happy that it's finally here. It's yellow when it's turned off and then when you turn it on it's this glowing orange color. Moving on to the closets over here I keep it quite simple. I dress fairly minimalist so these closets I feel really add to the aesthetic of the room. The fact that their own mirror makes the room look much larger and it helps reflect light and make it a lot brighter in here. So inside the closet obviously I keep all of my clothes and I try and keep it as neat as possible. I keep the t-shirts fairly color coordinated so all the reds, whites, blacks and grays close to each other and then on the right I have all of my hoodies because as you can tell I love wearing hoodies. I try to keep all my clothes quite clean and simple. I don't really like things with a lot of patterns or writing on them so that's mostly what you'll find me wearing. And then here my sock drawers, my trousers etc and I find it quite handy that there's a laundry basket down here because it doesn't take up space in the rest of the room and it's hidden away nicely at the bottom of the closet. Over here on the right I have all of my bigger jackets and my formal shirts that I wear to the hospital. Here I have my kangaroo onesie that you might recognize if you follow me on Instagram and if you want more updates on my day-to-day life at medical school make sure you follow me over there. So down here I have these two sort of trades that hold all of my shoes. Yeah they're also very good for making most use of the space that's here. And then finally let's talk about this shelf and what's behind my door. So this shelving unit over here really helps keep things organized in my room. At the very top I've got all the files and folders for notes from my old university and it's also where I keep everything that I need on file or a physical copy of so banking or tax-related forms and things like that. This shelf over here is sort of my medicine shelf. It has the netters anatomy flashcards that you might have seen in a previous vlog, has my stethoscope, my hospital cards, my portfolio and anything that I might need for studying medicine. This shelf over here is dedicated to people that I really care about but I don't get to see as often as I would like. So these two shelves over here is where I keep most of my camera gear and it's usually things that I use for vlogging like all these different tripods and then lenses and cameras and of course the camera that I'm using right now to film this video would sit over here. And then the last shelf at the bottom just has a couple of miscellaneous things like video games, some cards, sunglasses, whatever. And then finally behind my door I have two sets of hooks and this is a really good way to get some extra space in your room especially as a student. These are really helpful for just putting things up when I don't feel like folding them and putting them away in the closet.  And all right I think that's everything that I wanted to talk about in my room so this concludes my room tour. If you did enjoy please do leave a like on this video and subscribe to the channel to see more content from me in the future. I hope you got a little bit more insight into the space where I spend most of my time either studying or relaxing and maybe you got some inspiration for how to decorate your own room. And also I'll leave links to pretty much everything in the description down below so you guys can go down there if you're interested in any of the items that you saw. And I will see you in the next one. Peace! What is up guys Karma Medic here and welcome back to another dose for that sister. I'm sorry. One cool thing, one... Is that on a nifty blanket? Yeah nifty blanket. It also takes up very little space. Can you spin it around or just up down left and right? This shelf over here is dedicated to people who I really care about but I don't. As well as things like banking as well as... And it's also where I keep anything that I need to keep. Yeah girl! That's definitely going in. No no.


Hello. Good afternoon to all the audience here. And I'm very excited to be here. And I feel that all you are also excited to part of this Cube Day India. So here I am going to present this topic. Use Meridio to create network services for telecom use cases using secondary networking for Kubernetes. Little will talk about me. I'm Gaurav Bhatnagar. I'm playing the role of ceiling system manager as part of technology system management, business area, cloud services, business and operations support system. And you can see my LinkedIn profile here. We can always connect on the LinkedIn. And basically, I enjoy reading and playing tennis. I'm CK certified. I'm a AWS cloud practitioner also and have certifications from BrataStack as a team forum. So definitely we can connect here. So I have been involved with Kubernetes from last around seven or eight years, primarily in the telecom domain. So as part of Ericsson, we are basically in the journey of cloud native. And we are basically deploying our products on the Kubernetes platform. So this is the agenda that I want to cover. So what is Meridio and what is secondary networking for Kubernetes workloads, telco requirements for Kubernetes workloads, some telco use cases where Meridio can be useful. So secondary networking, so just starting with Meridio, what is Meridio itself? So it is an open source project. It provides various capabilities to attract the external traffic within Kubernetes via secondary networking. So it is available at the GitHub. And it is primarily written in the Guru Primary Alliance. And if you look into Meridio, basically it works on the Kubernetes 1.21. It requires Pyre, network service mesh, and Linux kernel. Basically, Pyre is just to give the mutual authentication between different workloads. So it provides a empty list kind of support there for various Meridio ports that are working and deploying on the Kubernetes. Network service mesh basically is a hybrid multi-cloud IP service mesh that, for example, it facilitates the multi-higher communication between L3 domain. So if you have your Kubernetes workload deployed across the clusters of VMware or across the clouds, you can use network service mesh to create connections between them. So basically, this is Meridio. And it is basically using these underlying softwares. So bulk of work or functionality that Meridio provides a build on the network service mesh. So what is secondary networking? So when I first heard this term, I was still confused. Because if you look into the documentation, it always talks about primary interface and all. So just to what it is all right, so we have the Kubernetes port with primary network interface, it's 0. So through this interface, all the communication is happening, the readiness probes are working on this, listening to the communication channels and all. So now if there are more network interfaces in your ports and these network interfaces can be used to talk to other networks, then we basically call these network attachments as secondary network interfaces. So anything other than primary is secondary network interface. So because when I first heard this term, I was a little confused. So I thought it would be good to catch it here. So again, coming to what is Meridio, right? So as you see, this is a Kubernetes cluster we have deployed here. And it is your user application that is running in a port. So user application is a container that is running in a user port. Now to use the Meridio, what it does, it provides a sidecar container called a Tapa Sidecar Container. It is called Target Access Point Ambassador. So it is based on the ambassador design pattern. So it provides an interface through which you can control what traffic that is coming to your secondary network interface. So it provides a GRPC API hosted as a Sidecar container. So your application is there as part of container itself in the user port. And the Meridio Sidecar container is providing you the GRPC interface to connect to the external world, right? Now there are more components to Meridio. So if you see, this Sidecar container in terms talks to a network service. Again, it's a port. And it basically provides two functionalities, a traffic classifier and network services. So again, Meridio operates on layer 3, layer 4 to provide distribution via, again, so-called secondary networking. So it holds the traffic separation on the default primary network within the cluster. So the primary network interface is not touched. It is the secondary interface through which all the user plan traffic is passing. And basically, there's a traffic classifier. You can use it to separate the user traffic into multiple groups. So I have an example for that. Maybe that will become more clear when we go through. So as we said, we use this term service attractor, right? So what Meridio does, basically, it is talking to the external gateways using routing protocols like BGP and all. And it can announce the virtual IP addresses to the external gateways. And through this week, it can attract the external traffic towards your Tapa sidecar container, right? So from the gateway itself, the data gateway, the traffic is coming to your network service. That is a port that is deployed in a Kubernetes cluster. And then through that, Tapa sidecar container, right? So user application has the logic to process the external traffic. While external traffic is reaches through your container in the user port through the GRPC API, right? So some of the Telecom requirements that we can basically look upon when we expect from the Kubernetes workload, right? So IT industry has been using Kubernetes workloads from quite some time. But Telco industry is catching up. And I'm also the part of that industry itself. But if you look into the Telecom perspective, we have some very specific requirements, like end-to-end traffic separation isolation. So basically, we want the traffic to be separated from the beginner hand to the actual port itself. Network address translation, there are many protocols in Telecom like SIP, STCP, that basically the network address translation is problematic there. Cluster-wide virtual IP as a source address for egress traffic for application ports. So we need a virtual IP or WIP that is same even for the egress traffic, right? So I have a slide for that. I think that will make much more sense. And high-performance user-plane traffic, right? For instance, we have the media servers that basically have to download or stream the media services or media traffic to the end customer. And there we need a high-performance, right? Dry networking of Kubernetes is not that sufficient for that. Support for non-IP protocols like Ethernet and all. So that is the price of the telco requirement that is basically affected from Kubernetes workloads, right? So what I try to cover, basically, try to capture some telecom use cases so that we can basically demonstrate or see what is the power of media, how, and which of the use cases where it can help, right? So one is SSM is a legal protocol, and it's still used in a number of telecom products. So how the legacy telecom products do IP validation while establishing connection between peers. So still the peers in SSM do the IP validation, right? Segregation of O&M and real-time traffic. So real-time traffic is the voice traffic that basically we are making voice calls. O&M traffic is the alarms and operational traffic. So as a telecom operator, they want to segregate this traffic into all together separate paths, right? So basically, you want to classify and steer traffic based on five tuples, basically the source and destination IP source and destination ports and protocol, right? So basically, using these five tuples, you want to segregate your traffic, right? So for instance, DNS protocol is there. It works on both TCP and UDP on port 53. So might be you want to handle UDP-based traffic and DNS TCP traffic separately, right? So this is an example of, again, first use case. We have a signaling protocol, SS7 communication happening. So our user pod and this can be a remote SS7 peer. So I can take, for instance, this user pod wants to do a location lookup through using this SS7 peer. This might be an HLR. So whenever the egress traffic originates, it takes the IP of the worker node, right? And when we configure in the remote SS7 peer, is the load balancer IP. So all the services or whatever services that I want to expose on the Kubernetes is through the load balancer only, right? So I have configured that load balancer IP at the remote SS7 peer. But when, suppose, my user pod wants to do some lookup, location inquiry, right? It's initiated traffic. And it initiates a query toward the remote SS7 peer. Now what happens is that this is the default behavior on the Kubernetes. Since user pod IPs are ephemeral, they don't exist or known to the outside world. So when it passes the boundary of the Kubernetes cluster, it is replaced by the worker node IP. Now this worker node IP is not recognized by the remote peer because we have configured the load balancer IP, right? And we don't know where the user pod might reside later on, right? It can be on the worker node A, then it can move to worker node B, and that way, right? So what this problem solved by Meridio is that, again, it has a concept of virtual IP. Or WIP, right? So rather than publishing the address of a load balancer, we publish or configure IP address of the WIP, right? Now, all the traffic that is passing through is through the Meridio pods. So the egress traffic, when it originates, basically user application invokes the GRPC API of the Tapa Sidecar Container, that is target access point ambassador, Sidecar. And then, basically, it passes through Meridio. And again, then it basically reaches a remote peer. So in this case, no NAT happens because the Meridio pods take care of that. And basically, your egress traffic has the same IP address that has been configured, right? So for the egress traffic as well as egress traffic, you have the same IP address, right? So that remote SS7 peer is able to establish a communication because it recognizes that IP, right? Now, there is a separate requirement in the telecom that we want to segregate O&M and real-time traffic, right? So I have some real-time traffic where voice calls happening. I have the O&M traffic where monitoring alarms and handling are happening. So again, we can use Meridio for this use case. For instance, in this use case, we have segregated this O&M traffic and the real-time traffic on two separate WIPs, that is WIP 1 and WIP 2, right? And this Meridio, what it does, it is a concept of trench. So trench, basically, again, it's a normally culture that is there used in Meridio. It's seen as an extension of the actual network. It is a trench that upholds the traffic separation inside each network, inside the realm of the cluster. So basically, trench, if I'm more going to it, it is a custom resource definition. But what it does, it gives you a functionality so that you can able to segregate your traffic, right? So for instance, here, what we have done, we have created two interfaces, NSM0, NSM1. And we have published WIP 1 and WIP 2. So using this, you can basically segregate your real-time traffic as well as O&M traffic on total different network paths. So for instance, one can be on the VLAN 100, other can be on VLAN 200, that way. Now, classification and steering of traffic is another important use case where Meridio can be very useful, right? So what I have taken a use case here, that suppose you want a requirement that you have a DNS traffic, and you want to handle DNS UDP traffic and TCP traffic separately, right? So here, what is in this diagram? What happens is, flow A takes care of the UDP part of the DNS, while flow B takes care of the TCP part. So flow A takes the UDP-based DNS traffic, which passes the conduit through stream 1 and into target A application, while flow B takes care of the TCP traffic, right? So let's go to the next slide. So starting point is that we have deployed our user part with a TAPA sidecar container. So we have the user application and the TAPA sidecar container as the user part. Now, we have also deployed the Meridio parts. So first, we created trench. So trench is, as we discussed, it's defined the extension of the external network into the Kubernetes cluster. So it's nothing but, again, a custom resource definition that is created, right? So when we create a trench, basically, we create a configuration within the Kubernetes in a conflict map, right? Now, we have created trench. So this trench can talk to external data center, right? Now, then we create a VIP. So VIP, basically, again, publishes a unique virtual IP that is known to the external data center. And these VIPs are basically user application is listening on these VIPs on the secondary network interface using the TAPA sidecar container. So basically, we have created a VIP here, and we have created a VIP with a trench. So we have created a VIP one. Similarly, VIP 2 can be created, and can be assigned an address, right? Again, conduit basically is, again, a custom resource definition there in Miradio. Basically, what it does, it provides as of now a stateless load balancer. So you have multiple pods, so it can load balance your traffic across that pod, right? So again, we create a conduit. We associate it with a trench, right? So now what we have done, basically, created a stream. Basically, streams a logical grouping of traffic flow. The stream points to the conduit where the traffic can pass. So it is an actual network entity in which you refer to in your target application, right? And so we basically create a stream. As in the example, ML, we associate with a conduit and trench, right? And now we create a flow, right? This flow basically what we are doing, this flow basically classifies your incoming traffic in the base of five tuples. That is, source address and destination address and the ports, right? And protocol, right? So basically, as part of this example, ML, we have created a flow, A, with that is associated with trench A with the stream one. And we also associated a VIP. And basically, it's accepting all the traffic. And the destination port is 53. And the protocol is UDP, right? Because we want to classify and define the use case itself, DNS traffic on the TCP and UDP, right? Now, what is basically, we have created a flow. But again, the traffic will come to the data center gateway. So we need to talk to the data center gateway. So there's another customer service definition in the media view that is called gateway. Basically, gateway is nothing but it basically it is a peer to the data center gateway node. And it basically mimics or basically have the configuration to talk to the data center gateway. So in this slide itself, we have created a gateway object, customer service definition that is associated with a trench A. And we basically give an address which gateway to talk to and what are the basically configuration or the specs through which it can talk to that gateway, right? Now, basically, this is the tractor part. So a tractor is the customer service definition in the media view that basically uses the gateway configuration to talk to the external gateway and extract the traffic from the external data center into the Kubernetes cluster. So it basically attracts and basically advertise this IP that is OK. This is a virtual IP that needs to and this IP needs to be this traffic, what's this IP, needs to be routed to this cluster. So it passes through the flow through this conduit stream and then to the target application, target application B. So using this, we basically can split the traffic, TCP and UDP traffic for the target A application and a target B application, right? So basically, you can use here an example of IPv4, but you can use a configuration for IPv6 also. So if in all, if I try to summarize, what are the key takeaway points that we want to summarize? So Billidio can be used to facilitate attraction and distribution of external traffic within Kubernetes. We have so-called secondary network with minimum changes to your application. So basically, your primary interface of Kubernetes is not touched. So all your readiness probes or liveness probes that our primary interface is not touched. So it is not expected to that you change your Kubernetes application to work with Billidio. But what you need to do, you need to deploy a STAPA sidecar container within your user port. And then you can use the GRPC API provided by the STAPA sidecar container to talk to the external world, right? So basically, it doesn't expect that you change your architecture or deployment model as such. It works with that only, right? It, again, next point, basically, provides a sidecar container using a GRPC API, right? So you can start-stop traffic toward the user port. So basically, using the GRPC API, you can start the traffic, stop the traffic, right? What is traffic that is coming from the external data center gateway towards your secondary networking interface, towards your user port? So using the STAPA sidecar container, using the API hosted there, you can basically start and stop the traffic, right? You have the control there. As a user application, your application is processing the data. So you have the control there on the traffic itself, how the traffic needs to come, how the traffic needs to be classified in that way. And the third point takeaway, I can say that, service address and web site are announced to the external data center. We have different kinds of routing protocols by a media front-end. So front-end, basically, is the component that is front-ending with the external data center. It basically hosts the attractor in the gateway components. So it is attracting the external traffic on that web itself. So it basically advertises that this is a virtual IP for which it wants to attract the traffic. And when the traffic lands on that Kubernetes cluster, it basically classifies the traffic based on your configuration, and it lands to the top-of-sidecar container and then consumes your user application, right? So again, it can solve different telco use cases which are quite tricky to solve using an existing primary network interface or Kubernetes workloads. So again, a lot many telco workloads or use cases can be solved here, right? Because primary interface, by default, what you get out of Kubernetes is a private interface. And there are a lot of use cases, like, for instance, we discussed that if you want to high-performance user plain traffic, right? Again, it's still tricky to solve through the primary interface. You need some second interface through which high-speed data that can come to your application itself, right? So these are the key takeaways. So I have captured here that some references that are quite good. For instance, one talk by Leon in this here, it is very good, so it gives you an internal working of how this midi-do is happening, right? It is very well explained there, right? So you can look into that stock itself to understand how midi-do is working, what is the role, and then some is playing there, right? So with this, thank you. Again, this is my Lincoln ID. You can always reach me on this Lincoln ID. And again, thanks for giving your patience listening and thanks for your time for coming for this talk. Thank you. Any questions? I'm here. Right, yes, that's all right. So my question is, if I want to expose multiple source IP address and each one for different, different destination IP address, then maximum how many source IP address it can retain for intercluster? Basically, you define a configuration that makes targets what you want to achieve, right? So you can configure multiple WIPs, right? So basically, it depends how much IP that you want to expose, right? Within your cluster, it's not an issue, but outside the cluster, outside the cluster, basically, when the issue is that when your egress traffic goes out, it takes the IP of your worker node, right? But you may not want that case to happen, right? You want to have your services to have a consistent IP even when your traffic goes out of the cluster itself, right? Yeah, so that IP, I don't want only one IP. See, if I want to talk destination one, I will use source one. If I want to talk destination two, I will use source two. And none of them will translate to worker node with this design. So how many source one, source two, up to source N? How many such N source IP, my port can have it? Virtual IP, can it have? You can add a number of virtual IP. As such, it depends on the application because these traffic that is coming on your, on these virtual IPs is consumed by your target application itself, right? So each IP represents a service itself, right? So there is not limit at such, right? But you can define a limit. For instance, you can define a max target itself that, OK, I want to only expose 100 target itself. But you can expose multiple VIPs. Even you can basically consume traffic of the multiple VIPs within your user cluster by the same port itself. OK, thank you. I have a small question in this context. And I'm just asking a very basic understanding. So this VIPs and all, do you need a multi-CNI to create this? How does it, because secondary interfaces are normally in telco world, they're always a multi-CNI. And then it creates this virtual Ethernet VTH0s and all. And then is it something in between? Or you can, and is it very similar to Envoy in primary? Is this meridio is very similar to Envoy? Is it correct understanding, actually, what? Correct, in a way that basically use the multis to create a secondary interface. But what it does, basically multis is a multiplexer, it basically creates a CNI or CNI. So it will help you create a secondary network interface. And then virtualize also. All of those use cases, what I was going to say. After creating a secondary interface, you want to attract traffic to that secondary interface. And basically you won't have services as published passing through. Because unless until you code in your application itself to listen to that secondary network interface, things will not happen. Where meridio help is, basically, it again uses an underlying NSM, right? A lot of wiring work is done by the NSM only. So it creates, basically, what multis is doing, it's done by this NSM itself. So it creates a secondary interface, adding your pod itself. And then basically meridio what it does, it basically using the attractors and gateways, attract the traffic, what the web app that you have published on that secondary interface. So NSM, the network service mesh, basically is what is doing that you will expect to multis. So but multis has a specific purpose, you create a secondary interface, and then you are there. But you have to basically develop an application or your logic itself to attract traffic to that. Your pod itself will not listen to that secondary network interface. Your all services will work on the primary network interface. Okay, so you're saying is that there's a network service mission on top of that, this architecture of meridio fridge? Right, right. There's no need of multis CNIs as such. As such, but there is a configuration where you can use multis also within the NSM also. Basically, multis, what is happening is basically is giving you flexibility or functionality to create secondary network interface. Absolutely. So NSM is also doing the same thing, right? Okay. So it's doing much more than that, right? Got it, got it. Thank you. So like in our environment, we are moving from service message to CELium to minimize that sidecar container context. And we were using multis in there, in user plane functions we have. So how like in here, Tapa is acting as a sidecar container. So is there any way that it can be handled without sidecar container? Like the kernel space, it can handle and there provision those. Okay, the present architecture, what we have is the meridio itself. It uses sidecar container because what it does, basically the sidecar container holds or basically give you a GRPC API interface where your user container within that user pod can query and basically get the data, right? So this is how it works, right? So basically the sidecar container is giving you a flexibility in the sense that your application is remained what it is, right? It only needs to call the GRPC API to get that external traffic, right? And definitely you need to do the configuration and all that for all this thing to working. But your application can work the same, right? You just need to add a sidecar container to your user pod and then call the API and then basically using the GRPC API you can consume that traffic itself, right? So you need to have a sidecar container, top of sidecar container to basically use this. Hi, so my question is how do we scale the pods here because is the VIP associated with each pod or is it at the service level? If it's at the pod level, then we might have to create the other resources, right? The range and the VIP CRs itself. Okay, it depends on your application how basically load intensive, how it is there, right? So Meridian, what it does, it is giving you a virtual IP where you are exposing your services, right? So same in the pod itself. So each pod will have a one sidecar container. That is the right. And number of pods that are there as part of the Meridian application itself. So it depends on how your application is basically looking, right? Or what are the throughput requirements of there, right? So that needs to be worked out, right, that way. So for each pod, we'll be having one trench defined or is it at the service level? So this, so again here, basically one user pod has one sidecar container and again these set of pods that are part of the Meridian pods are common, right? So these are, you can say this, this is part of something underline infrastructure that all pods there can use, right? So, but again, you might want to create a replica sets or replicas of the pod itself, right? So it depends on the throughput itself. But once deployed in your cluster itself, this is, the network service is common for all the pods, right? The top of sidecar container is specific to your user pod, but this, the network service is common. That's all. Thank you. Thank you. Again, thank you all for asking questions and coming here, giving your time. Thank you all.


I mean, we had really a good discussion there in the panel and what was very clear is that we will have a growing demand of spectrum and it will not be an easy task to get everybody the amount of spectrum that they need for the service in an unlimited way. So we need to think how we can work in the future. And I would not call it a surprise, but it was certainly good to see that there was some agreement that new technology and innovation may help in the future to satisfy all these different needs and can also contribute to the necessary harmonization of the spectrum. And also what is clear that we have to face this task because using radio technology will be one of the cornerstones to reach the sustainable development goals that we have for 2030. Well, what we have today, of course, is in the discussion that we have an ITU, so mainly when we talk about new allocations, for example, we have extensive discussions of what are the spectrum needs of a specific service. So when you come up with some new ideas, of course, you need to justify how much spectrum do you use. And also in ITU, we do a pretty good deal on the system requirements and the system characteristics, for example. And that also gives proof on how efficient the system is designed. And of course, we also need to work on this side and that all together. Of course, in the intensive discussions that we take during the study period, I believe that is the key that we can balance out these different requirements from the different services for the different users and, well, make the best decision at the end that everybody is satisfied. We had the view that technology is key to make a more efficient use of the spectrum. There are certain technology and, let's say, parts and pieces that we need to look into. One is very clear interference cancellation in different ways, of course. What we may also need to look at are the receiver requirements. I believe this is something what we need to have more in our focus to make also receiver more robust. This is important that for the services, if they are incumbent, that they are not get, let's say, some distortion from new services that arise in neighbor bands. So this is something we certainly need to look at. We can work on technologies that do offer more synergy for the same service. For example, you can use broadcast technologies that have a lot of things in common with mobile cellular technologies for the mobile communication and so on, and that makes it easier to harmonize them. Certainly, spectrum sharing is something which we need to look more and more in the future. We see the first implementations being there, like CBRS and DOS, and, of course, probably also do a little bit more on the spectrum sensing that we get more flexible. So if there's spectrum-free that we can use, that we can have this flexibility to use it for new services on a finite granularity. I believe this is certainly something which we need to look at, and all this basket of different technologies and innovations that we can do in the future help us to improve the spectrum harmonization and the spectrum usage. Well, I would say technology can push the limits further out. But at a certain point of time, you need to decide which services will occupy the spectrum, which services will have allocation. So this is certainly something that you need to decide in which technology not directly helps you with. So technology can push the limit, but also the demand is growing. So the discussion and the careful, well, let's say, considerations of all the spectrum needs for what is used, is it safety of life, also which kind of, let's say, characteristics of the spectrums are used for which service the physics, so to say, cannot be changed so directly. So these decisions need to be taken at the point of time where we all have already have, let's say, considered all technology improvements, but certainly we need to take a decision on how to do the allocations and how to use the service in a certain frequency band. I think that can technology not solve alone. I believe these online events was, so to say, the opportunity to stay in contact. And from our experience, I believe that it was a good way to at least keep the contact to people that you have already been working with. I mean, online events, as you may also know, is not the best place to make up the new contacts. And this brings me down that I'm looking forward that we can meet in person again. I think for many reasons, one of them is to make new contacts and also improve the ones you already have. There's nothing better than a physical meeting than the direct discussion with people in front of you. But when you look at the circumstances that we had during COVID times, I think this was the best thing we could do.