Hello everybody. Good morning. All right. So we're gonna kick off the first talk today in the Mezos internals track. Vinod and I will be track leads for today. So today as the first talk we have Jay Guil from IBM and Ben Mahler from Midsphere. They're gonna be talking about hierarchical roles and multi-roll frameworks in Apache Mezos. Going to the details about the multi-tenancy features that were added. Recently added as well as covering some of the future features that are coming up. All right. Can you guys hear me? How about now? Okay there you go. All right so we're gonna talk about multi-tenancy. Okay so obviously what's what's the first thing we need to talk about? What is multi-tenancy? Pretty straightforward it's this idea of having multiple tenants share a single software system. So an example this is you might have something like AWS or Google Cloud where you have two different companies Coke and Pepsi sharing that same system. It's that same platform. If you're a single company you might have a private cloud where you have let's say the engineering department and the finance department sharing the same platform and maybe they don't want to see each other and things like that. I think the more interesting question here is this definition of a tenant. Sometimes when multi-tenancy discussions happen people are specifically referring to you know one specific type of tenant like a company but you know how we want to think about a tenant is really it could be any of these things it could be a company it could be different business units within a company it could be teams or it could be individual employees and so this notion of tenancy is really hierarchical right. You have employees they work within teams they work within departments within organizations within companies and so we want to think of all of these things as being a tenant so right now in in mesos what we target with multi-tenancy is primarily using a single mesos cluster within a single company so that means you know you're sharing a mesos cluster across teams in the company or across employees and and so on. For different companies at the current time we would recommend using different mesos clusters and that's because in that that use case requires extremely strong security potentially stronger isolation if you're not okay with container level isolation you need really good prevention of denial of service attacks you don't want one company being able to affect service towards another company and you have to there's compliance things to deal with as well so it's a really hard problem and we don't recommend it right now but we will make progress towards sharing across companies in the future okay so what are some needs for multi-tenancy of course we need security isolation not just when it comes to making sure that different tenants don't land on the same host or or are using containers or VMs but also in terms of the API right you might want engineering to not be able to see sales depending on what you're doing and so you need good authorization and authentication to do that when you're isolating resources on the host you know not only do you need to do security isolation to make sure that the file systems isolated and the network and storage are also isolated but you need performance isolation as well you want to make sure that one tenant can't negatively affect the performance of another tenant and that goes for all the resources that that exist on the host another thing you need is this notion of guarantees about how many resources you can get in a cluster we do that with things like quota to give you a guaranteed amount and we have fair sharing when you want to go over your quota and we might introduce things like priorities and so on in the future and then you also need really good fault isolation right if one tenant is inducing a failure in some way that failure shouldn't cascade into other tenants okay so there's also some end to end platform needs for multi-tenancy right if you're running a scheduler on mesos either that scheduler itself needs to be multi-tenant or you need some software that helps tenants get a per tenant scheduler right you also need software to manage these tenants that exist in your platform and companies have existing LDAP setups that you might need to integrate with so there's software needed for that and then for compliance you might need things like audit logging and if you're building a platform you might want to charge those tents for how much they're using and there might be things that you need that mesos might not tackle like for example network isolation is often done with software defined networking and that's usually provided by you know a vendor or something like that so a lot of these things that you need for multi-tenancy in a platform are kind of out of the scope for mesos mesos is one piece of this but you kind of need an end-end platform solution like something you can build in-house or an existing one like DCOS okay so we're gonna talk a little bit about a little more in detail about what exists for multi-tenancy in mesos today so just as a reminder the vision for for mesos is that you know it's this kernel of a data center operating system and so one of its core responsibilities is is resource management so resource management means a few things it means sharing resources amongst tenants giving guarantees to those tenants about how many resources they can have providing isolation of those resources and then providing accounting so that you can tell on a per tenant basis how much people are using so as a terminology thing in mesos we don't have the word tenant we don't use that word what exists in mesos is this concept of a role and that represents a consumer of a resource and so that's the mapping here when you think about multi-tenancy in mesos is we use these rules to capture you know whether it's a business unit a team and so on okay so what are some features that exist today well I talked about roles and with roles you can get resource guarantees there's two things in place today one is fair sharing with the RF and the other thing is quota quota gives you an absolute amount of guarantee and fair sharing is relative and we have resource accounting to make sure that you can track how much each role is using in the system and then we have a lot of security isolation and performance isolation I won't talk about them here but you know it's kind of if you want to learn more about that you can attend containerization related talk and then we have authentication and built-in authorization as well so this stuff is built in but it's also customizable so if you want to say integrate with a Kerberos system that you have you could build an integration to that okay so Jay is going to highlight two features for multi-tenancy that we recently worked on so I'll pass it to him all right I'm gonna introduce these newly added feature like namely multi-role framework support and hierarchy roles in more details and as Ben just introduced like we normally will have for example multiple teams sharing one marathon instance and that marathon subscribe with Maysos and these teams will launch all different application services using that marathon but from Maysos point of view it's actually Maysos doesn't know that these teams like because marathon subscribe with Maysos with only one role which we currently support in this case is marathon and so Maysos only knows that okay they are jobs running under the role marathon but actually these teams are from and back end reporting they are all launching different workload or applications so the problem with that is you cannot enforce the like a quota or reservations using with these different teams like front and back end because Maysos simply doesn't know about it I mean one solution to that would be you can launch like single what like one marathon instance per team so that Maysos can and they basically subscribe to Maysos using different roles but obviously you don't want to do that for every team in your organization and you're adding I mean because tenant could be one department or but also can be one individual employee but obviously you don't want to do that for every single one tenant so what we want to do is you can subscribe a framework to Maysos with multiple roles so that Maysos is aware of it and can allocate resources accordingly so different teams using that marathon a single instance of marathon and can be allocated with different resources so you can leverage like quota and reservations that associated to the roles and to make better decisions of the resource allocation and of course you we don't want to implement that in the scheduler which meaning you need to implement all these features in every single scheduler not only marathon but we want to have a unified implementation in Maysos to make such a decision so that's basically the basic idea behind the multi-role framework support and for hierarchical role support it's it's quite straightforward like in the organization is often hierarchical so you don't really have front-end back-end report lead at the same level instead you will have for example front-end and back-end under the engineering department and reports and leads under sales department and furthermore you probably will have like front-end running different workload like Rails and UI and API login in back-end so you see this kind of hierarchical structure should be captured by the roles as well being that we want to oh we want schedulers to be able to subscribe with Maysos using this path like syntax so engineering slash front-end slash UI will depict the UI workload under this tree structure and when we come to a hierarchical role the DRF algorithm reservation quota may be slightly different than the flat ones for example we want to enable this a dedication meaning that if you allocate if you set a quota to the team engineering front-end and that team can be able to further subdivided quota to the different workload rails and UIs I want to split that quota and set quota for my sub teams further and also we want to do the authorization isolation for example I don't want front-end to see the siblings allocation being back-end in this case and again to combine this with the multiple support we want like my single instance marathon to subscribe to different nodes in the tree being like all these different path paths so that still in the the whole tree can use single instance for marathon but to manage all the resources allocated to the whole tree and the DRF like it's kind of straightforward because previously was flat so you just compute the fair share of every node and then sort them and allocate the new piece of resource to the node with lowest share in the hierarchical structure you simply calculate fair share recursively at each level so given the resource allocation in this case we will calculate the fair share the result will be like this like an inch department will take like 20% of the resource comparing to the sales department and then you recurse further down to the end front-end and end back-end which take like one-fifth and four-fifth resource of inch department and then you recurse down like you find the node with lowest fresh air so in this case we go to edge first and then edge front-end so we will allocate the new piece of resource to that node so this is like this is quite straightforward when you come to DRF in the hierarchical manner and the hierarchical reservation will be like given such a such a tree and we said we reserve resources to edge like 100 resources in this case and that reservation is actually shared in the tree so it's shared among edge, edge front-end and edge back-end something like this and then we introduce the semantics of refinement so you can refine the when edge front-end receive the resources from the reservation of edge it can further refine it so basically called reserve API to reserve that piece of resource to itself so that cannot be used by the back-end as a simple sibling and when it's done with it you can just call unreserved API to return that resource to the edge which is a pyramid role and I want to emphasize that when you reserve a resource to edge it's a shared among the subtree what does it mean basically for example you have a situation like this you have edge and you reserve resource 100 to it and you have a framework X subscribing to it and the framework X will get all these 100 resources of course but then you decide to divide your edge team further to edge front-end and edge back-end and you subscribe from Y from Z to these two newly added teams then you don't it's not guaranteed that framework X is getting all these 100 resources anymore because 100 is shared among the subtree so exactly XYZ are sharing this reserved resources so it's so basically you don't get 100 for X anymore but 100 is shared by XYZ but normally you will have like we said before one single instance for marathon instance subscribing to all these different roles and get all this 100 and schedule jobs based on their their roles so this is hierarchical reservation and when it comes to hierarchical quota will be kind of similar to reservation you can set a quota to edge again that's shared among the subtree like this and I want to again emphasize that you can dedicate quota it's like refinement but actually you set quota to a sub role of edge so that guarantee is of dedicated to front-end so the back-end won't get front-end it's guaranteed to get 40 out of 100 of the quota in the subtree but this introduce the semantics that you should always set the quota to the parent role because that's shared among the whole subtree and then further you set quota to the sub role so basically this in the subtree the quota is 100 you cannot have the the some of the sub roles like 40 plus 70 you cannot that exceeding the quota of the parent role and therefore you should always set the quota for the parent role and then set quota to the child role so basically you cannot do this because quota is by default is zero and you shouldn't you cannot it's it's allowed to set quota to end front-end before setting quota to edge and similar to the reservation before if you have framework X subscribing to edge first and it's guaranteed to get 100 quota resources and then you decide to add more sub teams to edge so you and then at this point it's not guaranteed anymore that X will get all the 100 resources because that's shared among the subtree so XYZ again will get the quota 100 resources and so to summarize it if you still want the old behavior like you want to guarantee some application to get all the like reserved resources or quota resources you should always subscribe that framework to a leaf role and set quota or reservation to that leaf role and so to basically to get the same behavior as previously you did like to guarantee a certain amount of resources to that application and I'm gonna hand over back to Ben to talk about the future work and roadmap all right so where is this stuff right now multi-role framework support shipped in 1.3 of mesos for hierarchical roles it's still a work in progress we're playing to finish it in 1.5 and 1.4 some of the work is done already but quota will not work as we showed just now so it's not recommended to use it and then you know after we finish this stuff we need to integrate this into multi-tenant frameworks like marathon or Aurora and so on and then you know any end-to-end solutions will need to integrate this stuff as well so I wanted to also talk about some upcoming work that is being worked on already and the first is revocation and then and the second one is a priority tiers okay so you know originally when you know in the past when we introduced research allocation to mesos what we did was we performed weighted DRF to fairly share the resources amongst all the tenants in the cluster and that's the only thing we did and we did this in a non-revocable way which means we can't take these resources back once they're allocated and then later in time we introduced this notion of quota and what this did was it made our single phase allocation become a two-phase allocation the first phase was well let's try to satisfy everyone's guarantees that we gave for quota and then with with what's left let's fairly share that as we did before and so one of the problems with this is that in order to guarantee everyone gets their quota we have to make sure that we leave enough room if they're not using their quota right now to satisfy it at a later point in time if they do want it and so this headroom here is essentially unutilized and there's no way for us to give that out without breaking our quota guarantees that we told people so this hurts utilization and what we want to do to solve that is introduce this third phase of revocable fair sharing where what we do is we can allocate all unallocated resources as revocable and that lets us take those resources back if we need to give it for quota to someone ideally you know in retrospect what we had originally done was introduced just quota and revocable fair sharing but this middle layer here would serve as a backwards compatibility layer for people who expect the old behavior and we would allow operators to to confine how much of this can happen or turn it off completely so that's revocable resources at a very high level and another thing that's been discussed is this notion of priority tiers so for priority tiers what this means is introducing essentially priorities in these particular allocation tiers so for revocable fair sharing what priorities means is that a higher priority tenant can revoke resources from a lower priority tenant whenever they like for quota since it's not revocable the only thing that priority gives you there is like first rights on the resources so if there's an outage and you need to get your core services up those higher priority services can get the resources first and stay running and the other lower priority quota tenants will not get their resources so that's also priority tiers at a very high level and so just just a reminder you know multi tenancy is is really a very multifaceted problem you really need an end-to-end platform that's going to give you you know all the pieces of this puzzle that you need and Mesa's is just one core piece of that puzzle that provides some multi tenancy primitives and that's that's all we have we just wanted to put this up here to thank contributors that have worked on this stuff so far looks like we have quite a bit of time so if you guys have questions we can take those I guess I'll run this microphone around I'll repeat it so it's on the stream I guess okay the question was do I know which version of marathon or DCS support multi role it's none of them do right now it's being worked on at the current time so it's possible the next release we'll have it I'm not sure but it's it's still being worked on so the question was how to transition to using harker role as part of the multi role work we also allowed frameworks to update their set of roles at any point in time currently you have to resubscribe to do that but we'll also add potentially a call so you can do this without resubscribing to Mesa's yeah so the question is about this non-enrollable headroom that I mentioned I'll go back to it so if I drew a line oh you can't see my mouse of course if I you know this dotted line at the top here of the of the headroom is the sum total of all the guarantees of quota that we that we have in the cluster and this middle line here is how much people are actually using of their allocated of their quota right now so if I asked for 10 there would be 10 total here and if I only used five there'd be an extra five that is left unused because we need to make sure that if I later want that extra five that I got a guarantee for I can get it does that make sense does I answer your question or okay we can chat after to you have more questions the question was do I have any examples of of quota and priorities definitely not for priorities it there's no design at the current time there's only been discussions there just there's a design being worked on for revocation as well and then for quota the design I think Michael is actually working on that so I don't know if we have examples right now but there's going to be a design published within the next week or so for the quota part yeah right okay we did that so the question was can can we update the UI to show resource for usage by role as part of the multi-year-old work we did update it so I don't know if you're running the latest version or not but yeah that's the plan is to display this stuff there's a new roles tab in the UI where you can see a breakdown per role of how much is allocated what their quota is and so on that is in it's probably 1.4 oh which is not yet released yeah but it's it's almost released question is what's the timeline for revocation and priority tiers there's no timeline for priority tiers the timeline for revocation I would say is I mean the design is being worked on right now 1.5 will be in two months is that correct yeah so that will be a little tight I think it might be maybe 1.6 or something if I had to guess how cool quote I think it'll be 1.5 I would should be yeah sure do you want to do it or do you want me to so the question was essentially just to clarify a little bit about how this hierarchical reservations work and yeah so I'll I can just reiterate what Jay said and and make sure that you understand so previously I wouldn't have these children here right I would just have engine engine and so when I made a reservation to edge of course only the end roll gets to use that now when you make a reservation to end you're just saying that the entire edge tree is reserved 100 resources so we'll make sure Mesa's will make sure that 100 resources goes to that tree but those resources are going to be shared amongst the tenants in amongst the different roles in that tree now I could I could refine that reservation I could be and front end and I could get some of it and I could further refine it to specialize it to be only for and front end that's this notion of reservation refinement so I could guarantee that like okay I got some of it and now I'm binding it to me so that it can't be shared with all the rest of the edge tree right now anymore and if I unbind that it'll go back to and I see you nodding so I guess it's making sense so far and then what else to keep in mind yeah I think does that is that writes it yeah so in this case like if you had something that you wanted to run at edge still if that's something you wanted to do we would recommend that you run it at edge slash default or end slash whatever you know a specific role to make sure that you can bind it to that thing and it can't go to other um you don't have to reserve to the parent in this picture you could reserve it directly to your leaf role but if let's say an operator said I want to make sure that engineering gets this whole machine by reserving the resources like let's say you have a public machine right it's exposed to a public network and you want to reserve that for only the public network things you could do that and it's then shared amongst all the public network things and they can reserve portions of it is that it that that's kind of one use case that you can imagine yeah yeah I think to kind of add an additional point to what Jay said is you know if you want to use um if you want to use reserved resources and you are expecting those to go back to you you should refine it to your leaf role so in this case like if I'm framework why and I get some of this engineering resources but I know that I I'm running an engineering front end thing I shouldn't assume that those engineering resources are going to always be reallocated to my engineering front end thing I should refine that reservation and make sure that it's guaranteed to come back to my leaf role yeah I mean we definitely looked at both ways of doing it the disadvantage of the other way of doing it where like it's bound to only end here is that you need to two kinds of reservations you need one flavor which is a tree like reservation and you need another flavor which is bound to that internal node in this tree and so I think when we were designing it we wanted to just simplify that and say that all reservations are bound to the entire subtree that they are made on and so of course what that means is we have to tell people use a leaf role if you want to make sure that it's it's you know guaranteed to that particular tenant yeah so the the question was is a reservation tied only like what single resource like cpu or can you make a reservation to multiple resources yeah so a reservation is on a bag of resources you can it can be a collection of any particular resources together yes oh this is 100 units of some resource of course in reality this is going to be a multi-dimensional reservation it's going to say 100 or like one cpu two gigs of memory 10 gigs of disk maybe these maybe these ports that would be a single reservation which is on a particular agent and there could be many of these across the agents so keep in mind that a reservation is bound to one agent it identifies exactly these resources on this agent whereas quota is globally I need this amount that's the distinction between a reservation and quota yeah so this the the reality of this picture today is that if you're a tenant in phase one you're not going to get any resources in phase two and vice versa so um that was because we didn't allow this that was because the intention was to move towards this model where to burst above quota you had to use revocable resources yeah so um in this what we might want to allow in this picture here is exactly what you said where like we might allow a tenant to burst over their quota using non revocable resources but the current implementation doesn't allow that so that might be something that we allow in the future when we do this work uh I don't I don't the question was do we have to do that after we do revocable by default I don't think so no but it's just that's what the current implementation is yeah yeah I think it's we we could do that today we just we just also need to make sure that we don't we don't like the the difficulty there is today if I get non revocable resources I don't know if it's from my quota or from the fair sharing that's happening um and so we want to improve that as well we want to be able to tell someone like hey here's the quota for your tenants in your scheduler and so with that knowledge you know what's going on um and here's maybe how much they're allowed to burst over their quota or how much they're allowed to use like what their fair share is for vocal resources as well so we want to give all that information to the scheduler as part of this work um yeah yeah I mean today quota is not very usable and that's what we want to improve yeah any last questions going ones no okay thank you


Five more people have died from carfetnal-related overdoses, bringing the total amount of deaths up to 10 in Hennepin County this year alone. The opioid is 10,000 times more potent than morphine and 100 times more potent than fentanyl. The drug is intended for use as an elephant tranquilizer and for other large animals. It can come in several forms including powder and spray where they could accidentally be absorbed through the skin or inhaled, potentially harming first responders and law enforcement. The deaths happened in February, March and April, but a specialized lab had to confirm the test results. If you've enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland Public Television.


The reason why the narcissist is never happy with you. The reason why they are never satisfied with anything you do for them. The reason why they always have something to complain about. The reason why they always demand more from you while never appreciating anything that you have already done. When the narcissist first targeted you, everything seemed to be fine. They didn't demand too much from you. It was like they were trying to do everything to make you comfortable. It was like they were trying to accommodate you. But this is just part of the manipulation. This is what is known as love bombing. They use it to lure you into the relationship. By making you think that things are always going to be this way. But really, they're just setting you up to be their caretaker. They're setting you up to be someone who is responsible for them. Because that's what they were looking for when they targeted you. They just disguised it as though they were there to support you. But if you look back at the time you have spent with them, you will always find that they didn't do a fraction of what you did for them. You were always the one who was taking them places. You were paying the bills. You were funding family vacations. You received little or no contributions from them. They just expected and demanded it from you. Although the character they displayed to you at the beginning is very different. They make you believe that they are about reciprocation. They make you believe that they don't want you to do too much for them. If you offer to help them out with something, they may even refuse your help. Or they will offer to do something for you in return. Because they want to portray this image of them being someone who doesn't need you. They want to make it look like you're the one who needs them. Because that's going to be the reason why they need to remain around you. They're eventually going to make you doubt your qualities and abilities.  They will make up anything to support their false narrative. But whatever you're seeing from them now, that's who they really are. And that's what they were held from you in the beginning. Because if you had known that you were dealing with someone who is greedy and self-absorbed, you never would have accommodated them.  At least until they can find a way to trap you. That's when the mask will come off and then they will show you who they really are. They will show you why they targeted you in the first place. They will show you what they want from you. When the narcissist targets you, there is always a hidden agenda behind it. They always have a hidden motive. They come to you saying that they just want to help you with something. But as time goes by, it seems that you're doing more and more from them. While they're doing less and less for you. While complaining that you're not doing enough. And at this point, it should become clear that you are their caretaker. They see you as an object that exists to serve them. That is the purpose that they have assigned to you. Which is why whenever you deviate from this and start focusing on yourself. They can't stand it. They demand all of your time, attention and money. Because they feel entitled to it. When they targeted you, they made this silent agreement in their own minds that you were unaware of. That their presence alone requires some form of payment or exchange from you. And they expect you to adhere to this agreement. Which you have no knowledge or awareness of. They expect you to fulfill these obligations that you never agreed to. And that will never explain to you. Which is why they can never be happy with you.  You agreed to satisfy that every whim. You agreed to be at their beck and call. So you should always be ready to obey their orders and commands immediately. You should be completely compliant and obedient to them. You should cater to their every need. That's just how they see it. But naturally, as time goes by, you are going to run yourself into the ground trying to please them.  You're going to want to spend more time alone. You're going to run out of energy and money. And this is when the problems begin. Because if you're dealing with a narcissist, all they really want from you is your attention and money. They want to use you. So that they can feel powerful and in control at your expense. So that they can feel comfortable. But it's never enough. They will take you to the cleaners. They will ruin you. And when it's all said and done, that is when they will leave. They're only going to stick around as long as they can get something out of you. Because that's what they targeted you in the first place. The reason why the narcissist is never happy with you has nothing to do with you and everything to do with them. They're not happy with you. Because they're not happy with themselves. They are emotionally unstable and they lack emotional depth. They cannot regulate their own emotions. Which is why you will often experience these long monologues from the narcissist. They can talk about their own feelings or their own situation for hours at a time without requiring any input from you. Because they don't care about your feelings or perspectives. They don't care about how you see it. All they care about is themselves. They are self-absorbed and they lack empathy. Which is how they're able to talk for such a long time about all their problems. Because they have no empathy for you. They can't put themselves in your shoes. They can't share your experience to see how that would make you feel. And even if they could, they wouldn't care. They're only concerned with their own feelings and needs. And they use other people to regulate their emotions. But it's never enough. They're like a bucket with a leak at the bottom. Whatever you point to them will always leak out. It will never be enough for them. They will always feel dissatisfied. While never taking responsibility for their feelings. But they see their feelings as facts. So if they feel dissatisfied. That must mean that you did something or didn't do something. To cause them to feel that way. Which means that you must do something to resolve it. You must do something to make them feel better. It's like dealing with a small child that has hurt themselves. And now they want you to make it better. They want you to patch up their wounds. But this is a wound that will never be healed. Because they attribute it to you doing something to them. When it's something that they need to resolve themselves. And this is why the problems never end. This is why no matter what you do. They will never be happy with you. Because they are not happy with themselves. And they're not going to take responsibility for their own emotions. They're not going to take the necessary step to heal themselves. Which means that you are always going to be held responsible for their feelings. You are always going to be held responsible for where they are in life. Thank you for watching and I'll talk to you soon.


Hello, my name is Vino Budavis. I'm the academic director of the Bachelor of Entrepreneurship and Business Innovation at Tilburg University. In the coming few minutes, I want to share some information with you that I hope will help you choose the right program, or in other words, I hope to help you deciding whether the Entrepreneurship and Business Innovation Program at Tilburg University is really the program that fits with what you want to do and who you are. Choosing a degree program is not easy. There are many different options and I can understand that it's hard to choose. At the same time, you should realize that you are committing to a three-year program. So if you choose, you really should know for yourself that this is what you really want to do. And that is not easy because you can look at the content of courses, you can look at the type of projects that you will have to do, but still it's difficult to decide whether this is really a program where you will fit in. And I hope that this short presentation will give you some perspective on the type of students that at least we hope to enroll in the program. Now, when you would start in August 2020 in Tilburg, we hope that you bring those qualities that you see here on the screen. We hope that all our students have a bias for action, that they are creative, rebellious and self-driven. We will not be able to make you a learner. We can provide lessons, we can provide classes, workshops, but you are the learner. You're in control of your own personal development and being self-driven is a tremendous help. But also for being an entrepreneur or an innovator, being self-driven, being creative, being a little bit rebellious is really, really important. At the same time, we expect our students to be curious and analytical. We expect them to have well-rounded interests and life experiences. And not that you should have special life experiences because someone gave you that experience. No, no, we're interested in the experience that you gave to yourself. We expect our students to be open-minded, interested in exploring new stuff, curious. We expect them to be outward-oriented, meaning that you'd like to engage getting to know new people, both in class, your fellow students, but also outside class, on campus and beyond campus. We expect all our students to have a strong social engagement. In our view, we don't do entrepreneurship. We do not embrace innovation, just to fill our wallets and to make money. Know that the idea should be to create value for society, to have a positive impact.  We don't want students in the program who would already be happy if they pass a course. We want you to try and excel, be better than others, to get the best out of yourself. As a result, we do think that most of the people we will have in class will recognize themselves, if they compare themselves to the classmates at high school, as people that were considered to be different, crazy, a little bit of misfit in the group, rebels, people who are doing different things than the average, the ordinary, the normal high school students. We think it's those qualities that will help you to change the world, to really have a positive impact through your initiative, through your creativity and through your ability to innovate valuable solutions. Okay, that's nice and easy to say, but how do you know whether this really applies to you? Because you may want to recognize yourself in this list, because you think it's an interesting program and you would like to tell yourself, well, I should do this. I really believe that you should do some soul searching. And sometimes it helps to ask others what they think about you. So if you would talk to your friends about what we expect you to have for this program, it could be that they say, well, that's absolutely you. It could also be, well, no, that's not really something that I would think that you will bring with you. Well, then perhaps you should not enroll into this program and choose one of the many other options. But if the people in your neighborhood, if your parents, if your friends would say, well, if there's anyone that has these types of qualities, it's you, then you may feel confident that you fit in. And that when you then start in August 2020 in Tilburg, that you can expect the whole group to be like you, all eager to learn to become successful as innovators and entrepreneurs and to have this positive impact on the planet. Choosing a degree program is a major commitment. It's also a choice that is not easy to change once you have made it. So it's really important that you know that the choice that you are about to make is for you the right choice. If we can help in making that choice, then please reach out to you because we want you to be successful, irrespective if you end up doing our program or choosing another program at Tilburg University or somewhere else. Thank you for your attention.


How's it going everybody? I'm Wade Lewis I'm James Denton and we're here for the no valid host was found trouble shooting trace backs and other common failure scenarios So as Wade introduced himself already, I'm James Denton principal architect for the rack space private cloud team Wait those I am a rack space private cloud architect as well. Yeah, we we work in the support team for the for the product What we're here to talk about is troubleshooting some common open stack issues includes trace backs some common Nova issues and common neutron issues After the presentation we will upload these slides to slide share at the link you see there We've got a lot of slides. We've got a lot of ground to cover so we apologize if we talk a little quickly But we just want to make sure that we get through everything and if we have some time we'll do some Q&A Great so I think we all know that open stack is a complex system. There's a lot of moving parts with limited visibility to problems via the API The old adage of turning it off and turning it back on again doesn't really work here anymore So you'll find yourself in the weeds troubleshooting the stuff on the infrastructure nodes, you know when you have problems So what is a trace back if you're not familiar with open stack and you're coming from say a vSphere ESX environment then you may not be too familiar with them But if you run open stack you've probably seen them before basically a trace back or a stack or stack trace is logged or it's an output or a log of an exception that was captured during the execution of a program When it's caught you can kind of go through and see the functions within the app that have triggered that exception and that'll help you discern what may be the case with the issue or the error that you're receiving Decidering it sometimes if you're new to this can be a lot like reading the matrix we've all seen that before but in this case the trace back looks very similar to this And if you don't rotate your logs or you're not doing so you might want to start now because it will fill up pretty quick The tips on reading a trace back so you know you get this trace back and you're like where do I start the easiest ways to start at the bottom and you can see here just in this example my SQL wasn't available So that's a great place to start if you're not familiar just go to the bottom take a look at it and see what you're working with Yep slides available again We have slides available Okay so let's move into Nova No valid host has found what the heck does that mean a lot of times that error is very ambiguous So we need to kind of get to the bottom of that and see the reasons for this particular error and what the case may be Likely reasons for this conditioning a condition occurring are that there really aren't any hosts available for whatever reason Networking issues on a compute node lack of resources disk RAM CPU etc The good news is these days I haven't actually deployed Liberty yet but with Kilo Juno and Ice House the reporting is a lot better so if you're on an older version of OpenStack Upgrading may help that situation a lot if you run into this particular error it may be more detailed Alright so let's get this thing up an instance you go to spin up an instance and you're in an error state The VM clearly has not failed to launch but a few things we notice here other than the error state is that you are given a network address So you have an IP here and that tells us at least that the neutron API is functioning properly to get you one of those an IP address that is So to get a little bit more information you know you hop into Nova show and you see well there's a stack trace there But among the stack trace as well as we know that we have a libvert instance ID that's good And we know that we have a compute node here as outlined in the red boxes So that being the case if you know it's been scheduled to a compute node the easiest thing to do is to hop over the compute node And start taking a look at some of your Nova compute logs and your neutron agent logs And in this case when we created this particular error it logs another stack trace yet again in the Nova compute log And this time we're getting a Nova exception unexpected vif type equals binding failed And so if you're a new operator you're like I don't know what this means I've been doing Linux for quite a while 15, 16 years now And the first time I saw it was like okay not sure what this is But basically when Nova creates a virtual machine it's got to plug each of the virtual network interfaces into a virtual bridge And the virtual network interface is known as a vif Okay so that's kind of a prereq you may need to know that you can google it usually that info is out there But Nova uses drivers that are specified in the Nova conf to configure that interface into the virtual bridge And when Nova is unable to interface with the network agent and properly plug that port in the vif type is set to binding failed And that's where you get that error So now that we kind of know that it's a network issue the next place to look would be your network logs In this case we're using the Linux bridge agent so we're going to take a look there And what do we find yet another stack trace But this time at the bottom of the stack trace you see the error 19 no such device So that's a little bit better you know as Linux administrators or Unix administrators that's not that uncommon to see across different apps and scenarios But the problem is we don't know what device it is it could be any type of device or their block devices network devices you name it So what is the device Alright now we're going to issue a bit of a warning James and I are operators and we like to you know dabble in python but we're not neutron developers or neutron coding python experts Go easy on us Yeah so we go back to the trace back that we mentioned just a minute ago that shows a no such device error And if you look at the trace back you see a function called get interface Mac from the utils.py file itself And get interface Mac seems pretty self-explanatory it's trying to get the MAC address of this interface but we don't know what that interface is So we dive into a little bit and you can see this code here it's pretty basic Okay and you don't need to know a lot but you know we took a look at it and said there's no error trapping here Let's go ahead and throw that in and see what we get back So we made a quick append to the file saved it through a little bit of exception handling into this it was literally three lines And restart the Linux bridge agent to see what it would log after that And by adding some exception handling we can see now that it's complaining about interface EHT2 So we both looked at it and said okay well that's dead simple someone fat fingered the interface ETH2 So the Linux bridge agent doesn't know what that device is it can't do anything with it and you get that error So we go into the Linux bridge configuration file and in this case the ML2 configuration file And we can see that there it is in the underlined section EHT2 and that's obviously incorrect So we'll make that change restart the agent and see what happens and in this case everything worked properly But what do you do when you do a nova show after you launched an instance and there isn't a compute node there There isn't a livevert instance ID or maybe there is in this case but you have a stack trace again so that's good But again in this particular case I'm not sure if it's kind of hard to see from there But it's a no valid host you get a 500 return from the API and you don't know what it is and now you don't have a compute node to actually go and look at So you're kind of like well now what well if the scheduler can't actually schedule the instance Then you probably should check the scheduler logs there the conductor logs and see what you've got And in this case it's a pretty easy fix Take a look at your agent or your excuse me your services and make sure that they're actually working And the top pass there with the red box we see that in this case we have load balanced API nodes And when it checked in it passed everything was fine it's labeled as up it's working But when we do another service list specifically a Nova service list to get an idea of what the service status looks like We can see it's down and we also know there's a big difference in time so right away that's a problem So what could cause that well when agents check in the database is updated with the time of that check in And other services such as the scheduler take that time and to determine if the service is available The scheduler determines the availability of a host by comparing the difference between its local time And its last seen time of the compute node by default that can exceed 60 seconds and if it does the services mark is down So in short we've got NTP NTP NTP listed there the time was off the scheduler determined the device was down And didn't schedule a host or excuse me an instance to the host so in this case the Nova compute service actually was down There wasn't anything more than just that particular scenario So when in doubt make sure across your node and all your nodes that the time is accurate with NTP or any other method So let's move on to neutron and I'll let James explain some of the scenarios we've got there Great thanks Wade yep So neutron itself is composed of you know many different services and agents that are responsible for constructing and maintaining the virtual network So you can see here failures can occur at any of those points either the DHCP agent L2 agent L3 agent just to name a few So let's start with the DHCP agent Now when instances are created Nova will create a port through the neutron API and an IP that's assigned to a port You know that's a statically assigned IP based on the subnet that you've you've specified The DHCP agent itself is responsible for creating a network namespace in a DNS mask process inside that namespace that is responsible for providing DHCP services to the network When things when the agent fails it can result in failures to get it get an initial lease or a failure to renew a lease Now when you when you create a new subnet with neutron create a port again DHCP agent updates DNS mask with those network attributes and it stores that file in a host file here at varlib neutron slash DHCP slash Network UID slash host and that's a file that's past the DNS mask so the DNS mask knows What IPs are eligible for leasing Here we can see that we have a MAC address that corresponds to a neutron port The host name of that particular instance and the IP address When an instance sends out a lease request DNS mask will hand out that lease and then update its active lease database in memory By default DNS mask will store those messages in or the DHCP cycle messages in varlog syslog or varlog messages depending on your distribution Here we can see that we see the DHCP discover from the client The DHCP offer which is a server presenting an IP address for use The client will turn around and send a DHCP request and then the server will acknowledge that request If you have issues obtaining an IP start with packet captures on a couple of different devices So you want to start on the tap interface of the compute node Tap interface of the instance on the compute node to verify that the instance is actually sending DHCP discover messages Then you can work your way down to the bridge interface on that respective node and then the physical interface You'll turn around and run the same type of captures on the network node with the DHCP agent To verify that the messages are actually making it through the physical and the virtual network stack When you run a DHCP dump or a TCP dump you want to make sure you're listening on UDP port 6768 You should see the full DHCP cycle of the four messages Unless you're running overlay network types that those packets may be encapsulated in a VxLanarGRE header So here we have a working example, right? In the first screenshot on the top I'm performing a TCP dump on the tap interface of an instance And I can see the full DHCP cycle there In a non-working example running the same packet capture I'm only seeing my instance sending out DHCP discover messages that go unanswered And in this case I had actually moved an interface out of the bridge to force of failure Not a likely scenario in production, but it does show that if there is an agent issue and it wasn't able to connect an interface This is the type of output that you might see when you perform a packet capture Great, so now that we're all good with DHCP Let's talk about a live bug that could be affecting a lot of you without realizing So let's say you spun up an instance, you know, no error You realize that the instance is not available Chances are that DHCP is not working and you're troubleshooting So we ran a packet capture on the interface And we see that the actual DHCP discover or the DHCP renewal is making it to the agent But the agent is sending a DHCP NAC packet What this likely means is that the DHCP agent was restarted and the active lease database was deleted So whenever an instance receives a DHCP NAC, it drops its IP off its interface and restarts the entire DHCP lifecycle again Which will likely result in a brief, you know, momentary downtime while that happens This issue was addressed in a patch However, there are some effects there when you're running multiple DHCP agents that this patch, you know, causes some issues with So the patch enables a flag for DNS mask that says DCP authoritative It expects one DHCP agent in the network And when you have more than one, what happens is when an instance goes to renew its lease Since the DHCP request packet is broadcast All the DHCP agents are going to see that packet The one that originally provided the lease will go ahead and renew the lease And the others will reject it So what ends up happening? Same thing, instance drops its IP and starts the whole process over again So right now there's a patch in Liberty that should be backported at some point Where Neutron will pre-populate the lease database when the DHCP agent is started So just like the host file where the agent pre-populates the host file so that the agent knows what IPs and MACs are available for leases It does the same thing with the lease database so that you never have to worry about, you know, getting dropped out of memory Alright, so let's talk about some L2 agent issues The L2 agent was responsible for programming the virtual switching infrastructure And also applying security group rules to Neutron ports A failure of the L2 agent can result in a lack of instance connectivity, security group issues, and an immediate error state during a nova boot So when you're troubleshooting some L2 connectivity issues Here we have an example of an OVS environment where we have one compute node, one network node And there's some interfaces there with stars on them Those are where you're going to want to run your packet captures to make sure that the traffic is actually making it through So we see we traverse tap interfaces, QBR bridges, some v-th pairs, integration bridge, so forth Physical switching infrastructure and then back up through the network node One thing you want to make sure is that in an OVS environment you have what's called the integration bridge And all of your tap interfaces for instances and some of your network agents, they're going to plug into this single bridge When you have different networks, each network gets its own local VLAN ID And that VLAN ID is specific to that node You'll see here in the example that for a particular port here has a VLAN tag of 2 That VLAN tag of 2 corresponds to some real segmentation ID for a network that was created by a user And then there are flow rules that exist on the bridge that are going to translate that local VLAN to the physical VLAN or the overlay segmentation ID If you ever see a tag missing, you'll want to restart the OVS agent on that node because every port in the integration bridge should have a VLAN tag If you ever see a tag that says 4095, 4095 is what they call a dead VLAN and that's sort of an error condition for that particular port So in this case, I had an instance that was on a compute node It was in VLAN 2, it's just the one I just showed you in the last screen I did a neutron port delete and I deleted that port out of the database Immediately on the compute node that port went into VLAN 4095 You can kind of consider that a security mechanism If that instance was still alive on the compute node, still plugged into its bridges But now because I deleted the port, neutron has automatically moved it out of that local VLAN into a dead VLAN So that there's no jeopardy of any sort of security issues Some useful commands you'll want to run when you're troubleshooting or just administering OVS is the OVS vs CTL show command It's going to give you a high level of view of the virtual bridges on that node And it'll also show you the local VLANs particular to that node You can run OVS, OF, CTL, dump flows in the bridge name It's going to actually show you the flow rules on that respective bridge that you've specified And the flow rules are responsible for manipulating traffic and help determine how that traffic should be forwarded across the network You'll often see that that local VLAN ID there will usually be a flow rule there that translates that local VLAN ID to the data link layer, the real VLAN Or forward it on to say the tunnel bridge where there's some rules there that translate it to the segmentation ID of the overlay network And lastly we have OVS, OF, CTL show and the bridge name That's going to give you a port level view of the bridge that you've specified It shows you the port IDs known by OpenVSwitch for every port that's plugged in You'll also see these port IDs specified in the flow rules themselves Great, so now we have a Linux bridge environment, one network node, one compute node The interfaces with the stars on them are where you can run packet captures if you've experienced some connectivity issues Here we have our little dot moving through there In a Linux bridge environment you're going to have a bridge for every network So rather than OVS having a bridge for that particular host, Linux bridge is going to have a bridge for every network that you create In this example here we have network A on top, that's a VXLAN network The bridge name starts with BRQ and then the ID that you see after that actually corresponds to the network ID, the neutron network ID Inside the bridge we have a tap interface that could correspond to DHCP agent or a router or an instance Then we have a VXLAN interface, VXLAN-48 The 48 corresponds to the segmentation ID of the VXLAN network Network B, same thing, a completely different network with ETH2.33, so that's ETH2 with VLAN33 And two tap interfaces that correspond to instances Some useful commands when you're working with Linux bridge, BRCTL show is going to show you a high level view of the virtual bridges on that node Bridge FDB show is going to show you the bridge forwarding database So it's useful for knowing how a MAC address is accessed You can consider it akin to say the CAM table on a physical switch And then IP neighbor show will show you the ARP cache on that node Great, so now here we are again, VIF type binding failed We saw this error in the nova example that Wade talked about So you usually see this when you're booting an instance or attaching an interface It's typically the result of a neutron misconfiguration or an agent issue And it's not just limited to instance ports as we'll show in the next slide So I apologize, it's a little bit hard to see But here we have a DHCP port and an L3 router port that are both in a binding failed status And what happened here is that a tenant created a network They enabled DHCP on a subnet and they attached that network to a router And what they realized is that their instance was not able to get an IP And when they assigned an IP manually inside the console, they were not able to hit the gateway So taking a look at the L2 agent log on the network node hosting those two devices We can see that there was an error in the log that says VXLANs enabled A valid local IP must be provided So in the ML2 configuration file when you're configuring interface mappings on the host You'll also configure VXLAN information like the VTEP address for that node And when the Linux bridge agent starts, it takes that IP and it tries to configure point-to-point tunnels between the hosts And when the IP is either wrong or it's not applied to an interface, you're going to see this error So what happens is that the neutron itself is not able to interface with the agent to determine how to plug in the DHCP or the L3 ports And you get a binding failed error So correcting that problem restarting the agent should result in successful port bindings Now one of the problems we have here is what do we do about the existing ports So our tenant is still having issues And what do you have to do to resolve this And it's probably bugged and we should report it We're not there yet To fix a router port what you would need to do is unschedule the tenant network from the router or the L3 agent Reschedule the tenant network to the L3 agent And that in turn is going to create a new port that should bind correctly The DHCP port is a little bit different When you unschedule the network from the DHCP agent, neutron actually puts the port into a reserved status The reason it does this is if you were ever to unschedule your network from your agent And it were to delete the port like it used to And let's just say that you spun up a thousand instances and exceeded your subnet pool Now if you were to schedule that network back to the agent there would no longer be any IPs to assign to that DHCP server So what you would have is an issue where you can hand out addresses So you can delete the port here Reschedule the tenant network to the DHCP agent and then create a new port I should add that it goes into reserved status When you reschedule the network to the agent it does not update the binding status So it remains in a failed state which is why we need to delete the port here So when you're troubleshooting the L2 agent a couple things you want to do Make sure that the respective L2 agent on the host is configured properly And is actually running and not in a constant restart Upstart and if there's a failure we'll continue to restart the service You'll notice the PIDs are changing so something to watch for You want to make sure that open vSwitch is running Most of the time OBS is not running on that host and you try to schedule an instance to that host You'll get an immediate failure so that something will look at there And then the agent logs are stored in var log neutron on that host Here we can see neutron slash plug-in Linux bridge agent dot log Alright last but not least the L3 agent So the L3 agent is responsible for creating network namespaces for each router that a tenant creates That router in turn provides routing services between tenant networks And it also provides NAT services to an instance When it fails you'll find that you're not able to route traffic And floating IPs themselves may be inoperable So inside the router namespace is running IP tables And for every floating IP that gets created there's some corresponding IP tables rules for that floating address Sorry it's hard to see but here we have a couple arrows that are pointing to IP tables rules specific to that floating address There's a D NAT rule, two D NAT rules and a SNAP rule for that All other traffic that isn't actually handled by a floating IP is going to be automatically sourced and added by the router If you have issues with floating IPs not operating properly You can actually go into the namespace, make sure those rules are there And if you want to you can add some by hand just to verify that the action will work properly But as soon as you restart the L3 agent all of those rules will get wiped So when troubleshooting the L3 agent you want to make sure that it's actually running again And you want to make sure that it's configured properly Perform packet captures within the router namespace on the tap interfaces inside the namespace To make sure that traffic that you're trying to reach is actually making it through the virtual switching infrastructure and to the namespace And if your floating IPs aren't working properly make sure that the IP tables rules that should exist for that floating IP are actually there And you can also check the log and farlog neutron to see if there's any stack traces being reported A little more neutron here A couple of things you want to be aware of when you're running overlay networks like VXLAN or GRE If you're running the default Ethernet MTU which is typically 1500 on your physical switch ports Which in turn is usually on the virtual switch ports as well And you use an overlay like VXLAN There's some headers that are added to each packet that can cause you to exceed the MTU and that traffic to be silently dropped So normally this will manifest itself in issues connecting to instances via SSH And when you enable verbose logging there you'll see that you may have the handshake but in the middle it just hangs And what you can do is update your subnet to pass DHCP option 26 Try dropping the MTU to about 1450 and then rebooting the instance More than likely that's going to fix that If an alternative to lowering the MTU on your instance may be enabling jumbo frames on the VTEP interfaces of each host and your switch ports And don't forget security groups too So I can tell you that we've troubleshot a lot of issues that turn around and end up being the lack of a security group rule or a misconfigured security group rule So if you find that the plumbing looks good and you're at your wit's end Try creating a security group rule and applying it to the port where you can enable connectivity from say the router namespace or the DHCP agent namespace Test connectivity with ICMP or something not impactful there Some other things that you may only see at scale are race conditions Caused by a lot of services doing things in parallel We used to see this a lot in Havana and Icehouse But you don't really see it as much in some of the newer releases There's a case where if you spin up you know the hundreds of instances that you know Neutron has to go through and set up DNS mask host files and do these things while Nova's simultaneously spinning up the instances They're ready sending DHCP requests Neutron hasn't caught up you may come into a condition where your instances have time out or given up And you have to reboot them Some other problems you may have some default system parameters are too low You may see this in default ARP table sizes where the threshold starts at around 512 ARP entries on a host If you have a lot of instances on a host You'll find that you have some random or sporadic connectivity issues You'll also see this when you're running the L2 population driver with Linux bridge What the L2 population driver does is that it pre-populates the forwarding table on a host With information about how to reach all the MAC addresses so that you don't see a lot of broadcast packets on the overlay Most notably on a network node say the L3 agent or DHCP agent A router that's connected to a couple of networks a lot of instances it has to know how to reach all of them So you will quickly exceed that table And that's a sysctl parameter that you can change on a host Also if you don't have any disk space available on a node and that service is dependent on writing to a file before actually proceeding You may prematurely kill a service by running out of disk space So make sure you keep an eye on that And last but not least syslog is your friend There's a lot of messages that aren't logged in a Neutron log file That are logged in a syslog file So if you're not finding anything check your syslog and you know think outside the box sometimes an error is somewhat nondescript But it could be completely related Thanks James Last we're going to close with some takeaways The Neutron failures can be traced back mainly to configuration file issues So have a good look there start with just a real basic config if you can And that should get you back to a running point To help you troubleshoot Follows, fails Yeah so Problem exists between the keyboard and chair a lot of times And if you can you know obviously you have to get the KVM The lower level workings of that would be nice Those little vertex file files All of that type of stuff Open vswitch OBS I mean that in itself is getting familiar with it would be very beneficial Especially when you see stuff like broadcast storms that you know where to look Linux bridging and IP tables Most certainly are important points to review if you have issues or you're running the Linux bridge agent Also familiarize yourself with the working environment if you can If you have a dev environment that's working you can use as a reference That will pay huge dividends Turn on debug mode we haven't mentioned that If you do just keep an eye on your logs they're going to fill up quick Start services by hand A lot of times you'll see errors that way that you don't see in the log Reach out to the community And gather as much information as possible before submitting a bug That way you don't waste your time when it gets rejected So that it doesn't get rejected And that's it don't be afraid to break stuff James and I have a ton of experience doing that So if you have any questions about breaking Neutron or Nova We'll be available to talk Thank you very much for your time I just wanted to add to that a little bit So when we say don't be afraid to break things obviously not production But I'm a real big advocate of creating lab environments Virtual box is a great place to start if you don't have the resources Get in there docs.openstack.org is great They have a lot of different scenarios that will help you figure things out When you're breaking stuff You know, spin up an instance Let it get connected to a bridge Make sure it works Then start pulling interfaces out of bridges and just see what the behavior is Try and create failure scenarios So that you can sort of reverse engineer when you do have failures in production You know where to look And then lastly so We're giving away some books here During the morning and afternoon breaks In the morning they're giving away the OpenStack Cloud Computing Cookbook The latest third edition In the afternoon they're giving away The first edition And then coming soon At the next summit we'll have the second edition of the same book So in there you'll learn More or less you'll get a good foundation On what Neutron should look like And a reference architecture With some troubleshooting So stop by and grab a copy And if you want the slides Again after this Sometime this afternoon we'll try and throw them up on SlideShare But it looks like we have some time So we can be at the back of the room And see what kind of solutions you've got Or do our best Thank you


All right, are you ready? I can't hear you. I can't hear you. Are you saying you want full auto? Full full. He's nodding his head folks. Okay So see another installment here Matt wants the full auto. We're gonna give it to him in 30 pound. Let's do this Well friends, that was rather anti-climactic My apologies, but a lot of fun with this full auto hot on blitz. I must say anyway What's not anti-climactic is this SoC giveaway go on over to pyramidair.com slash SoC Get in on your chance to win some huge shopping spree prize packages and some awesome prizes this very week We'll see you next time