Hello and welcome to the CMC markets market webinar with myself market analyst David Nadal Today's date is Monday the 2nd of October and the time is 12 15 as always with our webinars I will just leave the risk warning up on the screen for you guys to have a read of it It basically says Whatever is whatever I state in this webinar. It's just is it's not to be construed as explicit investment advice or trading advice or trading strategies This is just purely some observations and some personal opinions of mine and some commentary on what's going on in the news and also Discussing the discussion about what may or may not happen in some of the big markets that we're going to be looking at Also by a by you guys reading through this this risk warning disclaimer It also keep our compliance department very very happy It's just a necessary part of the business, but that just is how it goes It'll be the same rundown as always for the webinar. So I conduct every single Monday at 12 15 whereby we take a quick discussion about what happened over the over the last say 72 hours since Friday's closed in London and then also we talk about the you have a brief look at the week ahead the big corporate and Economic events on the on the week ahead and then we run through the major markets the major indices and some of the some of the commodities currency pairs What have you and as always if there any markets are having a covered being free to just mention in the chat box And we will then go on from and we happily have take a look at the market from there So this is the this is just the platform what we can see here is With the exception of Spain, we've already seen a positive risk on day for the equity markets in Europe The footsie is out of two and a half three week high Whereas the the DAX and the CAC in France are at multi-month highs both that about kind of three month highs Maybe just a bit more than three month highs The Italian market is pushed higher as well, but but the Spanish market I'm sure yeah, you've seen the news over the weekend on the back of the catholic Catalonian chaos the referendum that was held in in the Catalonian region in the region of Catalonia in Spain yesterday There was some quite There was some quite horrific scenes coming out of that some heavy handed Police brutality, I think a lot of viewers around the world were shocked and disgusted By what was in some cases what we've seen over the week of the weekends and the heightened tensions that we did see In Catalonia yesterday Has really has really spooked the markets as the Spanish market the Spain 35 Ibex 35 index is down well over 1% today Considering that when you take a look at how the some some of the major European markets are doing as I mentioned The footsie is at a you know here a three week high pushing higher and the CAC The DAX are at multi-month highs and there you have the Spanish market languishing in kind of last place of all the kind of major markets That has kind of been kind of the really big news over the weekend Today is a public holiday in China both over the weekend. We did have both manufacturing and also non-manufacturing data out of China which both showed increases both readings are both Increases and improvements on the previous month reading so all the Chinese manufacturing and non manufacturing sectors are Expand are growing at a quicker rate and they were only last on the previous for the previous reading So we've all seen on the back of that In turn on the back of that is higher copper prices and in turn the usual players Plank or Anglo-American real tint or be she built it using by the companies They've also been in fairly positive territory and the back of that Over the weekend over the weekend Theresa May announced addition additional 10 billion pounds or the health to buy scheme So property companies such as persimmon Taylor Wimpy And then you have you have Barclay development Red row all those sort of major players Have the big home builders listed on the stock exchange have also done quite well out of that stocks like Stocks like per persimmon have actually traded at an all-time high today on the back of that Looking taking a look at some of the economic data. We've had a mixed enough Manufacturing numbers of the eurozone Germany is probably the only kind of the kind of class of euros or numbers Germany saw and the increase In the expansion rate where as Italy France and Spain were actually kind of old kind of actually if anything kind of muted to Actually, maybe a slight deceleration in growth We also saw a slight deceleration in the growth rate of the British manufacturing sector as well So I said so that's real cake pound versus a dollar was already kind of losing a bit of ground Overnight on the current of the strong green back. So and then when we had a base on the UK Manufacturing P9 numbers it just kind of slightly accelerated the sell-off Taking a look at the week ahead. I'm sure if you're regular regular personally regular Subscriber to these podcasts, you know where they are But if you don't if you go to our home page and then under the news and analysis section click on news analysis This is updated several times a day This is the the main news and articles myself I'm like and the other analysts a CMC in the English-speaking world Published on this site some of our articles because gets published here others get published on inside Who's it's on our training platform, which I'll show you later on, but we also what we do is every Friday There will be a week ahead article which gets posted to the website So we're set here filter by click on topic and third one down weekly outlook Click on the weekly outlook here, and it gives you a rundown about the major corporate and economic stories for the week ahead of us so looking at we have the we have a Interest rate decision from the from the Reserve Bank of Australia or be a tomorrow Looking ahead to Wednesday and bearing in mind the strength of the Australian dollar could be something Which potentially curtail growth rate in Australia? So the consensus is for the or be a to keep rates that hold a one point one point five percent And obviously the Reserve Bank of Australia. It's sort of in their interest to keep their currency As well a bit softer if they can because the Aussie dollar has to do fairly well recently a strong currency Couldn't have actually curtailing their growth Looking to Wednesday as I mentioned at the top of the show. We have the manufacturing numbers from China from the Eurozone in the UK today We're gonna have a plethora of service matter service PMI reports coming out on Wednesday So keep an eye on that We have numbers coming out from from Tesco the big UK retailer on Wednesday And also we have their numbers coming off on the US drinks company PepsiCo We do have a couple a couple of other companies reporting their numbers this week, but to be perfectly honest It's a bit sort of slim pickings as I mentioned via Pepsi on Wednesday I also mentioned meeting the test go Wednesday scrolling down here Costco fourth quarter numbers on Thursday And then that'll be and then DFS furniture have full your figures out on on Thursday, but to be honest, it's a relatively quiet week in terms of the corporate reporting Take a look now to the footsie 100. It's broken. Well, it's broken nicely above The 7400 level as you can see here. So the big picture For the footsie 100 has been has been pushing higher created an all-time high in June Lost some ground over the summertime, but we are seeing every sergeant now So just zoomed the camera zoomed the camera. They did the yet It kind of the scroller in I took a closer look at how the footsies doing so The strength of the sterling really kind of drove drove the at the footsie lower here To the canloses of September, but since then Over the last two two and a half weeks nearly three weeks the footsie has been pushing higher as you can see It was in negative momentum the negative Declined then I was swung to positive momentum so you can be kind of more confident that this upward move Is to continue we're now just trading just pretty much on the one word on a day moving average And it's never here in around seven thousand four hundred and twenty. I'm seeing as we has a bit of a bit of previous Form of kind of acting as acting as support while while the market was was above that metric and results of Metric with allocation acting as resistance to the market was that was was below it Trying to push above it. So if we do break north and of the of the one-day moving average at seven thousand four hundred and twenty We could be then looking towards the September high of seven thousand four hundred sixty one Beyond that we could potentially look to the August high of seven thousand five hundred and fifty two And then beyond that again, we look through the all-time high of just seven to shy of seven thousand six hundred As I mentioned in UK service numbers are coming out during the week That's going to impact potentially impact the pound and in turn impact the footsie one hundred any kind of moves To the south on the footsie one hundred could potentially get support from the two-day moving average in around seven thousand three hundred and forty in our just such south of that seven thousand three hundred and thirty seven Notice how once the market had a very decisive break through it If they any kind of values were sort of running running out of steam just before it kind of got to it And then I find the actually made a decisive breakthrough it so That may act as support should we move south again And if we do take out the turning moving average to the downside We could then be looking back towards seven thousand two hundred thirty three and then south of that We could potentially see a retesting of the September low at seven thousand one hundred and ninety five The DAX is in far better shape We're talking about a three month high or you know, 14 week. How it's done quite well It's actually not too far away from the all-time high on the DAX So keep an eye on what's been going on the German market the downward kind of challenge that I was in is broke clearly out of that Nearly you know nearly one month ago as we kind of pushing higher It's slightly concerning though that that positive momentum isn't particularly high It certainly has been taking higher, but the most important indicator to keep an eye on is the price and the prices is pushing higher So that is the most important thing you need to be keeping an eye on so as you can see here It's creating multi-week highs multi-month highs The all-time high comes into play at twelve thousand nine hundred and fifty four So we're about probably 98 points shy of that and bearing in mind on the DAX That isn't a major move That's probably a percentage of about zero point eight percent eight tenths of one percent So which isn't a massive move for the DAX So we're within less than a hundred points of the old of the all-time high in the DAX So, you know for near record record high for multi-month highs And we're very close to recently close to the all-time high It's got to show you kind of bullet sentiment that is out there for the Germany 30 of the DAX Any pullbacks? We do see and the on the on the Germany 30 may get support in around this this area here in around 12,855 and then south of that maybe down towards 12,700 The US markets are even are even in better shape yet again taking a look now at what's going on with the Dow Jones the US 30 As you can see here, I know the official cash trading hasn't begun today But the futures market are indicating yet another record high for the Dow Jones buying on the dips has been a popular strategy With some traders in the last number of months as the markets going on to create record highs That's an indication of the overall sentiment is and as I mentioned price most important indicator The markets are regularly ratcheting up your record highs. That's that's the That's the that's the position you could you could be looking towards Say the kind of moves lower in the Dow Jones the US 30 may find support in around this price area here just shy of 22,300 and in south of that again the kind of the big Pullback that we saw it in the kind of mid late September at 22,216 and then to the to the upside people The traders are often keep an eye out because it's fresh territory Traders will have to be kind of keeping out for big numbers such as, you know 25,600 600 700 and so on As you mentioned before it's ever so slightly concerning that while the markets are pushing on here to do Pushing on to record highs. We're not really. Let's see that have been reflected in the momentum It could be it could be a sign that because of buying rate because energy that the bulls have it's kind of running out of steam But nonetheless if the markets printing are it's going to get a printing and all time highs I would you know the price in this case is more important than what the momentum indicator is showing It's a reasonably similar view if you look now at the S&P 500 Oh, I'll do that open up the wrong chart Simmer situation whereby the big picture the trend has been higher. We have seen some fairly decent pullbacks, but those pullbacks Didn't last particularly not well those pullbacks work within Where they would put what we're then made up and then we're done to create new highs as you can see here buying on the deep has been a popular strategy over the last number of months and similar to the Dow Jones the S&P 500 features are pointed to yet another new all-time high So once again any kind of pullbacks we do see on the S&P 500 may find support in around this price area here of 2,510 2,507 South of that because it's the steep pullback we saw in September at 2,488 and then south of that in 2,480 and then similar to the similar to the Dow Jones Traders will often be looking towards kind of hand big big numbers Because this is you know fresh territory fresh records. So you could be looking towards 2,530 4050 so on and so forth It's a bit more encouraging that you see as the the market was pushing higher here want to create in the last couple of training sessions Fresh all-time highs. It was a bit more encouraging to see that at the momentum here the positive momentum Is actually increasing so that way you can feel a bit more confident That the upward move is likely to continue because when you have the when you have the price as a Momentum moving in the same direction that that's when you can be more confident that the move is going to continue On the flip side of this positive news for the equity markets is poor old gold which has which has been kind of Pushing lower the last number of weeks. So In kind of mid-September gold early September gold hit a 13-month high And has been giving up ground ever since I've been pushing lower now For over two weeks well for over three weeks now. It's now trading a 1273 the 100 day moving average isn't too far away from here comes into play around 1272 1270 And notice how the 100 day moving average did act as a bit of support in August And also in July as well So it does have a bit of form of providing support For the gold market when the gold market was pushing higher So you could potentially see some buyers enter the fold in around the kind of 1272 1270 region for gold but The price is pushing lower. It's it's a notice how we're kind of bounced off of the 50 day moving average Had a bounce back and then I decided move south of the 50 50 moving average So that 50 day moving average may now act as resistance If you do if you do see any kind of moves higher on the price of gold Which should be in around the 1296 region and then of course they're going to psychological 1300 the big number and then north of that 1316 these are all levels for potential for resistance if you do see the price of gold push higher and then north of that again 1334 and the 13 month high at 1358 If you do break south of the 100 day moving average in around the 1272 1270 region that we could see a Further push lower a week back down towards the 200 day moving average Which comes into play just south of the 1250 region 1249 notice how Negative momentum is is still is still quite high. So it gives you an idea of where we're where I'm how much kind of Power is behind the sellers. We've yet to see any kind of waning of the of the momentum Negative momentum at the end at the same time We get to see any kind of a sign that this doubt that this is the downward trend has come to an end taking a look now at the price of The oil markets, I think like the Brent bearing the mind Brent had a really good run in the last few weeks Whereby Brent went on to hit a 26 month high only last week This actually just just six six straight six days ago in a 22 months at 26 month high right rather And it's been giving up some ground since then so but then again if you had a 26 month high You are going to potentially see it's already surprising to see some profit taking So it's kind of went out to hit a multi-year high ever since then we've been giving up some of some of the gains That I had that I was registered for the time being it may find support in around the it's just it's just north of 56 dollars in parallel for Brent We may find some support at 56 the number itself and then if he moved south of that We could be we could see a retesting of 55 or in south of that again. We could see a retesting of 53 83 But bearing in mind the big picture, you know, if you're at multi-year highs That does tell you this a lot of kind of pull a sentiment still in the market It's been in a fairly clear and concise upward trend since since June It's only for kind of looking head and back south of the the fifth day moving average Which comes into play in around fifty fifty three fifty or the tour the moving average at fifty two fifty three And move south of that may then bring the kind of upward move from from June into question And if that were to be the case We could then see a move and be testing of the one or two moving average at fifty one dollars and twenty one cents Notice how we did see a bit of support at the one or two moving average price in both July and also August It is worth pointing out that as the market was coming off here positive momentum Slip and then of course actually swung at the negative momentum. So we're so the price moving lower momentum is moving It's a negative territory So we could see a bit of a current we could see a continuation of the downward move and then whether that Whether that's just a dip a correction in the wider positive move or whether the beginning of something new We'll have to see kind of what level it takes out on the way down It's a a similar ish looking church when we turn our attention to WTI WTI didn't have the kind of quite Polish run That Brent data, but it did hit a four month high only at the back end of last week So you can see here WTI West Texas and intermediate training at its highest level Since since April this year. So it got to a four month high and similarly went like like a Brent We have seen a bit of a tailing off in the in the price But it should be kind of continue to move lower in the price of Brent We could potentially find support in in at the tune of a moving average at forty nine dollars and thirty cents As it as it's at it as a recent hit in multi-month high It was kind of suggests that I did take out the may high It would suggest that the kind of overall sentiment is still fairly Polish on the energy market Bearing the mind we have the OPEC meeting coming up in November OPEC have already ready to be making noise about potentially or some OPEC members are making noise about potentially extending the production freeze By three months out until the end of June 2018 So that's something to keep an eye on it's also worth pointing out that we actually saw an increase in the number of active rigs on The Baker Hughes recount from the United States over the weekend So just keep an eye on that But should we should we remain north of the two or they move the average on WTI at forty nine dollars and thirty cents It's possible. It's possible. It's a potential possibility That we could see the continuation of the wider upward move in Brent in WTI And if that is the case The levels to watch out for to the upside would be the September high of fifty dollars and fifty three cents Keep an eye on that level and then some some buyers might be then looking towards the April high at fifty three dollars and fifty six cents and then north of that again Some bulls might be keeping an eye on the February high of fifty four dollars and sixty three cents Whereas if we move if you take out the two day moving average in the downside We may find some we may find some buying commit to play in round the forty eight dollars per barrel level Just turning our attention now to the currency markets Having a look at the euro versus the US dollar so After a great run for many months, you know, I'm hitting a multi-year high The euro dollar in September. I've been largely been selling off since then as you can see here The price has been pushing lower negative momentum is on this has been on the has been on the rise and if on the Ever since a trend itself of the fifth day moving average It's not difficult to get back above that metric So for the time being while we remain south of the fifth day moving average on the euro dollar Which is in around the kind of 118 50 region or 118 49 region We could see This the negative move that's been in place for the last few weeks continue And if you do if you do kind of keep to keep south of that metric We could see if we could see a pullback to the August low of 116 sorry 116 16 to 62 and then south of that In the one of these one of the class one of the spikes lower in July at 116 13 And then if you take on that level as well We might even see a pullback to the 100 a moving average at 115 75 But should we take should we recapture retake the fifth day moving average? Which is in around the kind of 118 50 region? We can then see a potential move back up towards 119 120 and if you get back up to kind of 120 handle that could be a sign that the kind of But that at the kind of negative move with you've seen over the last few weeks It was only just a correction and they kind of wider up for trend and then the trainers and bulls We could potentially be looking towards the recent the September high Of 120 at 92 and then they could towards 121 and 122 But notice how the momentum is still very much in negative territory You know you would want to see a cooling of that as the price is pushing higher cable there the pound versus the US dollar is Not too dissimilar to what we've seen on the euro versus the US dollar So so the kind of big picture has been quite positive over the last four months on the pound versus the US dollar Lurching higher the pullback Want to go to new highs onto new kind of multi-week highs multi-month highs followed by a lower by a higher low And after they can a great run ahead in September We have been pushing lower since this this area here The wider trend is still is upward trend is still in play But seeing as price has has been coming off quite aggressively and we can see a fairly clear and concise Decline the positive momentum that even a swing into negative momentum The momentum is clearly to the downside We haven't seen any any of the buying pressure Any cool as of yet so we could see a further move south from the in the pound versus the US dollar So a potential ever to watch out for to the downside is this price here About which is which is the which is 132 67 and then south of that we made we could find some could find some support in around this area here Which is the 50-day moving average in at 131 96, sorry one 130 what 131? 29 29 rather the 50-day moving average and then south of that a Water-day moving average comes into play at 130 These areas that we potentially see if the market does continue to move lower We could see in the near term moves Down towards these metrics, but also as the track as the kind of wider trend upward trend is still in play These are also metrics, which we could see buyers enter the phone because You can notice here at the market was pushing higher We did see a bit of support from the water on a day moving average And also both metrics are provided as a bit of support in the early quarter of the year as well But if you do happen to take out water-day moving average then some trend then looking towards the 30 moving average in at 127 46 Just change over To the euro versus the US dollar So what Matthew December to the euro dollar and also the pound dollar whereby the euro for the euro starting It's a great one for many months on until it all came until we saw a large seller until it hitting a until and 30 large turn around Which began over the wash you about five or six weeks ago So quite an aggressive decline in the euro versus the British pound Starting from late August crash through the 50-day moving average crass below the water-day moving average And now it seems to be sort of trapped in between the Water-day moving average to the upside and this price here in at From the July low in at 87 to 38 to the downside So it would be nice to get a break either side either direction out of this kind of relatively Tight range so a break to the upside should be taken off the water-day moving average, which is in around the kind of 8900 level We could then see a push higher up to the fifth of the moving average at 923 and then north of that up towards 1988 and then to 9161 But if you can't break north of the water-day moving average And if you do take off via the the July low at 87 38 We could we could see some buying commitment to play in on the 30 moving average in at 87 24 But then south of that we keep it back towards the 86 region Notice how while the price was in the kind of sharp decline here We did see and they're quite an aggressive ramping up of the negative in the of the negative momentum and Then we saw the all the kind of intense setting the pressure dissipate here So so it's all we're almost back to kind of almost like almost a neutral momentum. So it's almost like the kind of Bulls are sorry the bears rather I was kind of run out of steam And they kind of severe setting pressure has declined So it's almost like the market is at one of those points whereby I have this quite aggressive rally from the lows of April Up to the high in August Is this move just going to be a correction in the wider positive trend? Or is it going to be something that when the market turns over on itself? And that's where if we do get a break over this region could be indication of whether we're heading back up to retest the August highs or whether we're actually going to be actually the The market is going to be turning over on itself So what if the market does get the market does break north? What you'd like to see is you'd also like to see a increase You'd like to see see positive momentum come back on the radar and then increase From there or vice versa if the if the markets is pushing lower You would also like to see that being mimicked by an increase in negative momentum just turning our attention now to the the dot the Japanese yen Europe dollar versus the Japanese yen If there's anything you'd like to meet you cover But I haven't covered already feel free to stick it in the box And I will just happily talk about that So take a look now at the dollar yen the dollar yen has been hanging on nicely Despite the fact that's been broadly been pushing lower largely throughout 2017 the fact that it's had quite an aggressive turned around in September because it has been a broad increase in the US dollar It's managed to kind of take out some of it some of the the lower highs that we witnessed from in July and August Now it now has managed to retake the 200 day moving average Why we remain north of the tour day moving average which comes into play Roughly speaking around 112 the outlook could potentially remain positive from the dollar versus the end And if that is the case levels to watch to the upside to potentially look out for could be this What one of the highs from July in at 113 57 and then north of that again the actual high in July at 1 14 49 But if you do manage to Break below the tour day moving average in around 112 again that level may then become active resistance to any kind of moves higher And it moves to the downside might find support You know the one or two moving average at 1 11 0 5 notice how the one or two moving average was acting as a bit of resistance Well stroke support in around this price area here as it was going north if I the difficulty break north of it But once it got above it at that price action that metric then actually began to support the end of positive move And then a move south of the of 111 0 5 could then see a return down towards 1 on 9 and 55 and then south of that again the September low of 107 32 So now that we've covered The main markets obviously be quick as always show you a quick kind of run down of where we we also have other items on our Trading platform at the top of the show. I talked about how we have the week ahead cap week ahead We have week ahead article talking about the main economic and also a cover events of the week If you go to the market pulse tab fourth option down, you can see the market calendar Give you full breakdown of the major Major economic events of the week ahead of us You just scroll through it here It'll give you the previous reading and also give you the forecast reading and then once the number is actually out itself It'll then actually be in the box Populate immediately as soon as the number itself is out So this is obviously keeping if you're trading in predicted currencies It's something that you need to be aware of and keep an eye on obviously with the big The big one to keep off what keep an eye off for this week is going to be the non-farm The non-farm payrolls which I mentioned in our non-farm payrolls in just one second So under that tab there market pulse We also have the insight section and this is the insights here as I mentioned some of our some of our news Analysis that we analyst create goes to our news Goes on to our website where some of it Goes on to the actual inside of self on the trading platform like it's up updated regularly throughout the day What we also do as well is what you can see here chart forums from a self and other other analysts What we do is we upload very kind of short a Few hundred characters a few hundred words of what's going on with any particular chart and discuss prices We may see which could be of interest in the future And also lastly looking at other events that we that we are we are holding here Looking at other events What we can see here is Tomorrow in our London office at 6 30 p.m. London time We do have we do have an in-house webinar So we do have any an in-house seminar rather And the title of that seminar is called mastery the mind and the and the markets on Wednesday at 7 30 p.m. London time We have a webinar covering the defined trading strategy and On Friday the 6th of October at 1 15 burn a summer time We will have our non-farm payrolls webinar covering What's going the big move covering the numbers and the big moves that are going on in the numbers as well to do The market that is moving by far the most today and it's it's in the it's in the headlines for all the wrong reasons Is the Spanish market? the ibex 35 unlike What we saw in the CV district could we just ignore the Catalan situation for us one moment The Spanish 35 has been actually been kind of pushing lower all the way Broadly speaking throughout the summer whereby we did see us a push We did see it at a decline and then a snap higher about the about the German market and also the French market Whereby the Spanish market has improved the pushing lower throughout throughout the entire throughout the summer time With this but the exception of this kind of snap push higher where it did break north on the 50-day moving average It was improved these speaking Being held up as well as it's broadly found the difficult to to gain ground north of the 50-day moving average And we can see here even at the back end of last week Everybody struggled with that metric So while the Spanish market remains south on the 50-day moving average which comes into play at 10386 It's likely we could see the market Remain under pressure But at the same time it's also getting a bit of support from the two-day moving average which comes into play at 10,029 So keeping an eye on the on the market here itself. This level here is enough to keep an eye out for on 10,000 100 or 10,087 I believe is the low in September And then south of that Chairs are then been looking towards the kind of the 10,000 level itself. Should we break north of the 50-day moving average at 10381 we can then find potentially some resistance in this price area here at 10,445 And then north of that at the one or two moving average of 10,546 Now I appreciate your time feel free to tune in next Monday at 1215 and as I mentioned we do have other for both seminars and a one in-house webinar Going on at our London office this week. I've been David Madden. Thank you for tuning in Please tune in next week. Have a good trading week and good luck


to them what we do in Cape Town and the response has been really great. All the interest, I love the show, it's been great. Estes are. Is that where the investors are? Yeah, we want to show them a bit of Cape Town, show them that Cape Town is the place to invest. That is a development we just started, it's in Observatory, the Observatory area, and we're in the demolition phase of it right now.


Hello, everyone. My name is Gaurav Kumar. I work for the Uber's Compute Platform team in Amsterdam. Unfortunately, my colleague Amit couldn't be here today due to logistical reasons. So I'll be leading this session solo. In my roles and responsibilities at Compute Platform, we are mostly focused on running different kinds of workloads. And in this particular talk, I'll be talking about scheduling GPU workloads on Kubernetes and the kind of insights that we have had so far while scheduling such workloads. So yeah, let's get started. As the outline of the talk, I just want to clarify what this talk is not about. So this talk is not about how to set up GPU workloads at the beginning. So basically, I'll not talk about how to set up your clusters to run GPU workloads. Rather, once you've set your clusters up, what sort of challenges and requirements that you face along the way, I would go through a brief overview of how Compute works at Uber. And I'll give an overview of Kubernetes clusters for GPUs at Uber. After that, I'll cover the sections about efficiency and precision, what are the goals and the challenges and the solutions that we implemented to achieve these efficiency and precision targets. I'll also cover a little bit about benefits for scheduling efficiently and precisely on the Kubernetes clusters. We'll cover some common pitfalls that we encountered along the way to enable such GPU workloads. I'll briefly touch on future work that we have in the plans. And if we have times, we'll cover a short Q&A session as well. This is the overview of Compute at Uber. At the top, we have two federation layers. We have a stateless federation layer and a batch federation layer. These federation layers manage resources efficiently. And they are responsible for different types of workloads. So for stateless, we have a federation layer known as UP. For batch, we have our own federation layer known as Batch Federator. I would recommend everyone to see the talk on batch federation on KubeCon 2023 North America, in which my colleagues talked about the batch federation layer. Our Compute platform has been consistently evolving throughout. We were based on Mesos and our in-house scheduler known as Peloton. And over the year, we have transitioned to Kubernetes. Below the stack, we have our presence across multiple cloud providers, such as OCI and GCP. And we have our on-prem data centers as well, where we run our own hosts. The overview for GPU use at Uber is like this. We use GPUs for model training for our AI and ML workloads. We have data science notebooks, which are used for exploratory data science work. So for example, if you're a machine learning scientist or a data scientist and you need access to a GPU, you can just launch a session, work on that session, have the access to the GPU, and return it back to the cluster. Some use cases include for the models that we train like ETA prediction. So you'll be able to see on the app like how long will it take for your cab or food to arrive or how long will you be reaching to your destination. So we have ETA prediction models. We also have LLM-based chatbots, which will do automated customer support. And we have LLM use cases for our own internal users. In terms of scale, we have more than 4,000 GPUs. And we have about 50,000 jobs which run every week on those GPUs. In terms of GPU models, it's kind of like a mixed bag. We have NVIDIA H100, NVIDIA A100, some QuadroRTX4000, and some 1080TIs. So it's like modern GPUs as well as some legacy GPUs as well. So yeah, this is what cluster level overview for a GPU cluster looks like at Uber. So we have an in-house component known as Kubernetes Object Pusher, which is responsible for syncing the NVIDIA device plug-in daemon set spec to all the clusters. It reads from a repository where we describe the spec for the device plug-in. Once that spec is synced, the device plug-in daemon set part gets launched on the node. And we are able to advertise GPU resources to the cluster. The device plug-in and the kubelet communicate via Unix socket as standard. On the node level, this is what it looks like. We use the NVIDIA container toolkit, which contains the implementation of the alternate container runtime. Our container runtime on production is mostly based out of container D. So we have configured container D to be able to hook into the NVIDIA container runtime whenever we receive a pod which has the runtime class name as NVIDIA. If it does not, it goes through the default runtime. So those are mostly stateless pod and non-GPU workloads. We also use CAdvisor for being able to monitor GPU metrics. We have Rabble written on top of CAdvisor to be able to label the metrics on a workload level because we want to do attribution of what jobs are basically using what GPUs. So for example, here's a screenshot of a job which is using four GPUs. And those four GPUs, they span across two pods. And those two pods each have two GPUs across different nodes. So now that you have set up your clusters to be able to use GPUs, life should be fine. But unfortunately, it isn't. Once you're able to just set up your clusters for GPU workloads, you come into requirements for efficiency and precision. I'll get into detail for how that happens here. So first, you need to support a heterogeneous cluster. So a cluster can have CPU-only and GPU nodes as well. And theoretically, CPU workloads can run on GPU nodes. It's not the best utilization of them. So your cluster should be able to support heterogeneous nodes. That would be one requirement. We came into this requirement mostly when our machine learning scientists sort of wanted to use Ray jobs. Ray is a framework by any scale where you can launch a job which can have CPU-only workloads on some parts and GPU-dedicated workloads on some other pods. You have to be able to support multiple stock-keeping units for GPUs. So GPU workloads should be scheduled on the right skew based on what the customer is asking for. And the workloads which do not require a specific GPU skew should not be running on those specialized GPU skews. You would also need to minimize the fragmentation of GPU resources to be able to schedule efficiently. So I'll get into details of all of these sections in the slides later. Yeah, so when you have to support heterogeneous clusters, the key points here is that GPU resources are limited and they are costly. So you don't want to waste GPU resources by running non-GPU workloads on them. And here, as you can see in this example, we have a cluster where a CPU-only pod is scheduled on a GPU node. This tends to happen if you just let the cluster run amuck and you don't have any guardrails in place to avoid this. So as you can see, a CPU pod is running on a GPU node and a general GPU pod will anyways run on a GPU node. There's also one catch that you need your device plugin to be able to run on only GPU nodes and not CPU nodes because that's not optimal. Also your device plugin will just keep crashing in that particular case. So how do you solve that? Well, there can be multiple solutions to do this. First, you can use to, first you can label your GPUs to be able to advertise that you have a GPU on this node and what kinds of GPUs are there. Node labeling, in our case, I mean, you can use different solutions. As I think Kevin and Shiva mentioned, you can use NFD. For us, we use a custom way of labeling GPU nodes. We already have an on-host daemon, which uses NVIDIA libraries to fetch the information about the GPUs. At boot time when we launch the cubelet, there's a wrapper which calls to that daemon, gets the information about the GPU nodes, parses it, and labels the node once the cubelet is booted. Once you have labeled your GPU nodes, you need to add the necessary node selectors to the pods of your device plugin so that your device plugin only runs on those GPU nodes. And you can use a GPU management filter plugin to basically ensure resource isolation between CPU workloads and GPU workloads. Just a bit of context here. What do I mean when I say a GPU management filter plugin? So Kubernetes allows you to write plugins for various points in the scheduling cycle of a pod.  So here what it looks like when I say if you have to implement a GPU filter plugin. So if you have a pod which is not requesting GPUs, the GPU filter plugin will filter out all GPU nodes from its possible candidate replacements. And your CPU-only workloads will only run on a CPU nodes. If you have a pod which is requesting for GPUs, it will be a know-up, and it will directly get scheduled on GPU nodes. So the GPU management filter plugin is sort of a know-up for GPU resources. Well, once you've ensured the isolation between CPU and GPU nodes, you think that, again, life should be comfortable. But again, unfortunately, it is not. The reason for that is now you have another problem where you have to support multiple types of GPU SKUs in the same clusters. So your product teams would come up to you saying that now we want to train LLMs. And LLM training requires specialized hardware. For example, NVA100 and the best use case for training those LLMs, best use case for using those specialized hardware is to train certain GPU workloads and not let anything else run on that for some reason.  But again, it's not a very optimal use case for those workloads. Also, GPU features differ completely on the SKU types. So not all GPUs are created equal. Some GPUs have more VRAM. Some GPUs have different kinds of compute. Each of the GPUs cost a different amount.  And one more key thing was that model accuracy is highly dependent on the kind of SKUs that you use. So if you train your workloads on a certain kind of SKU, we've seen in production that the model accuracy varies by a few percentage points. So as you can see here in the cluster, since we have ensured that CPU-only workloads will run on CPU nodes and GPU workloads will run on GPU nodes, the CPU pods are fine. But if you have a generic GPU workload, it can get scheduled on specialized GPUs. But you have to prevent that. So again, how can you address this problem? Well, you can, again, write a new filter plug-in, which is called the specialized GPU management filter plug-in. So plug-in one will be your global GPU management filter plug-in. It filters out all GPU nodes from a pod which does not request those. The plug-in two will be a specialized GPU management filter plug-in, which will filter out all the specialized GPUs from any pod which is just requesting for general GPU resource and does not want to be placed on a specific SKU. And at the end, we have placement strategies based on node selectors.  What does this do? So once you have done this, you are able to isolate between CPU nodes, generate GPU nodes, and specialize GPU nodes. This ensures better hardware utilization in our clusters and, obviously, once we are able to schedule workloads on the right nodes, we are able to achieve better model accuracy. This also gives us cost savings because, obviously, we are able to utilize the hardware better. The last time we did some evaluation on the kind of cost savings that it gives us, it was around half a million dollars a year. OK, now since you've done that, you're able to isolate resources. You should be able to rest on your laurels. But again, you see another problem. And the problem is based on GPU fragmentation. What tends to happen in our use case with scheduling other kinds of workload was that we used to prefer load aware placement. When we say load aware placement, we try to place the workloads on the nodes which are not being utilized to the fullest so that we ensure there's a uniform utilization of resources across our clusters. This works well in the CPU world, but in the GPU world, what that results in is fragmentation. So here you can see if you have a cluster. We have nine GPUs available to schedule your workloads. But if you want to place a pod which requires more than two GPUs, you won't be able to place the particular pod in the cluster. And we've had complaints from our customers saying that even though I have GPU resources in the cluster, I'm not able to schedule my pods because of this fragmentation issue. So how do you mitigate fragmentation? Well, you could use something like bin packing placement. So Kubernetes by default has a node resource fit plugin.  And then as you can see, you have nodes available with more than three GPUs. So your pod can be placed on those nodes. With that being said, I'm not saying that we have completely solved the fragmentation problem in our clusters. The reason for that is it's more of an art than a science because you have multiple scorers inside your cluster competing for the placement. And then you have to tune the weights of different kinds of scheduler plugins that you write so that you don't run into issues. Well, so now you are able to isolate resources. You're able to schedule workloads on the appropriate SKUs.  You should be fine with most of the scenarios that you see. But then life is not again so easy. You run into some common pitfalls. Pitfalls is just a fancy name for bugs that we encountered along the way. So yeah, one of the pitfalls that we found maybe after a week of deploying the plugins was that the device plugin pod and the filter plugin do not play very well. And the reason for that was really simple. I'll get into the reason first. So what basically happens is when your node joins the cluster, it is effectively like a CPU node. You don't have GPU resources being advertised on your node. And you have the GPU filter plugin in place. Once the demon set spec is synced and the demon set pod gets scheduled on that particular node, you start to advertise the GPU resources on that particular node and all as well.  So basically, the GPU filter plugin was making a decision that is the pod requesting for GPUs and is the node a GPU node. And if that's the case, then I don't want to schedule pod on this node because device plugin essentially is a CPU-only workload. Well, that's not ideal. So just for the simplicity of the solution, you can add an exception for your device plugin pod in your GPU management filter plugin. And it just not only applies to device plugin pod, it applies to all the system pods that need to run on your GPU node, but those are not GPU workloads. Another common pitfall that we encountered was that pods requesting for more than GPUs than they are available on a node. So for example, we have a cluster here. It's a theoretical cluster. We have nodes with four GPUs, six GPUs, and eight GPUs. And if a pod tries to place a workloads, which contains overall 10 GPUs, what used to happen was like it would go through the admission control. And we'll try to place the workload on the cluster. And we'll go through multiple scheduling attempts. And we'll keep on facing placement failures because essentially no node can accommodate this pod. And if no node can accommodate this particular pod, you get alerted that your job has been stuck for a while and you cannot run your workloads. Well, what you could possibly do is to prevent the admission of such pods in your cluster, which contain the request for resources more than of what your particular cluster at that point of time can provide. This can also be solved at the federation layer, but I'm just talking about on a cluster level. So if you try to place a pod which requests for 10 GPUs and your cluster does not have a node which can accommodate 10 GPUs, you just deny admission of that particular pod. Well, it's again not so simple because you can't just hardcore saying that, OK, in my particular cluster, I have pod which can support eight GPUs. So I'll just hardcore eight and then ask users to send their workloads. And once the scheduling happens, I'll be able to deny admission to pods which contain more than eight GPUs. Because your cluster essentially is a living entity. Nodes keep coming in and nodes keep going out. So tomorrow you can have a node which contains 16 GPUs or, I don't know, 10 GPUs, or your eight GPU node can move out of the cluster for some repairs or maintenance. So how do you do that? Well, you can write an admission check using a node informer. So you will watch on this particular cluster. I'll watch for node updates. I'll keep account of the maximum number of GPUs that a node can have that a one node has in this particular cluster. My admission control check will just validate that is my pod requesting for a number of GPUs which is higher than a node can supply? If yes, then I'll just deny admission. If not, then I allow the pod to be admitted. This is a very simplified solution. We also need to take into account the fact that your pod spec also contains init containers. And when you have init containers, you need to evaluate the maximum of an init container as opposed to the sum of the number of GPUs that are being requested by your pods, requested by your containers. Another thing that you would probably run into is the fact that you have privileged containers running in your clusters. So just for a bit of context, a privileged container is a container which has elevated privileges, and these are not limited by your container runtime. If you run privileged containers on your cluster, apart from just the security risks that are associated with running such containers, you'll be seeing that a privileged container has access to basically all the GPUs on a particular node. And this affects resource attribution. It also affects your metrics in terms of what GPUs are accessible to this particular container. And yeah, you run into problems. Well, we took a decision to avoid running such containers into our clusters. The necessary use cases for privileged containers are governed by some other route of deployment rather than Kubernetes. So we do another admission check. We say that if the pod has privileged capabilities, for example, it has capsus admin or capsus chroot or anything like that, we just deny admission to it. Also, if the pod security context allows it to its privileges to be escalated, we do not admit such pods into our clusters as well. And if everything is no, then it's a no op and your pods can get placed. So yeah, I'll just briefly talk about the future work that we plan on doing. We plan on supporting fractional GPUs. So in our current setup, GPUs cannot be shared by containers. With that being said, we do train multiple models on our single GPUs using software techniques, such as low rank resource attribution. Workloads, if they are not able to fully utilize the GPU, will essentially waste resources. So we are planning to use mix in order to enhance the utilization of GPUs in our clusters. We found use cases of certain inference workloads, which scale well with mix. So yeah, the plan is to support this. In the future, we also plan to support multiple GPU vendors, like Intel and AMD. The few reasons that we are planning to do this is because of price to performance ratios of certain workloads with these different GPU providers. These providers have open source drivers in stack. And the device plugin for these vendors are also open source. So you can just check them out. With that being said, I don't expect it to be very smooth, just like the experience that we have had within VGA GPUs. A bit of a shameless plug. This is the team that has worked on most of these features. Feel free to reach out to us on LinkedIn if you have any questions or feedback. And with that being said, I'll open the floor for questions if you have any. And please scan the QR code if you have any feedback for the session. Thank you. You have mentioned Ray. Are you using Kubray for Ray? And are you using Ray self? I'm sorry. Can you repeat the question? Yes. You have mentioned Ray at the beginning. I guess that you are using the Ray framework on Kubernetes, Kubray? Yeah, so we are using Ray workloads. Those are running on our clusters to train the machine learning models. So we have multiple sorts of bad jobs that run on our clusters. One of them is Ray, which is a framework by any scale. Exactly. And I think that there is an option to use functional GPUs with Ray. And are you using anything other than Ray serve? Because currently what we are using is mainly serving the ML models with Ray serve, nothing else. So I guess that you are using also Ray jobs? Yes, there are Ray jobs. There are also Spark jobs. So yeah, it's multiple sort of jobs that run on the batch platform. It's not just Ray. Super nice. Thank you so much. Thank you. Thanks for the talk. I have two questions for the first one. Are you separating the federation layer based on CPU and GPU workloads? Is that what you meant on the first slide? Oh, so yeah, our federation layer, so the question is that are we separating GPU and CPU clusters? Like when you said you have separate federation layer. Yeah, we have separate federation layer, but it's mostly on job types. So we have a federation layer for stateless workloads, which run like microservices and other things. And we have a separate federation layer for batch workloads, which run these like bad jobs like Spark jobs, Ray jobs, and other machine learning models. I see. For batch workload, how are you guys approaching curing batch workloads on the federation layer? I see. So that's a good question. We already have a batch federation well through something called a multi cluster federator. So what that federation layer does is it admits the job from the customers, and then it tries to distribute the job to the appropriate clusters depending on the resources available in those clusters in this sort of like a multi cluster queue. So it will redirect the job to the appropriate cluster depending on the amount of resources that particular cluster has. So yeah, I would encourage you to watch the KubeCon Chicago North America talk if you want to learn more about the federation layer for batch that we have at Uber. I'll link it in the presentation as well. Awesome, thanks. Just one final follow up question. Sure. If you could share more on what are you guys prioritizing in terms of that queue? Are you guys prioritizing fair sharing or maximum GP utilization rate, et cetera? Yes, we have that fair sharing model. I can maybe get back to you on the details of how that fair sharing works on a cluster level. I am mostly part of the platform team, which runs the workloads on Kubernetes clusters. So that federation layer sits on top of the platform. Awesome, thank you. Thank you very much. Sure. Thank you so much for a great talk. I have a question. I'm just curious if you follow all the best practices and avoid those pitfalls? What is the highest CPU and GPU utilization in the cluster that you've seen? So that's a really nice question. When it comes to CPU utilization, we are highly overprovisioned. We are still working on improving the overall CPU utilization across our clusters. But just to answer your question, majority of our workloads are probably like the number of the peak CPU utilization that reaches in a cluster is close to 80%. So yeah, you can never really reach a very high CPU utilization, because then you will run into issues with the services being able to not perform as expected. And when it comes to the GPU utilization, I think when it comes to utilization, more than 50 to 70, but never beyond 80. Thank you so much. OK, thank you very much for your time, guys.


The best kind of doors are the doors you have to explain. All right, welcome, welcome to Unhinged with the DoorDork today. We do have two very special guests, Mr. Bryce and Tom from Capital Lock. I'll jump on to the next one. Bryce and Tom were generous enough to send over a couple of photos from their own stock. We like to call them adventures, you know? What their team has found out in the wild. And so these next two photos are gonna be because of Capital Lock. So thank you guys for contributing. And I'm excited to see this. This is my favorite though, because we don't often have the backstory. So today we'll get that, because often it leaves more questions. Right. Okay, are you guys ready? We are. Yep, yeah. So I'll give a little bit of the backstory on this one. We're doing a walkthrough at a hospital here and they have some medicines that need to get stored in a cabinet. And so the solution that the previous Locksmith or installer decided to do is they took a salto mortise lock meant for a door, turned it on its side and installed it into a drawer. So they use a door mortise lock as their card access for a medicine drawer. And it surprisingly is put in really well and it actually functions as planned. Wow, who would have thought that could have worked? Yeah, we were walking through the facility trying to identify where these were at. And my Locksmith looks over and he's been doing this for about 30 years. He caught his eye and he's like, is that a lock on a drawer? And he walked over there and sure as shit, there it is. He said this is probably one of the most creative solutions he's ever seen in his 30 years as a mortise lock for a door installed on a drawer. Yeah, that's definitely creative. I've never seen anything like that before. It's almost like salto needs to have the cabinet lock or something. I feel like they actually have the solution for this. Yeah, so they have about 40 cabinet locks throughout the facility. So I don't know if this was just a leftover. And they said, you know, some guys said, I'm gonna be creative on this one is one of the challenge. I don't know what they were thinking but well executed it function. I appreciate it, especially if they're gonna use one of our mortise locks as the functionality, I love it. Yeah, I would have been worried that in a laying down orientation, all the components inside would have still done what they were supposed to do. I've never really thought about it. I've never really thought about turning the mortise lock on its side and then making sure it continues to function. So good. And this probably, if they had them laying around, great. If they purchased them for this, this cost them many times more than what they needed to spend on this solution. But if it's working, I mean that lever also acts as a great drawer pull. It's true. I'm just now like trying to orient myself. Lever is perpendicular, like a regular lever. Like you would use it to pull it open, but like you would also use it, does it latch that way? Or is it? No, so they actually turn the lever, so it's 90 degrees. I think with just the sets, we didn't take it all the way apart, but they turned it, so that is how it is normally is. And then you can pull up or down after scan, just like if it was on a normal door, you'd be going left and right, but... Yeah, there's nothing that would really prevent you from doing that, even on any of the like Osa Abloy product, like the mortise levers, just how they're mounted, right? You could, there's nothing to say as an industry, we couldn't just have to go with this orientation. So yeah, I mean, I think it's an interesting way to solve a problem. Big application, I love it. I would like to see like how they even thought of that. Yeah, I don't know. It took some custom fabrication just to make it work. So it's not clear what exactly they were thinking. On that discussion on the inside of the door, it kind of looks like there's a key cylinder there. I cannot remember if there actually was a key cylinder in there or not, but somebody actually could, if it's a classroom function, make that unlocked and then it kind of makes the card access not so effective. So possible issue. If this runs out of batteries or like I know there's like systems in place to help that not happen, but that you could have a locked drawer, you might have to bust open this drawer. I'm just curious on how many times someone has run into that, caught their lanyard on it, or anything looking down into that drawer. I'm sure people have thought about that of, well, why is this thing on here? Yeah, I'm glad it has the return on there. If it was just a standard lever, that's probably like catching people's hips all day long. And being in a hospital, like there's carts and beds and like, especially if this is for pharmacy use or drug use, like they have those carts that they're always running around that it walked down with the drugs inside of them. Okay, any last comments before we give it a knocking score? What do you guys think? I'd say we rate it. Well, you know, it's overall still pretty functional. It serves its purpose. Obviously, you know, touching on me at point, it probably went way over budget with it. I'm going to give it a 1.2 because, you know, I like the ingenuity. Now that you've pointed out, catching the handle, I don't know if there's a fire and life safety codes for drawer handles. And if that's a factor at all. But, and also like I said, maybe the lock cylinder on the inside could cause a security issue if it can be made unlocked. I'm going to give this a 2.1 just for a slight risk there, but still overall a very functional and effective part what they were trying to accomplish. I'm going to go with a 1.5. 1.5. Because luckily it doesn't have to do with entering or exiting any place in the building. It's just to house medication. And since we have levers on doors, I'm not worried about anybody hooking themselves on. And since it has a return, that's there to prevent anything like hoses or anything from getting hooked. And if they're catching their lanyards, they should have breakaway lanyards as part of their hospital policy. So risk is low with the exception of being able to unlock it with the key inside and override the access control. I'm going to give this an eight or a nine on creativity. Although I think it's a genius that they came up with something like this. And the word genius, just thinking outside of the box, like thinking outside of the norm, it probably has a better higher security than your average cabinet lock bonus points for them. And maybe that's what's their idea behind it is they wanted extra security on there. But yeah, as far as a knocking score, not too knocking bad, one or two maybe. Okay, you guys ready for the last photo? This one's a doozy. Oh yeah, took the words out of my mouth. You can notice quite a few violations and life safety code in addition to fire code here. So the background story on this one, I was the one who took this picture out in the field. You can see one of our texts pointing at it directly and laughing there. He's the one with 30 years of experience out in the field. So this is a fire door into the stairwell. As you can see, there is a deadbolt on there. And then where the exit sign is supposed to be displayed, they crudely taped a cardboard cutout over. So it used to be a library and they were renting out the basement to people who played Dungeons and Dragons. And they didn't want the Dungeons and Dragons people to be able to access the other levels and come up. It has since then been bought out by a medical clinic. So we told them, you know, we need to get this right. And convert it back over and make sure we're up to fire code here. What it comes down to was Dungeons and Dragons. That is fantastic. Yeah, you got to keep them down in the dungeon, you know? At first I was like, oh, they taped over the exit sign. They must have converted this to something else, you know, because in previous episodes, we've had something happen and they've left the exit sign up there, right? And so I was like, oh, good for them. They've at least covered that it's an exit. No, no. Yeah, unfortunately, Nova still goes down into the basement and about 30 feet down the hallway on the other side there, there is the fire exit. Not a great one here. We were telling them, you know, sort of with a sense of urgency, we need to get this fixed. Yeah, I was thinking the same thing, Mia, when I first saw this photo was like, oh, it's no longer an exit, like the occupancy or they remodeled or something changed and it's no longer a needed exit. But especially if it's now a medical clinic that extra layers of code on top of that. And so that this is definitely not okay. Wow, this is pretty bad. Yeah, unfortunately, it's probably something you're just gonna have to replace the whole door and frame. I mean, you could potentially fill it at the top one. You know, Bryce is now certified as a fire door inspector. So we were kind of looking to devise a plan of, okay, can we salvage this door? Is there a proper way we can do this where it is still going to be a fire door? But unfortunately, I think we are going to have to order a new door for this one.  I can't tell what kind of deadbolt it is, if it's larger than an inch cutout. And that's the whole thing, it is a quick set. So I believe that was what inch and an eighth. So it's already more than the inch. So yeah. I appreciate that they matched the finish of the deadbolt to the exit device. You don't normally see that on jobs like this, guys. Unfortunately, it might have been my photography skills. It is a 612 on the exit device. And then it's a 619. Lack of good lighting and not great photography skills, I mean. All right. So my point is proven. You don't normally see them get matched in after the fact. But from here, yeah, it looks like it was matched. What's up with the little co-panger on there? They used to hang a sign on there saying Dungeons and Dragons in session. So you wouldn't interrupt their quest. To be fair, at least it looks like it's a command hook, so it's just stuck on there. So at least they didn't drill a hole for that, you know. No fire code violation on that one. Also, if they were removing this as an exit, that cardboard sign is not compliant either. You would still have to remove the sign and make sure all the signs leading down to this pathway is removed as well. And you would have to have another form of egress depending on the occupancy of the building as well. I heard that they replaced that piece of cardboard. They used to tape up different signs, depending on what land they were in for Dungeons and Dragons. So... This is not the exit you're looking for. Exit, if you dare. Yeah, and then our locksmith was talking about too, if there's ever, especially in today's day and age, sort of the code read, someone intruder in the building that shouldn't be in there, that's really your only exit on that side of the building. No, this is one of the worst I've seen, I think. Any last comments before we give it a knocking score? This one's a doozy. I think we pretty much covered it all and all the lands that could possibly associated with it.  I mean, as a now certified fire door inspector, I think I have to give this a 10. It is just unacceptable and needs to be taken care of ASAP. So I'm gonna give it a Bryce Clark sound on this one. It can't function as it's intended. Yeah, it's probably gonna have to be a 10, right? Can't get out. If there's a fire, you can't get out here. Yeah, it's pretty black and white on this one. I know sometimes we have a little bit of gray area, but especially now that we know the behind the scenes story of why that's set up in this application. Dungeons and Dragons, that's very important, but it sounds like it's a very dangerous quest that they just campaigned. And you know, you think about all the dragon fire that might be associated, you gotta have that fire exit. I'm with you guys, we might have another 10. Bumping up the scores. All right, well, thank you. Thank you, Bryce and Tom for joining us. It's a pleasure having you on the show, especially after meeting you in person at the MSC conference and learn a little bit more about your guys' business. It's awesome to put your hardware nerds together. It's awesome when we come together. Thank you as well, Mia, for hosting and putting this amazing show on that we can learn and laugh and cry all at the same time. And make sure you guys join us for the next episode of Unhinged. Our doors are always open because they're unhinged. If you wanna be featured on a future episode of Unhinged, please leave a comment down below. Thanks for watching.


Warning, this video is not meant to be taken seriously. It is merely for entertainment purposes because not all stereotypes are true so don't get triggered. Also, the video is full of satirical comments that PlayoffWater Society is like. Hi friends, this week I wanted to make a video about the gays. Considering October 11th is National Accommodation Day and I also came out on October 9th, fun fact. But I wanted to do with my YouTube friend Pearl. Hi, my name is Pearl and I'm your local lesbian. Thank you to Frederick for inviting me here but honestly I'm only doing this to defend my people. This is gays versus lesbians ultimate fighting championship.   What about the bisexuals?  Excuse me, y'all don't even exist, you're looking for attention. Lesbians are better than gays exhibit A. Sports. We came out of the womb like this. Last week when I went to go get a blood test, the doctors couldn't find any blood. Just Gatorade and testosterone. Lesbians love every type of ball. Well, except for two. I mean, gays aren't good at sports too. I'll just go and grind her. Gays have really good fashion sense. Skinny jeans, pastels, really, really, really short shorts.  Hey, lesbians are fashionable too. Spicy mama. Okay, so I might look like a creepy, best said boy but who says they aren't sexy? Um, I love all. Okay, fine, we're only good at shopping at Home Depot. But look how many pockets there are. Think of how handy that is. Hey, you got a hammer I can borrow? No, I'm sorry dude. Hatchet? Good. But if there's one thing we have in common, it's that we both love planos. Yeah, we do. As for our homosexual morning routine, I don't think we have a lot of steps.  Hi and welcome back to my lesbian channel. Today I'm showing you my morning routine. The first thing I like to do after I wake up is assert my masculinity. Then I head to the bathroom.  This is a season puzzle. First I wash my face. Then I comb my hair. Then I don't shave my armpit hair to make a feminist statement. And finally, if it's a really special day, I put some chapstick on. To put it lightly, gays are extra.  As in, you can handle me for an hour before you're gonna choke me. As in, you're up here and I'ma need you down here. Gays also have a lot of slang. Yes. Oh honey, you better work. Slaaaang.  Also there's a stereotype that gays have a feminine voice. I don't know what you're talking about.  Full disclaimer, lesbians have a really limited vocabulary. So, dude, bro.  Don't worry, we're chillin'. We're like pot heads, but without the pot.  We communicate solely with smirks, nods, playful slaps in the back. And that's all we have for you today. If you enjoyed this video, like, leave a comment down below, or better, subscribe because it posts videos every Saturday. And go subscribe to Pearl right here. This is her name. It's also gonna be right here and in the description. But wait, there's more! Just for you. Head on over to my channel to watch part two of Gays vs Lesbians. All right, I love you guys. Everything is less than three. Everything is love!


My dear viewer, welcome again today to our series of 40 days of prayer. I'm so excited that we are on the day, that night just one more day to go. For this it is beautiful sabat morning, I want to greet you wherever you are, happy sabat. It's very exciting to have a walk with the Lord for 40 days, pray and seeking him, and listening from him. I'm sure we have received numerous blessings and miracles and testimonies. The Lord has struck lives for the experience of 40 days of prayer. I know even today as we pray one more time together, the Lord is with us. Of course, for those of us who've been following this program, I know in most part of the world this program is aiding on Sabbath, but for us because of when we're studying, we are aiding tomorrow on Sunday. That's when we'll be having our 40th day, but we are happy that Sabbath morning to have you as part of our ministry. We are concluding on the series, the subseries of the free angels' messages. Yesterday, we looked at that angel's message that was calling people that they may be able to be faithful even at the point of death, that when it comes to the issue worship, we have nothing to compromise. We shall not accept to receive the mark of the beast, but we shall remain faithful unto the Lord, for we know that Babylon is fallen. So join with me as we get to the conclusion of these messages of free angels' messages. Again, going back to the everlasting gospel as you see how it is powerful and able to change lives that we can be encouraged to continue preaching and reaching out to many out here. Let's pray before we get to the text this morning. Thank you so much for the precious moment that we have to be before you one more time. This morning Lord we thank you and we praise you that you will continue blessing us and meeting us at our very point of needs as we look at the power of the everlasting gospel. Lord, may you speak to us in Jesus' name we pray. Amen. The text is of Revelation chapter 14. That where we began when we started speaking about the free angels' message that I want to especially read verse number six. And I saw another angel fly in the midst of heaven having the everlasting gospel to preach unto them that dwell on the earth and to every nation and kindred and tongue and people. I will read one more time. And I saw another angel fly in the midst of heaven having the everlasting gospel to preach unto them that dwell on the earth and to every nation and kindred and tongue and people. Not yesterday I said that the angels that we are looking at here are not the literal angels but these are the gospelers. These are the gospel ministers. These are the messengers of the Lord who are in different shapes and sizes male and female God people that is using to go out to speak of the great news of the second coming of Jesus Christ. Those who have devoted themselves for the ministry of the world. That is where I belong. That is where perhaps you also belong. The angels of our age are those who are preaching the gospel. Those who are walking out of their comfort zone going out and looking for the people in the world telling them of the love of God speaking to them about the grace of Jesus Christ and leading them to the cross of the caravan where their sins are being forgiven that they may be revived, encouraged, be transformed and be prepared for the second coming. Now John says I saw an angel fly in the midst of the sky with an everlasting gospel and I want to pose here today to say that the gospel, the good news, the mess in the world of God that we are preaching is everlasting. You see there are people out here preaching various types of gospel. I don't know what kind of a gospel you are preaching but for me I am preaching the everlasting gospel. There is good news of the second coming of Lord Jesus Christ. It is deemed the good news of how we have been saved from the penalty of sin. How we have been rescued from the hands of the evil one and we have been settled in the kingdom of God in the marvelous light where Christ is our high priest. This is everlasting gospel because this ministry, this gospel is not ending. This is Jesus Christ. You see Jesus is a gospel. He has a good news. Paul says I am not ashamed to preach about him because it is a power unto salvation. This gospel is everlasting because of who it is preaching about. The gospel is everlasting because it is the story of Jesus Christ. The chairs of him who came from heaven who once in the beginning was with God and who was God himself and who left the glories of heaven to come to incarnate, become like one of us and be born lowly, even lower than any other human being that he may reach out to you and to me we the good news that it is forgiven your sins are forgiven that we can have an access to the holy of holies through his blood. He is the gospel. So the gospel is everlasting because Jesus who is the gospel is everlasting and the angel here is flying and preaching the everlasting gospel. Paul says when I came to you I did not come in the knowledge of men and tongues but I came having purpose to know him and him crucified. Jesus is the everlasting gospel. It is him that we are telling the world to come to know. It is him as we move out as we get totally involved in the mission we are asking you to get totally involved in the life of Jesus. You see there are many people who come to church and they take up the gospel ministry but they do not preach the everlasting gospel because you have not preached the everlasting gospel until you are preaching him and him crucified. Until you preach Jesus and him crucified you are not touching about the everlasting gospel and so I am excited because the angel of this age is seen by John in the midst of the sky with Jesus Christ lifting the banner of the cross lifting the banner of salvation and telling people beyond him the Lamb of God that take away the sins of the world that they may come before giving of their sins be prepared for the second coming. You see friends as we end our 40 days of prayer it did for me it has been an experience redressing with the Lord every day having ma moment with the Lord every day but more importantly how this world of God has been able to minister to me in various ways through has he journeyed through the verses every single day listening from the scriptures and listening from the spirit of God and being guided by his will I am enriched and more I want to continue facing every single battle in the Lord because I know the message that I have is truly an everlasting gospel I want to invite you this moment as we go to prayer Who should not be excited to know that once you are out here with the wonder of God you are actually preaching an everlasting thing many of the things that we see here in the world many of the things that people men and women love to spend their life with and in they are not everlasting they are perishing they are passing away but the wonder of God is everlasting from eternity to eternity this wonder of God is immutable this wonder of God is powerful this wonder of God can change lives this wonder of God is able to transform this wonder of God has brought testimonies in our lifetime this wonder of God indeed is everlasting gospel and whoever desires to be ministered but this wonder of God will never in any way be ashamed this wonder of God is everlasting from eternity to eternity it is a means through which you and me should be found in this kingdom but not just you and me but even those who are not aware of what we know it is us through this world we shall step out of our comfort zone go and tell them that they may come and enjoy every joy with us as we celebrate the power of the gospel you see friends today as we are going to pray I want to invite you to especially pray for the Holy Spirit to lead in the final days of the general conference session we are so glad that as we are here on this morning the general conference is waiting up they are sitting to do the celebrating together with the other members in the world church we have new set of leadership we thank God for what he has done we are praying the Lord may be in the midst of the meetings and we have seen indeed it has been a very peaceful experience we have now have men and women that God has set apart for the next few years they may continue serving the church and leading the church we want to thank the Lord for what he has done pray for those coming into positions of new leadership and the Holy Spirit would equip them for the task ahead of them we also pray for opportunities to reach out to your family friends and neighbors and coworkers with the gospel of Jesus best everlasting gospel praying for your seven member list even after the 40 days of prayer I want to encourage you to continue praying for them because they daily need your intervention they daily need the presence of God they daily need your sacrifice they daily need your presence in their life through the intercessory prayer continue praying for them pray for God to give you at least one person who you can lead to Christ in this coming year I want to thank you so much because you've been so faithful in this ministry as you pray and you pray for so many people please pray for me pray for your pastor I need Jesus I need God and in his presence and in the power of the Holy Spirit don't say that man of God always God he has him when he prays I need you also to pray for me because without your prayers the devil will take advantage let's join together or struggling and fighting that we may finally have an access to the eternal life this together with what you have and many that the Lord will impress upon your thoughts and what heart as you pray please let's join together as we pray O gracious Father in Heaven we thank you so much for this precious moment that we have to be before you this about mourning Lord we thank you for having worked with us the experience of 40 days of prayer Lord we thank you because you are coming to the very end of this season of prayer and many are things you have done in our lives we have witnessed miracles Lord I want to thank you so much that even as we continue even after the 40 days of prayer may you Lord help us to still remain faithful in a work or prayer that every single day we will not give up we will not stop because 40 days have expired but Lord this was to teach us that it is possible to have a genuine prayer work with you every single day Lord I am praying that because I know you have revived us that we will not backslide we will never turn back again but we will continue in this good path we will continue experiencing your presence we will continue experiencing the revival we will continue experiencing the forms in our lives we will continue being excited about the gospel and being collaborators together with you reaching out to many people out here telling them of the everlasting gospel that not only ask that you are preparing but even our friends or brothers or sisters or family members or co workers together with us our workplaces that we all will be preparing for the second coming we want to thank you so much for what you have done through the sessions at the general conference that we thank you for the appointment of the new team we thank you Lord for the man your manservant elder Pastor Ted Wilson thank you for giving him a second chance to serve you one more time brother we know you are still in need of him if you wanted him to come out of this position you could have used your own ways but I know you still have an assignment with him for this particular church in this particular time Lord I pray that you continue using him and feeling him and the Lord we may not see him that we may see you through him together with this wife and family Lord we cover them with the blood of your son feel them with your spirit as they lean this great church globally they need you, they need us to hold their hands my father pray that you can touch every single church member out here I know that those who are having different opinions but Lord what shall we say than to say thank you God for your doing because you are always writing all that you do and we know that indeed the church will grow in the next days coming because of the leadership we pray for all the assistant presidents we pray the secretary we pray for departmental leaders of the general conference we pray for all the division presidents Lord we pray for the entire team of the division leadership and even down to the unions the conferences to the local church this is your church Lord take charge of your church those dissenting voices we rebuke them in the name of Jesus because we know Lord none of us here has a church this is a church and we can always cooperate with you we are not to fight for you but we are here to cooperate with you and to wait to see the deliverance that comes from you feel us with your spirit revive the church remind us of the great mission of the many perishing souls and dwell on the issues not important but Lord you shall agitate us right from deep within our hearts to see how we ought to be out here the opportunities around us the many souls are perishing that we may not be found idle but Lord every time be hastened to do your will to do your mission and lead many together with yourself to be received into a kingdom bless us Lord and meet as at our very point point of needs just spiritually but even socially economically, mentally Lord we are children revivers and come through for our struggles we need you every day we need you every afternoon we need you every evening we need you every morning we need you Lord every time please Father come those who say kill them I pray in Jesus name those who are looking for job opportunities those who are at fruze in Jesus name those who are struggling the marriages my Father pray for peace and love in those marriages I pray even for our children who are in school Lord may you make them to be great men and women in the days to come and I pray oh my Father when everything has been done and said yonda where the trumpet was sound I pray for myself my family, my church members Nairobi central church that shall be received into a kingdom for eternal life is my prayer in Jesus name amen amen thank you my dear viewer for being with me being with us in this journey for the 40 days this day for that nine we thank you so much just invite you one more time you have not clicked the button subscribe to our channel please do because even after 40 days of prayer I can promise you many other programs upcoming that you will be notified every time also want to encourage you and thank you so much for sharing continue sharing with us many friends and many people as you can so that we all may be best and prepared for the second coming tune tomorrow as we wind up these programs may the Lord bless you as worship today God be with you


Look at bee Bee $20 to anybody who knows what like semi Beyonce reference that was so the photos have finally arrived The whole twin anticipation is over a lot of you guys were acting as if you were carrying them right along with her And you couldn't wait for them to come and they are here and they've arrived But today I want to discuss specifically the photo why we all need to look at this photo a little deeper and a little longer The three things I want to talk about is one her making eye contact with the camera why that's important Revealing a little bit of skin what's important and why JZ never seems to be in any of the maternity maternal photo shoots And why there is some importance to that as well too first off Let's discuss that gaze that direct to camera gaze now a lot of times when you look at photos of babies number one The mother isn't in the most newborn photos are just of the child wearing a 1940s little one of those things called those little side those little side hats that are also headbands Shepo anyways, they're usually the baby by itself and if the mother is in it She is cradling her faces down or it's the emphasis really on the child And what Beyonce has done throughout her pregnancy both with bloom and now with Rumi and sir is asserting a sense of self Her not losing herself in motherhood embracing it as an extension of who she is not a transition or replacement of who she is And there's a lot of people who do that who abandon their erotic self when they put on this new maternal role And they forget that part of themselves needs to exist is important at all and has a place in society Has a role that anybody wants to see them as a sexual person because now they are mother and mother and lover are no longer To be used in the same sentence and I felt encouraged to watch not just Beyonce, but so many women Specifically in media who take on the role of motherhood and don't allow that to be the end of their erotic story And have their place as an individual I think it's problematic sometimes when I see mothers who exchange their profile photos on social media for that of their kids Like you can't lose your identity and while this an extension of you and a beautiful extension of you It is not a replacement for who you are and it is certainly Is not an excuse to abandon your erotic self which needs to thrive. We just it needs to thrive Now I want to talk about the little itty bits of skin We got showing now whether or not she had the mommy makeover Which I think is important to know because the average woman shouldn't be blind to the whole snapback phenomenon that happens Anyways, so for many women the transition into motherhood is the separation or the divorce of sexuality Not again just from herself, but also from society as well And they don't help by getting the bowl cut and the shapeless t-shirts and that soggy robe and those funky looking Slippers and nonetheless, we're all pretty comfortable to keep them in this box That that's what makes us feel like what you're doing your job as a mother So long as you're depriving yourself as a sexual person as long as you don't engage your children or expose them to any kind of sexuality They have some semblance of hopefully growing up and not becoming sexual themselves because God forbid we raise healthy viable confident sexually active adults and young adults nonetheless Her not relinquishing that part of herself because it's not motherly and her Determined to allow that to shine through in that little piece of thigh. It's just a massive It's so massive you to all of that and finally because it just got hella loud because I live in LA Thank you very much to the city for doing their job and fixing the roads But no, thank you for doing now. I'm trying to film a video son anyhow though I want to talk about the fact that Jay-Z is never present in any of these massive maternal moments whether it be the reveal the love on top reveal of Baby blue and the bump or of course the iconic twin pregnancy real photos and then most recently and whether that's done to preserve his Rap career or his image that he has surrounding his rap career I would I'm not really sure but I would like to think that it's done to give Beyonce the space to be adorned Appreciated and lusted after for her to be an object of desire and whether that desire comes from other women or it comes From men who wished that that was a mother of their children or dream of that being their kids Nonetheless of giving her that space to still be a sexual icon There's actually this website that exists out there where people can submit a photo of their partner Consensually and it's usually like a sexy photo of their partner and it is encouraged of people in the comments section Make sexual advances about that Individual or talk about self-pleasure themselves to that photo or talk about what they would do if they were with that person in that photo And that is for the partner of that person to see that okay They're still desired out there in the world that my partner still has a place as a sexual individual They still have a seat at the table when it comes to being adored appreciated and seen by others So I couldn't I would like to think that all that is done so that we can do what we're doing right now Which is center our attention and our focus on the sensuality the beauty the sexiness the ownership the accomplishment that is Sacrificing your body and finding yourself in the process and giving life in the process and Hopefully connecting with a better and bigger more beautiful more vibrant more sexy more sensual version of you Which is why I like to think what is I actually always remember that Beyonce said in one of her documentaries that? She wrote partition Well after she just had blue and it was at a time when that Part of her was so far from reality because she was nursing and going through the whole spit-up phase and you know having vomit in Her hair and all the above But that she allowed herself to revisit the erotic as a part of reconnecting with a piece of herself That she missed and wanted to reclaim and how much fun shooting that video was and how healthy Shooting that video was and so I encourage all you mothers out there to write your own partition to rewrite the script to Rewrite the narrative of what it means to be sensual and a mother and the importance of maintaining Eroticness through your very important and of course crucial job of raising a healthy individual So I want to hear from you guys in the comment section below. What did you think about the Beyonce photo? What was your gut reaction? I'm just like y'all honey like how did it really make you feel when you tuned into yourself? And let's hypothesize because it's fun. Why do you think that Jay-Z is never in any of these like major?  Perverting what's been your struggle with perverting yourself? Due to popular demand. I am now podcasting daily on anchor FM Now that is a place you guys can go to ask questions You actually voice notes and then I answer your questions So go to anchor FM slash Shamboody to sign up and to have your daily questions answered